[Help]: Writing Geotiff

@AlexeyPechnikov ,Hello, I was saving my results as geotiff with the following snippet. While it was working 2-3 weeks ago, now I am getting errors and cannot print the results. How can I solve this?

disp_subsett2 = disp_sbas_finish.rio.write_crs("epsg:4326", inplace=False)
disp_subsett2.rio.set_spatial_dims('lon', 'lat', inplace=True)
disp_subsett2.rio.to_raster(f'disp_sbas.tiff')

INFO:distributed.core:Event loop was unresponsive in Nanny for 9.99s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.99s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.91s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 10.00s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 10.00s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 11.12s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 11.12s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 11.12s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 10.12s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 10.12s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.81s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 9.81s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.79s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.79s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.79s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 11.17s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 11.18s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.98s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.98s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.91s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 9.91s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.07s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 10.30s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 10.51s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 10.50s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 10.50s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.94s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.94s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.73s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.73s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 20.85s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.80s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.79s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.98s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.99s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 20.15s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.33s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.34s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.15s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-2f2a6a160e5cca29371a725244bf31c5', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7d8c899dd4b0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7d8c899dd3f0>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:42143 remote=tcp://<IP_ADDRESS>:46968> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-2f2a6a160e5cca29371a725244bf31c5', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7d8c899dd6c0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7d8c899dd720>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:42143 remote=tcp://<IP_ADDRESS>:46968> already closed.
INFO:distributed.core:Connection to tcp://<IP_ADDRESS>:46968 has been closed.
INFO:distributed.scheduler:Remove client Client-worker-fe041166-02d4-11ef-9f2d-0242ac1c000c
INFO:distributed.scheduler:Receive client connection: Client-worker-fe0471b1-02d4-11ef-9f27-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:42562
INFO:distributed.scheduler:Receive client connection: Client-worker-fe041166-02d4-11ef-9f2d-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:42564
INFO:distributed.core:Event loop was unresponsive in Nanny for 20.29s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Connection to tcp://<IP_ADDRESS>:47004 has been closed.
INFO:distributed.scheduler:Remove client Client-worker-fed43ffe-02d4-11ef-9f21-0242ac1c000c
INFO:distributed.core:Event loop was unresponsive in Scheduler for 19.16s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.33s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.33s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.33s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.scheduler:Close client connection: Client-worker-fe041166-02d4-11ef-9f2d-0242ac1c000c
INFO:distributed.core:Event loop was unresponsive in Nanny for 9.20s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.batched:Batched Comm Closed <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:42143 remote=tcp://<IP_ADDRESS>:47004>
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 115, in _background_send
nbytes = yield coro
File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 767, in run
value = future.result()
File "/usr/local/lib/python3.10/dist-packages/distributed/comm/tcp.py", line 262, in write
raise CommClosedError()
distributed.comm.core.CommClosedError
INFO:distributed.scheduler:Receive client connection: Client-worker-fed43ffe-02d4-11ef-9f21-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:36614
INFO:distributed.scheduler:Close client connection: Client-worker-fed43ffe-02d4-11ef-9f21-0242ac1c000c
INFO:distributed.core:Connection to tcp://<IP_ADDRESS>:46982 has been closed.
INFO:distributed.scheduler:Remove client Client-worker-fe0471b1-02d4-11ef-9f27-0242ac1c000c
CancelledError                            Traceback (most recent call last)
 in <cell line: 3>()
1 disp_subsett2 = disp_sbas_finish.rio.write_crs("epsg:4326", inplace=False)
2 disp_subsett2.rio.set_spatial_dims('lon', 'lat', inplace=True)
----> 3 disp_subsett2.rio.to_raster(f'disp_sbas.tiff')
12 frames
/usr/local/lib/python3.10/dist-packages/distributed/client.py in _gather()
2231                         else:
2232                             raise exception.with_traceback(traceback)
-> 2233                         raise exc
2234                     if errors == "skip":
2235                         bad_keys.add(key)
CancelledError: ('getitem-7bc26de7f9999889faa36778be8593d0', 0, 0)

You can try to restart Dask scheduler.

@AlexeyPechnikov How do I do that, should I restart the runtime?

To restart Dask without lost of your current state re-execute this cell:
# cleanup for repeatable runs
if 'client' in globals():
    client.close()
client = Client()
client

When I got the error I mentioned above, I tried your suggestion and ran it again, but it gave the error again. I don't understand why this section, which always works, now gives errors, can you help?
INFO:distributed.scheduler:Receive client connection: Client-worker-aefae76f-0397-11ef-b3b3-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:55322
INFO:distributed.scheduler:Receive client connection: Client-worker-aefbf145-0397-11ef-b3a9-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:55358
INFO:distributed.scheduler:Receive client connection: Client-worker-aefb4dc0-0397-11ef-b3b5-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:55346
INFO:distributed.scheduler:Receive client connection: Client-worker-aefb4587-0397-11ef-b3ae-0242ac1c000c
INFO:distributed.core:Starting established connection to tcp://<IP_ADDRESS>:55338
INFO:distributed.core:Event loop was unresponsive in Nanny for 8.69s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 8.69s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 8.69s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 8.84s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 8.84s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.39s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.38s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 16.38s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.38s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 23.13s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 7.00s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 7.03s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 7.06s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 7.06s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.51s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 16.54s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.54s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.54s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.49s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.77s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 16.71s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.76s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.76s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.76s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 7.66s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 7.65s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 15.72s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Connection to tcp://<IP_ADDRESS>:55358 has been closed.
INFO:distributed.scheduler:Remove client Client-worker-aefbf145-0397-11ef-b3a9-0242ac1c000c
INFO:distributed.core:Event loop was unresponsive in Nanny for 30.65s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 30.65s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Nanny for 24.25s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 24.25s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
INFO:distributed.core:Event loop was unresponsive in Nanny for 16.13s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.scheduler:Close client connection: Client-worker-aefbf145-0397-11ef-b3a9-0242ac1c000c
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-4b0b85a4b992c9c92e509e4b19026e7b', 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081215570>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0812152d0>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-9e75dca11455a3812cd04b6fb1274b4e', 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081215180>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214df0>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-be2d503f4356c066ea6589c2861aca67', 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812149a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214640>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-ebddb1e0e6c9622df7b4f88e5599a8a9', 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c16ef0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c14220>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-d8f7a2cd7dbbdda3bdf448ee415bf604', 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c148e0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c14610>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-29d756474573f7e9b25ce3321b7ddba6', 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c146d0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c147c0>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-d3e7c4063ca67f326af952e1db795259', 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c14b80>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c14d30>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': 'original-xarray-correlation-aa49928eb1a8c35b11cedd952f043584'}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--41b80a94332ef20c3ed709e58f7500a6', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--8f3d587a87441f5ef9a64fb660fcac10', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--0a86185bfa68c433e1d1106944d7421b', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--2381badfd2c9e4e472a4bc0bcea480af', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray-ele-getitem-3dff0c0b1297f41c4c840dab18997453', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--9081f3a009580b2bdbdf45ab29f4a636', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('array-a46f6f15799e48fad8fde929682ebaf2', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--a84c523e18bdf364a6a8c89a63f8001b', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('mean_agg-aggregate-5ba091f58b50c5152b3a23f84a632321', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--03a074e697c65c1dcfa5108783844155', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--66835645657a33f1cee70013a61950ae', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('array-497117a73dfd25c61cca2e435389967a', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--9b575535d46ac4471b0cd6e970213457', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
INFO:distributed.core:Event loop was unresponsive in Nanny for 6.75s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
INFO:distributed.core:Event loop was unresponsive in Scheduler for 6.74s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-67e075ce076bbb85dc28c17b673d3531', 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c32ce0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c32f20>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-8dd107733eec2a4017b992a660a9cd27', 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c32b30>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c32c20>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write [{'op': 'task-erred', 'key': ('getitem-a90d0488e2ae55b85f016009c3cf21c7', 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c330a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c33310>}]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 6074, in send_all
c.send(*msgs)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 14, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0825970a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0825970d0>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': 'original-xarray-correlation-aa49928eb1a8c35b11cedd952f043584'}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--41b80a94332ef20c3ed709e58f7500a6', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--8f3d587a87441f5ef9a64fb660fcac10', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--0a86185bfa68c433e1d1106944d7421b', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--2381badfd2c9e4e472a4bc0bcea480af', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('getitem-3dff0c0b1297f41c4c840dab18997453', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--9081f3a009580b2bdbdf45ab29f4a636', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('array-a46f6f15799e48fad8fde929682ebaf2', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--a84c523e18bdf364a6a8c89a63f8001b', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('mean_agg-aggregate-5ba091f58b50c5152b3a23f84a632321', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--03a074e697c65c1dcfa5108783844155', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--66835645657a33f1cee70013a61950ae', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('array-497117a73dfd25c61cca2e435389967a', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'key-in-memory', 'key': ('xarray--9b575535d46ac4471b0cd6e970213457', 0, 0)}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 12, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe082597130>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe082597160>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55322> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
INFO:distributed.core:Connection to tcp://<IP_ADDRESS>:55322 has been closed.
INFO:distributed.scheduler:Remove client Client-worker-aefae76f-0397-11ef-b3b3-0242ac1c000c
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 10, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5270>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe0839b5450>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 11, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe0812152a0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081214f10>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 0, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b32ad0>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b31e40>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 13, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe083c15d20>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe083c15390>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55338> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/distributed/scheduler.py", line 5669, in report
c.send(msg)
File "/usr/local/lib/python3.10/dist-packages/distributed/batched.py", line 156, in send
raise CommClosedError(f"Comm {self.comm!r} already closed.")
distributed.comm.core.CommClosedError: Comm <TCP (closed) Scheduler->Client local=tcp://<IP_ADDRESS>:35395 remote=tcp://<IP_ADDRESS>:55346> already closed.
CRITICAL:distributed.scheduler:Closed comm <BatchedSend: closed> while trying to write {'op': 'task-erred', 'key': ('getitem-b00c286a4d2c5aedc20f053ecf45334d', 15, 0, 0), 'exception': <distributed.protocol.serialize.Serialized object at 0x7fe081b52710>, 'traceback': <distributed.protocol.serialize.Serialized object at 0x7fe081b53d00>}
Traceback (most recent call last):