Shuffle prototype: Feedback (disk usage + workers dying)

I have tried #8223 on a ~3.4TB gzipped Parquet dataset.
I tried four runs so far, with two different behaviours

First I tried the whole dataset. I got to the last step (to_parquet), but then ran out of disk space with over 4TB being used by the shuffle
The subsequent three runs I have tried to use a subset (by passing a smaller number of files to read_parquet). I did not get over the set_index step. Workers seem to die and the computation hangs, always towards the end. Errors below.

Based on the first run, this does look like it could've actually been successful if I had more disk space. That's quite exciting, as external sorting has been a big issue for me.
Code used
Sadly I cannot share the data. I am at least sharing the code I'm using.
from dask.distributed import Client, LocalCluster
from pathlib import Path

cluster = LocalCluster(n_workers=10, memory_limit='25GB', threads_per_worker=2)
client = Client(cluster)
paths_files = list(Path('.').glob('*/*.parquet'))
columns = ['col_a', 'col_b']

data = dd.read_parquet(paths_files, columns=columns)

# Here is where workers fail and computation hangs (3 out of 4 tries with a subset)
data = data.set_index('col_a', shuffle="service")
# Here is where it ran out of disk space (1 out of 4 tries)
data.to_parquet('cache-imported.gz.parq', compression='gzip')

Errors

Orginal error
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:38001
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 205, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 319, in connect
    handshake = await asyncio.wait_for(comm.read(), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:38001 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:38215
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 245, in write
    async def write(self, msg, serializers=None, on_error="message"):
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 452, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 320, in connect
    await asyncio.wait_for(comm.write(local_info), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 454, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:38215 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:39419
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 245, in write
    async def write(self, msg, serializers=None, on_error="message"):
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 452, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 320, in connect
    await asyncio.wait_for(comm.write(local_info), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 454, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:39419 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:42233
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 245, in write
    async def write(self, msg, serializers=None, on_error="message"):
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 452, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 320, in connect
    await asyncio.wait_for(comm.write(local_info), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 454, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:42233 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:33333
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 245, in write
    async def write(self, msg, serializers=None, on_error="message"):
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 452, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 320, in connect
    await asyncio.wait_for(comm.write(local_info), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 454, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:33333 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:38215
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 245, in write
    async def write(self, msg, serializers=None, on_error="message"):
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 452, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 320, in connect
    await asyncio.wait_for(comm.write(local_info), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 454, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:38215 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:33333
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 245, in write
    async def write(self, msg, serializers=None, on_error="message"):
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 452, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 320, in connect
    await asyncio.wait_for(comm.write(local_info), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 454, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:33333 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:33333
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 245, in write
    async def write(self, msg, serializers=None, on_error="message"):
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 452, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 320, in connect
    await asyncio.wait_for(comm.write(local_info), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 454, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:33333 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:36893
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 205, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 319, in connect
    handshake = await asyncio.wait_for(comm.read(), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:36893 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:41795
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 205, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 319, in connect
    handshake = await asyncio.wait_for(comm.read(), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:41795 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:41795
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 245, in write
    async def write(self, msg, serializers=None, on_error="message"):
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 452, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 320, in connect
    await asyncio.wait_for(comm.write(local_info), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 454, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:41795 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:41795
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 245, in write
    async def write(self, msg, serializers=None, on_error="message"):
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 452, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 320, in connect
    await asyncio.wait_for(comm.write(local_info), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 454, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:41795 after 30 s
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:38707
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 205, in read
    frames_nbytes = await stream.read_bytes(fmt_size)
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 319, in connect
    handshake = await asyncio.wait_for(comm.read(), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2654, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3982, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 3959, in _get_data
    comm = await rpc.connect(worker)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:38707 after 30 s
distributed.utils - ERROR - ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", [("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'register-replica', 'released', 'compute-task-1635104458.1960478',<PHONE_NUMBER>.872155), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'released', 'fetch', {}, 'compute-task-1635104458.1960478',<PHONE_NUMBER>.8722844), ('gather-dependencies', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'stimulus',<PHONE_NUMBER>.872853), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'flight', {}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.8728676), ('request-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.384826), ('receive-dep-failed', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.914738), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'missing-dep'), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'fetch', {}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.9147604), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'missing', {}, 'ensure-communicating-1635104503.91477',<PHONE_NUMBER>.9147818), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'missing', 'fetch', {}, 'find-missing-1635104504.5391095',<PHONE_NUMBER>.5398495), ('gather-dependencies', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'stimulus',<PHONE_NUMBER>.5398893), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'flight', {}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.5398993), ('request-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.5399654), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'cancelled', {}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5406826), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'cancelled', 'cancelled', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'released'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5406876), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'cancelled', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'released'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.540689), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'release-key', 'processing-released-1635104504.5400004'), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'cancelled', 'released', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'forgotten'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5407002), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'released', 'forgotten', {}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5407038), ('lost-during-gather', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538'), ('receive-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.287821)])
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils.py", line 648, in log_errors
    yield
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2745, in gather_dep
    assert ts, (d, self.story(d))
AssertionError: ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", [("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'register-replica', 'released', 'compute-task-1635104458.1960478',<PHONE_NUMBER>.872155), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'released', 'fetch', {}, 'compute-task-1635104458.1960478',<PHONE_NUMBER>.8722844), ('gather-dependencies', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'stimulus',<PHONE_NUMBER>.872853), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'flight', {}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.8728676), ('request-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.384826), ('receive-dep-failed', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.914738), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'missing-dep'), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'fetch', {}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.9147604), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'missing', {}, 'ensure-communicating-1635104503.91477',<PHONE_NUMBER>.9147818), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'missing', 'fetch', {}, 'find-missing-1635104504.5391095',<PHONE_NUMBER>.5398495), ('gather-dependencies', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'stimulus',<PHONE_NUMBER>.5398893), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'flight', {}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.5398993), ('request-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.5399654), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'cancelled', {}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5406826), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'cancelled', 'cancelled', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'released'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5406876), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'cancelled', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'released'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.540689), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'release-key', 'processing-released-1635104504.5400004'), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'cancelled', 'released', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'forgotten'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5407002), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'released', 'forgotten', {}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5407038), ('lost-during-gather', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538'), ('receive-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.287821)])
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7f5f45925ac0>>, <Task finished name='Task-8548' coro=<Worker.gather_dep() done, defined at /root/miniconda3/lib/python3.9/site-packages/distributed/worker.py:2588> exception=AssertionError(("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", [("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'register-replica', 'released', 'compute-task-1635104458.1960478',<PHONE_NUMBER>.872155), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'released', 'fetch', {}, 'compute-task-1635104458.1960478',<PHONE_NUMBER>.8722844), ('gather-dependencies', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'stimulus',<PHONE_NUMBER>.872853), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'flight', {}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.8728676), ('request-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.384826), ('receive-dep-failed', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.914738), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'missing-dep'), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'fetch', {}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.9147604), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'missing', {}, 'ensure-communicating-1635104503.91477',<PHONE_NUMBER>.9147818), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'missing', 'fetch', {}, 'find-missing-1635104504.5391095',<PHONE_NUMBER>.5398495), ('gather-dependencies', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'stimulus',<PHONE_NUMBER>.5398893), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'flight', {}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.5398993), ('request-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.5399654), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'cancelled', {}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5406826), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'cancelled', 'cancelled', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'released'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5406876), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'cancelled', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'released'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.540689), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'release-key', 'processing-released-1635104504.5400004'), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'cancelled', 'released', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'forgotten'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5407002), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'released', 'forgotten', {}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5407038), ('lost-during-gather', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538'), ('receive-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.287821)]))>)
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2745, in gather_dep
    assert ts, (d, self.story(d))
AssertionError: ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", [("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'register-replica', 'released', 'compute-task-1635104458.1960478',<PHONE_NUMBER>.872155), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'released', 'fetch', {}, 'compute-task-1635104458.1960478',<PHONE_NUMBER>.8722844), ('gather-dependencies', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'stimulus',<PHONE_NUMBER>.872853), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'flight', {}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.8728676), ('request-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.384826), ('receive-dep-failed', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.914738), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'missing-dep'), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'fetch', {}, 'ensure-communicating-1635104468.8727045',<PHONE_NUMBER>.9147604), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'missing', {}, 'ensure-communicating-1635104503.91477',<PHONE_NUMBER>.9147818), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'missing', 'fetch', {}, 'find-missing-1635104504.5391095',<PHONE_NUMBER>.5398495), ('gather-dependencies', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'stimulus',<PHONE_NUMBER>.5398893), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'fetch', 'flight', {}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.5398993), ('request-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.5399654), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'cancelled', {}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5406826), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'cancelled', 'cancelled', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'released'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5406876), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'flight', 'cancelled', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'released'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.540689), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'release-key', 'processing-released-1635104504.5400004'), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'cancelled', 'released', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)": 'forgotten'}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5407002), ("('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)", 'released', 'forgotten', {}, 'processing-released-1635104504.5400004',<PHONE_NUMBER>.5407038), ('lost-during-gather', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538'), ('receive-dep', 'tcp://<IP_ADDRESS>:41795', {"('re-quantiles-1-59cbe1d46b9bfe7bf5a9da9c07036ff5', 75)"}, 'ensure-communicating-1635104504.5398538',<PHONE_NUMBER>.287821)])

 Another error during `to_parquet`
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:37667
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2741, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4031, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4011, in _get_data
    response = await send_recv(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 640, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://<IP_ADDRESS>:37530 remote=tcp://<IP_ADDRESS>:37667>: ConnectionResetError: [Errno 104] Connection reset by peer
distributed.worker - ERROR - failed during get data with tcp://<IP_ADDRESS>:37667 -> tcp://<IP_ADDRESS>:34511
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 1649, in get_data
    response = await comm.read(deserializers=serializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed)  local=tcp://<IP_ADDRESS>:37667 remote=tcp://<IP_ADDRESS>:37530>: TimeoutError: [Errno 110] Connection timed out
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:36281
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2741, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4031, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4011, in _get_data
    response = await send_recv(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 640, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://<IP_ADDRESS>:45000 remote=tcp://<IP_ADDRESS>:36281>: ConnectionResetError: [Errno 104] Connection reset by peer
distributed.worker - ERROR - failed during get data with tcp://<IP_ADDRESS>:36281 -> tcp://<IP_ADDRESS>:34511
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 1649, in get_data
    response = await comm.read(deserializers=serializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed)  local=tcp://<IP_ADDRESS>:36281 remote=tcp://<IP_ADDRESS>:45000>: TimeoutError: [Errno 110] Connection timed out
tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOLoop object at 0x7f348fe7bac0>>, <Task finished name='Task-9116' coro=<Worker.heartbeat() done, defined at /root/miniconda3/lib/python3.9/site-packages/distributed/worker.py:1182> exception=OSError('Timed out during handshake while connecting to tcp://<IP_ADDRESS>:43431 after 30 s')>)
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 245, in write
    async def write(self, msg, serializers=None, on_error="message"):
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 452, in wait_for
    fut.result()
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 320, in connect
    await asyncio.wait_for(comm.write(local_info), time_left())
  File "/root/miniconda3/lib/python3.9/asyncio/tasks.py", line 454, in wait_for
    raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/ioloop.py", line 741, in _run_callback
    ret = callback()
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/ioloop.py", line 765, in _discard_future_result
    future.result()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 1231, in heartbeat
    raise e
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 1190, in heartbeat
    response = await retry_operation(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 860, in send_recv_from_rpc
    comm = await self.pool.connect(self.addr)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1048, in connect
    raise exc
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 1032, in connect
    comm = await fut
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/core.py", line 324, in connect
    raise OSError(
OSError: Timed out during handshake while connecting to tcp://<IP_ADDRESS>:43431 after 30 s
distributed.worker - ERROR - failed during get data with tcp://<IP_ADDRESS>:36281 -> tcp://<IP_ADDRESS>:37667
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 1649, in get_data
    response = await comm.read(deserializers=serializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed)  local=tcp://<IP_ADDRESS>:36281 remote=tcp://<IP_ADDRESS>:44898>: TimeoutError: [Errno 110] Connection timed out
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:36281
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2741, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4031, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4011, in _get_data
    response = await send_recv(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 640, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://<IP_ADDRESS>:44898 remote=tcp://<IP_ADDRESS>:36281>: ConnectionResetError: [Errno 104] Connection reset by peer
distributed.worker - ERROR - failed during get data with tcp://<IP_ADDRESS>:43515 -> tcp://<IP_ADDRESS>:37275
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 1649, in get_data
    response = await comm.read(deserializers=serializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed)  local=tcp://<IP_ADDRESS>:43515 remote=tcp://<IP_ADDRESS>:45898>: TimeoutError: [Errno 110] Connection timed out
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:43515
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2741, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4031, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4011, in _get_data
    response = await send_recv(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 640, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://<IP_ADDRESS>:45898 remote=tcp://<IP_ADDRESS>:43515>: ConnectionResetError: [Errno 104] Connection reset by peer
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:37275
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2741, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4031, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4011, in _get_data
    response = await send_recv(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 640, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://<IP_ADDRESS>:40650 remote=tcp://<IP_ADDRESS>:37275>: ConnectionResetError: [Errno 104] Connection reset by peer
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:43515
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2741, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4031, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4011, in _get_data
    response = await send_recv(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 640, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://<IP_ADDRESS>:46346 remote=tcp://<IP_ADDRESS>:43515>: ConnectionResetError: [Errno 104] Connection reset by peer
distributed.worker - ERROR - failed during get data with tcp://<IP_ADDRESS>:43515 -> tcp://<IP_ADDRESS>:37667
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 1649, in get_data
    response = await comm.read(deserializers=serializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed)  local=tcp://<IP_ADDRESS>:43515 remote=tcp://<IP_ADDRESS>:46346>: TimeoutError: [Errno 110] Connection timed out
distributed.worker - ERROR - failed during get data with tcp://<IP_ADDRESS>:37275 -> tcp://<IP_ADDRESS>:34511
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 1649, in get_data
    response = await comm.read(deserializers=serializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed)  local=tcp://<IP_ADDRESS>:37275 remote=tcp://<IP_ADDRESS>:40650>: TimeoutError: [Errno 110] Connection timed out
distributed.worker - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker.html#memtrim for more information. -- Unmanaged memory: 29.25 GiB -- Worker memory limit: 46.57 GiB
distributed.worker - ERROR - failed during get data with tcp://<IP_ADDRESS>:37667 -> tcp://<IP_ADDRESS>:36281
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
TimeoutError: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 1649, in get_data
    response = await comm.read(deserializers=serializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed)  local=tcp://<IP_ADDRESS>:37667 remote=tcp://<IP_ADDRESS>:37070>: TimeoutError: [Errno 110] Connection timed out
distributed.worker - ERROR - Worker stream died during communication: tcp://<IP_ADDRESS>:37667
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 867, in _read_to_buffer
    bytes_read = self.read_from_fd(buf)
  File "/root/miniconda3/lib/python3.9/site-packages/tornado/iostream.py", line 1140, in read_from_fd
    return self.socket.recv_into(buf, len(buf))
ConnectionResetError: [Errno 104] Connection reset by peer

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 2741, in gather_dep
    response = await get_data_from_worker(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4031, in get_data_from_worker
    return await retry_operation(_get_data, operation="get_data_from_worker")
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 385, in retry_operation
    return await retry(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/utils_comm.py", line 370, in retry
    return await coro()
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/worker.py", line 4011, in _get_data
    response = await send_recv(
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/core.py", line 640, in send_recv
    response = await comm.read(deserializers=deserializers)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 221, in read
    convert_stream_closed_error(self, e)
  File "/root/miniconda3/lib/python3.9/site-packages/distributed/comm/tcp.py", line 126, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) Ephemeral Worker->Worker for gather local=tcp://<IP_ADDRESS>:37070 remote=tcp://<IP_ADDRESS>:37667>: ConnectionResetError: [Errno 104] Connection reset by peer
distributed.worker - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker.html#memtrim for more information. -- Unmanaged memory: 33.46 GiB -- Worker memory limit: 46.57 GiB
distributed.worker - WARNING - Unmanaged memory use is high. This may indicate a memory leak or the memory may not be released to the OS; see https://distributed.dask.org/en/latest/worker.html#memtrim for more information. -- Unmanaged memory: 33.46 GiB -- Worker memory limit: 46.57 GiB
distributed.worker - WARNING - Compute Failed
Function:  unpack
args:      (<dask.dataframe.shuffle_service.ShuffleService object at 0x7f4627812d00>, 240, None)
kwargs:    {}
Exception: 'AttributeError("\'ShuffleService\' object has no attribute \'retrieve_futures\'")'

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/ipykernel_815540/1044375382.py in <module>
----> 1 data.to_parquet('cache-imported.gz.parq', compression='gzip')

~/miniconda3/lib/python3.9/site-packages/dask/dataframe/core.py in to_parquet(self, path, *args, **kwargs)
   4558         from .io import to_parquet
   4559
-> 4560         return to_parquet(self, path, *args, **kwargs)
   4561
   4562     def to_orc(self, path, *args, **kwargs):

~/miniconda3/lib/python3.9/site-packages/dask/dataframe/io/parquet/core.py in to_parquet(df, path, engine, compression, write_index, append, overwrite, ignore_divisions, partition_on, storage_options, custom_metadata, write_metadata_file, compute, compute_kwargs, schema, **kwargs)
    723     if compute:
    724         if write_metadata_file:
--> 725             return compute_as_if_collection(
    726                 DataFrame, graph, (final_name, 0), **compute_kwargs
    727             )

~/miniconda3/lib/python3.9/site-packages/dask/base.py in compute_as_if_collection(cls, dsk, keys, scheduler, get, **kwargs)
    313     schedule = get_scheduler(scheduler=scheduler, cls=cls, get=get)
    314     dsk2 = optimization_function(cls)(dsk, keys, **kwargs)
--> 315     return schedule(dsk2, keys, **kwargs)
    316
    317

~/miniconda3/lib/python3.9/site-packages/distributed/client.py in get(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
   2691                     should_rejoin = False
   2692             try:
-> 2693                 results = self.gather(packed, asynchronous=asynchronous, direct=direct)
   2694             finally:
   2695                 for f in futures.values():

~/miniconda3/lib/python3.9/site-packages/distributed/client.py in gather(self, futures, errors, direct, asynchronous)
   1967             else:
   1968                 local_worker = None
-> 1969             return self.sync(
   1970                 self._gather,
   1971                 futures,

~/miniconda3/lib/python3.9/site-packages/distributed/client.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
    863             return future
    864         else:
--> 865             return sync(
    866                 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
    867             )

~/miniconda3/lib/python3.9/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    325     if error[0]:
    326         typ, exc, tb = error[0]
--> 327         raise exc.with_traceback(tb)
    328     else:
    329         return result[0]

~/miniconda3/lib/python3.9/site-packages/distributed/utils.py in f()
    308             if callback_timeout is not None:
    309                 future = asyncio.wait_for(future, callback_timeout)
--> 310             result[0] = yield future
    311         except Exception:
    312             error[0] = sys.exc_info()

~/miniconda3/lib/python3.9/site-packages/tornado/gen.py in run(self)
    760
    761                     try:
--> 762                         value = future.result()
    763                     except Exception:
    764                         exc_info = sys.exc_info()

~/miniconda3/lib/python3.9/site-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
   1832                             exc = CancelledError(key)
   1833                         else:
-> 1834                             raise exception.with_traceback(traceback)
   1835                         raise exc
   1836                     if errors == "skip":

~/miniconda3/lib/python3.9/site-packages/dask/dataframe/shuffle_service.py in unpack()
     49     This is a task in the task graph
     50     """
---> 51     return service.get(i)
     52
     53

~/miniconda3/lib/python3.9/site-packages/dask/dataframe/shuffle_service.py in get()
    726                 # ^ this causes the `process_received_shards` threads to flush any data out
    727                 # of `worker.shuffler`, then stop.
--> 728                 concurrent.futures.wait(self.retrieve_futures)
    729                 self.retrieve_futures.clear()
    730

AttributeError: 'ShuffleService' object has no attribute 'retrieve_futures'

Environment

Dask version: '2021.09.1+26.gfd1b02b6' (#8223)
Pandas: 1.3.1
Python version: 3.9.5
Operating System: Ubuntu 18.04.5 LTS
Install method (conda, pip, source): conda

Hey @DahnJ, thanks for trying this out. Sorry it's taken a while to get back to you.
First:

How are you deploying the cluster? How many workers? CPUs, threads, memory, and disk per worker?
How many partitions in your parquet dataset, and how large is each partition?
Did you restart the cluster after each failure (client.restart()?) Did you try recreating it from scratch?

If you're restarting instead of creating from scratch, are your workers using Nannies? You can check this with {addr: info['nanny'] is not None for addr, info in client.scheduler_info()['workers'].items()}

Obvious question, but a good first check: do you actually have enough disk space to hold both the input dataset and the shuffled output dataset? As in, >6.8TB? Because AFAICT you're not overwriting your input dataset.

I got to the last step (to_parquet), but then ran out of disk space with over 4TB being used by the shuffle

to_parquet isn't really the "last step"—that is, it's not like the entire shuffle happens, and then the to_parquet happens. Instead, they're interleaved.
Basically, the way the shuffle works is that the entire input DataFrame gets temporarily written to disk in little fragments. At the midpoint in the shuffle, all of the data is on disk. Past the midpoint, we're reading data from disk back into memory for each output partition—on the dashboard, you'll see this as tasks called unpack. After an unpack task runs, the to_parquet task for that partition will run immediately, writing the shuffled partition back to disk. (Otherwise, if you wanted for all the unpacks to run before doing any to_parquet, you'd have to have all 3.4TB of data in memory at once!)
So at the point at which the very first to_parquet runs, I'd expect ~3.4TB of disk to be used by all these temporary fragments. Probably more (maybe less), because we're using a different file format.
Now, each unpack does delete its temporary fragment from disk, so in theory you should be able to "trade" the disk space taken by the temporary fragment for the output parquet partition.
But in practice, the temporary fragments are using a different file format (that's uncompressed and possibly not that efficient), so they're probably not exactly 3.4TB. If they're a lot larger than parquet, then maybe we're using so much space with the temporary fragments that there isn't enough room for cache-imported.gz.parq. This still doesn't quite make sense for me, since in this case I'd expect peak disk usage to be at the midpoint, and with every unpack->to_parquet it would go down. So if you were running out of disk space, I'd expect to see an error in transfer (before the midpoint), not in to_parquet.

I did not get over the set_index step. Workers seem to die and the computation hangs, always towards the end.

Can you try completely recreating the cluster before doing these subsets? #8223 is very bad at cleaning up after itself after an error. There may be some internal state that's broken after your first try failed.
From one of your error messages, it's looking like the re-quantiles step in set_index is failing. This isn't actually part of the shuffle, per se; it's when dask computes your entire input data once to pick good divisions for you automatically. If you pass divisions= to set_index, you can skip this.
It seems like your workers are running out of memory? That may be why they're crashing. Knowing the resources you have and the size of the input partitions would help understand this.

Thank you for the reply @gjoseph92 !

How are you deploying the cluster? How many workers? CPUs, threads, memory, and disk per worker?

I am starting a LocalCluster on a single machine with these parameters:
LocalCluster(n_workers=10, memory_limit='25GB', threads_per_worker=2)

Since it's a local cluster, I assume the disk per worker is only limited by the disk space left on the device.

How many partitions in your parquet dataset, and how large is each partition?

5800 partitions, each about 620M, when gzipped (as in .gz.parquet)

Did you restart the cluster after each failure (client.restart()?) Did you try recreating it from scratch?

Yes. I run the cluster from a Jupyter notebook and I restart the kernel each time, which I believe should clean everything completely. Some of the worker files are only deleted when the new cluster starts.

Obvious question, but a good first check: do you actually have enough disk space to hold both the input dataset and the shuffled output dataset? As in, >6.8TB? Because AFAICT you're not overwriting your input dataset.

Yes, at the time of writing I had 7.4TB. I might be able to try with even more.

I got to the last step (to_parquet), but then ran out of disk space with over 4TB being used by the shuffle

to_parquet isn't really the "last step"—that is, it's not like the entire shuffle happens, and then the to_parquet happens. Instead, they're interleaved.

What I meant by that is that the set_index partition calculation already ran through.

But in practice, the temporary fragments are using a different file format (that's uncompressed and possibly not that efficient), so they're probably not exactly 3.4TB.

Aha, if it's uncompressed, then that would mean I probably need quite a bit more storage! When I uncompress, the 620M partition becomes 1.3G. So I would probably need at least 7-8TB + whatever inefficiency in the temporary storage requires, is that correct?

From one of your error messages, it's looking like the re-quantiles step in set_index is failing. This isn't actually part of the shuffle, per se; it's when dask computes your entire input data once to pick good divisions for you automatically. If you pass divisions= to set_index, you can skip this.

Good point, I will try to provide some divisions and see what happens. Not a straightforward thing, but suppose I could e.g. sample the string ids and split them by first n characters.

It seems like your workers are running out of memory? That may be why they're crashing. Knowing the resources you have and the size of the input  #partitions would help understand this.

Perhaps I could lower the number of workers/threads and increase the memory limit per worker.

I am starting a LocalCluster on a single machine

Overall note: this new shuffle is designed specifically for multi-machine clusters. On a single machine, you'd probably be better off using shuffle="disk". It's sort of similar to this new shuffle, except it may have more efficient serialization, plus it skips the unnecessary networking step of sending the data back to itself.

When I uncompress, the 620M partition becomes 1.3G. So I would probably need at least 7-8TB + whatever inefficiency in the temporary storage requires, is that correct?

Yes, I think this is the problem!

You just don't have enough disk space. You'd need space for the entire compressed input dataset, plus the entire uncompressed input dataset as temporary files. Plus maybe a little extra.

1.3G partitions are really large. In general, dask won't perform very well when partition sizes are that big.
How much memory does your machine have?
Most operations will involve duplicating the input partition, sometimes multiple times. Also, dask can be slow to clean up data (https://github.com/dask/distributed/issues/5114), and data transfer involves even more copies (https://github.com/dask/distributed/pull/5208). So when your partitions are 100MB, a couple extra copies per thread is still bad, but it's a lot less total memory than a couple extra copies of a 1.3GB thing.
I also haven't tested the shuffle service on partitions that large.  And the shuffling process itself involves making intermediate copies, which we try to drop a quickly as possible, but sometimes have to hang onto for a little while.

Good point, I will try to provide some divisions and see what happens. Not a straightforward thing, but suppose I could e.g. sample the string ids and split them by first n characters.

Another think you can do is run reindexed = df.set_index(...) once, and then just look at reindexed.divisions and hardcode whatever dask calculated for next time :)

On a single machine, you'd probably be better off using shuffle="disk".

Aha! Thanks, it never ceases to amaze me how many Dask details I miss out on. I will give it a try and see if it works for my case.

How much memory does your machine have?

~230GB available

Most operations will involve duplicating the input partition, sometimes multiple times. Also, dask can be slow to clean up data ([Idea] Could workers sometimes know when to release keys on their own? distributed#5114), and data transfer involves even more copies (Deserialization: zero-copy merge subframes when possible [better than last PR] distributed#5208). So when your partitions are 100MB, a couple extra copies per thread is still bad, but it's a lot less total memory than a couple extra copies of a 1.3GB thing.

Again, this was very helpful and enlightening, thanks!

Another think you can do is run reindexed = df.set_index(...) once, and then just look at reindexed.divisions and hardcode whatever dask calculated for next time :)

Good point


[OTHER] How to Contribute to the Frontend of Taipy

What would you like to share or ask?
Hi team,
I was wondering if there is a way to contribute to the frontend of Taipy. I'm good with Next.js and React, but I wasn't able to run it locally even after following the instructions in the README (Taipy-GUI and Taipy frontend).
Do you accept contributions in that part, such as optimization of the existing Taipy website?
Thanks!
Code of Conduct

[X] I have checked the existing issues.
[ ] I am willing to work on this issue (optional)

You can take any issue with "good first issue" or "Help wanted" labels.
This list includes several issues related to the GUI that might interest you. Contributing to the website part seems difficult.
Could you describe what error you got when you ran locally so that we can help you?

@FlorianJacta
Thank you!
I missed the notification somehow, I understand the codebase structure better now so I'm closing this issue. I will make sure to pick up good first issues related to the GUI.

Great! We are here to help if you need anything


feat: add hardcoded shortcut amounts for binance (only binance for now)

https://github.com/map3xyz/supercharge_flutter/pull/2


Offset.top is hardcoded in DateTimeField

After calculating the dynamic offset.top correctly it is hardcoded to 40 here when no direction prop is provided.  I might be misunderstanding the logic here but it seems the DateTimeField will always be positioned at this offset.

+1 for a PR enhancement

In the meantime, you can use a shim to overwrite... it's hacky but if you need it..well, you need it!

see #155


Change Sirupsen to sirupsen

The project has been renamed from Sirupsen to sirupsen.
The conflicting imports mixing upper- and lower-case of
the package are causing issues.
More details here:
https://github.com/sirupsen/logrus/issues/570#issuecomment-313933276
I encountered this issue while trying to import a package which was using Sirupsen with lowercase.
I got the following error:
case-insensitive import collision: "github.com/openvswitch/ovn-kubernetes/go-controller/vendor/github.com/sirupsen/logrus" and "github.com/openvswitch/ovn-kubernetes/go-controller/vendor/github.com/Sirupsen/logrus"

Please add a Signed-off-by.

Added Signed-off-by and fixed the merge conflict.


CREATE_FAILED with 2.9.1

Hi all,
I am trying to create a new cluster using ParallelCluster 2.9.1, reusing resources already in use by a previous cluster created with 2.7.0.
The reason to reuse the VPC, subnet and SG is to be able to mount the same EFS staying on the same AZ.
The new cluster should have 2 queues, one with spot nodes and the other with ondemand nodes. Both with c5a.12xlarge and c5.12xlarge instance types.
cloud-init.log shows script /var/lib/cloud/instance/scripts/part-002 failing with error. See lines 714 to 728 in cloud-init.log .
Environment:

AWS ParallelCluster 2.9.1
Configuration file attached config.txt

How to reproduce:
(apc-2.9.1) [centos@ip-172-31-6-186 ~]$ pcluster create -nr -c /home/centos/.parallelcluster/config-2.9.1-v1 HR-11
Beginning cluster creation for cluster: HR-11
Creating stack named: parallelcluster-HR-11
Status: parallelcluster-HR-11 - CREATE_FAILED
Cluster creation failed.  Failed events:
  - AWS::CloudFormation::Stack parallelcluster-HR-11 The following resource(s) failed to create: [MasterServerSubstack].
  - AWS::CloudFormation::Stack MasterServerSubstack Embedded stack arn:aws:cloudformation:eu-west-1:434355056914:stack/parallelcluster-HR-11-MasterServerSubstack-3HGXLXUYW8KU/3b2d3af0-f855-11ea-ba71-0adb030b5f58 was not successfully created: The following resource(s) failed to create: [MasterServer].
    - AWS::CloudFormation::Stack parallelcluster-HR-11-MasterServerSubstack-3HGXLXUYW8KU The following resource(s) failed to create: [MasterServer].
    - AWS::EC2::Instance MasterServer Received FAILURE signal with UniqueId i-0c9bb67914cb297d8

Attached log files:
messages.txt
cloud-init.log
Kind regards,
Richard

Hi Richard,
can you please provide /var/log/chef-client.log and /var/log/cfn-init.log from the head node? They are also available in CloudWatch if you prefer downloading them from there.
Francesco

Hi Franceso,
cfn-init.log
chef-client.log is not there.
[centos@ip-10-0-5-109 ~]$ ls -l /var/log
total 560
drwxr-xr-x. 2 root   root       37 Sep 14 22:55 amazon
drwxr-xr-x. 2 root   root        6 Sep 14 23:10 anaconda
drwx------. 2 root   root       23 Sep 16 19:47 audit
-rw-------  1 root   utmp        0 Sep 16 19:47 btmp
-rw-r--r--  1 root   root        0 Sep 16 19:48 cfn-init-cmd.log
-rw-r--r--  1 root   root     4400 Sep 16 19:54 cfn-init.log
-rw-r--r--  1 root   root      467 Sep 16 19:54 cfn-wire.log
drwxr-xr-x. 2 chrony chrony      6 Aug  8  2019 chrony
-rw-------  1 root   root   231211 Sep 17 15:40 cloud-init.log
-rw-------  1 root   root     2101 Sep 17 15:40 cron
drwxr-xr-x. 2 lp     sys        57 Sep 16 19:48 cups
drwxrwxrwt. 2 dcv    dcv         6 Aug 21 15:20 dcv
-rw-r--r--  1 root   root    31435 Sep 17 15:40 dmesg
-rw-r--r--  1 root   root    31626 Sep 16 19:47 dmesg.old
drwx--x--x. 2 root   gdm         6 Apr  1 02:41 gdm
drwxr-xr-x. 2 root   root        6 Apr  2 13:17 glusterfs
drwx------. 2 root   root        6 Apr  2 13:14 httpd
-rw-r--r--. 1 root   root   292292 Sep 17 15:41 lastlog
drwx------. 3 root   root       18 Sep 14 23:07 libvirt
drwxr-xr-x. 2 root   root        6 Sep 14 23:10 mail
-rw-------  1 root   root      378 Sep 17 15:40 maillog
-rw-r--r--  1 root   root   181039 Sep 17 15:41 messages
drwx------. 2 munge  root        6 Sep 14 22:56 munge
drwxr-xr-x. 3 root   root       33 Sep 14 22:48 nvidia
drwxr-xr-x. 3 root   root       18 Sep 14 23:07 pluto
drwxr-xr-x. 2 root   root        6 Aug  8  2019 qemu-ga
drwxr-xr-x. 2 root   root        6 Apr 22 09:10 rhsm
drwx------. 3 root   root       17 Sep 14 23:06 samba
-rw-------  1 root   root     9413 Sep 17 15:41 secure
drwx------. 2 root   root        6 Jun 10  2014 speech-dispatcher
drwxr-xr-x. 3 root   root       21 Sep 14 23:07 swtpm
drwxr-xr-x. 2 root   root       23 Sep 16 19:48 tuned
-rw-rw-r--  1 root   utmp     6912 Sep 17 15:41 wtmp
Regards,
Richard

Hi Richard,
Cluster creation issue
From the log files, your instance is not able to contact the CloudFormation AWS Endpoint.
The main reason could be that you're specifying use_public_ips = false and probably the subnets you're specifying don't have an internet gateway or a nat gateway, so the instances are not able to retrieve packages and data from the web.
As described in the networking documentation:

When use_public_ips is set to false, the VPC must be correctly set up to use the Proxy for all traffic. Web access is required for both head and compute nodes.

Is the master_subnet_id a public subnet (with an internet gateway) or a private one (with a nat gateway)?
I'd re-try by removing the use_public_ips configuration parameter.
Reusing resources created by another cluster
I see you're using a VPC security group from the old cluster (vpc_security_group_id).
This security group has been probably created within the old cluster.
You can reuse the same VPC and Subnets but we don't recommend to share resources like the Security Groups if they have been automatically created during cluster creation, because by associating them to another cluster you wont be able to remove the old cluster resources.
If you need to access the EFS from another subnet you can add another EFS mount target in the availability zone/subnet you want to use for the new cluster.
The same is valid for EFS resources.
If the EFS has been created with your previous cluster it will be a problem when you'll try to delete the old cluster.
How to share resources between clusters
The best way to share resources is to create them before cluster creation and pass them as configuration parameters.
E.g. : create a new EFS from the UI,  backup your old EFS content into a new one, and then associate it to the new cluster with efs_fs_id.
Let us know if it helps.

Hi Enrico,
your answer helped a lot and our problem is solved.
Setting  removing use_public_ips did it. I had wrongly let it unset on the previous cluster, thus getting compute nodes with public IPs that I did not want to have. Now on this new cluster I set it to false, but missed the corresponding gateway settings.
I created now a new SG and added it to the EFS, so that's fine now.
I think I've created the EFS by hand last time, but not sure. Where could I check that? In the old stack events or resources?
Many thanks for your help,
Richard

Hi Richard,
you can look at the CloudFormation console and look at the value of the EFSOptions input parameter of your old stack.
EFSOptions is a comma separated list of EFS related options and the second one corresponds to the value of the efs_fs_id parameter you used to create the cluster.
If it's valued (fs-xxx) it means you created the cluster by passing an existing EFS file system id.

Hi Enrico,
Perfect. Found it. The fs-xxx value is there, so I created the EFS myself. Good.
Thanks again for all your help, closing this now.
Kind regards,
Richard


Record the screen and record the computer microphone

Is your feature request related to a problem? Please describe.
When using the recording screen, I want to record the sound of the computer microphone at the same time. This is very useful when I record courses. I tried many screen recording software, but the color restoration of the recorded video is not very good (with color difference), and the color restoration of the video recorded with scrpy is the same as that of the mobile phone or tablet, but scrpy can't record the microphone sound of the computer, which is a great pity.

Refs #4044.


[BUG] - No action detected in task

Describe the bug
ERROR! no action detected in task. This often indicates a misspelled module name, or incorrect module path.

The error appears to have been in '~/.ansible/roles/ebbba.bigbluebutton/tasks/firewall/ufw_enable.yml': line 9, column 5, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  - name: Activate ufw
    ^ here

Hi @singpolyma,
I never encountered such an error and could not reproduce it. Can you try with the latest master commit?

CLOSED DUE TO INACTIVITY


WARNING: Tried to load angular more than once

This issue is created based on this comment.
I guess this is a build problem.

Now I'm getting "Uncaught ReferenceError: require is not defined" in Chrome 55 on MacOS.

Ok, now its fixed in version 1.2.9. If there is sill a problem you can reopen the issue...


Add folder for hashtables

hash table implementation in python

you can create it by yourself.

Hi, kindly add the hacktoberfest accepted tag.
Thanks
On Tue, Oct 4, 2022 at 4:52 PM Gantavya Malviya @.***>
wrote:

Closed #302
https://github.com/codemistic/Data-Structures-and-Algorithms/issues/302
as completed.
—
Reply to this email directly, view it on GitHub
https://github.com/codemistic/Data-Structures-and-Algorithms/issues/302#event-7515392147,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASVK5JGCWT5Q6KDB5DEEQVLWBQHPHANCNFSM6AAAAAAQ3Y5PIQ
.
You are receiving this because you were assigned.Message ID:
<codemistic/Data-Structures-and-Algorithms/issue/302/issue_event/7515392147
@github.com>

Please merge to main as well.
On Tue, Oct 4, 2022 at 6:41 PM soham datta @.***> wrote:

Hi, kindly add the hacktoberfest accepted tag.
Thanks
On Tue, Oct 4, 2022 at 4:52 PM Gantavya Malviya @.***>
wrote:

Closed #302
https://github.com/codemistic/Data-Structures-and-Algorithms/issues/302
as completed.
—
Reply to this email directly, view it on GitHub
https://github.com/codemistic/Data-Structures-and-Algorithms/issues/302#event-7515392147,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ASVK5JGCWT5Q6KDB5DEEQVLWBQHPHANCNFSM6AAAAAAQ3Y5PIQ
.
You are receiving this because you were assigned.Message ID:
<codemistic/Data-Structures-and-Algorithms/issue/302/issue_event/7515392147
@github.com>


Use columns for properties when dumping collections?

One of the handy things with LinqPad .Dump() is when you use it on a collection, you get a table with columns for each property. e.g. in LinqPad the code:
void Main()
{
	var people = new[]
	{
		new Person("Marie", "Curie"),
		new Person("Albert", "Einstein"),
		new Person("Ada", "Lovelace"),
		new Person("Cleopatra", "Philopator"),
		new Person("Nikola", "Tesla")
	};

	people.Dump();
}

record Person(string FirstName, string LastName);

gives:

but with Dumpify, it shows:

So I'd like to be able to have the more concise column-for-property format with collections.

Hi @wilka, this feature is already supported but it is turned-off be default. It can be turned-on either globally for all dumps or per-dump:
Globally:
DumpConfig.Default.TableConfig.ShowRowSeparators = true;
Per dump:
people.Dump(tableConfig: new TableConfig { ShowRowSeparators = true });
Turning this option on results in the following output:

If it makes any difference, I'm not hugely concerned about the terminal width limits. I tend to run my Windows Terminal at full screen, and I have a large display.  But I realise that won't be the case for everyone.
I've been using Dumpify as a stepping stone from taking some LinqPad prototype code, and then turning it into a console app.