modin.pandas.merge  MemoryError  while pandas  works file

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS 7.6.1810
Modin version (modin.__version__):   0.14.1
Python version:  3.7.6
Ray version:   1.12.1 ( Ray cluster: 1 head node + 2 worker nodes,   32G mem/node)
**Csv file info **:  500w50f.csv: 5million rows * 50 cols, 3G, ,   1wID.csv: 10000 rows * 1 col,  0.3M
Code we can use to reproduce:

def pandas_merge_df(csv_file_path1:str, csv_file_path2:str):
    df1 = pd.read_csv(csv_file_path1)
    print(f'pandas_merge df1 shape = {df1.shape}')
    df2 = pd.read_csv(csv_file_path2)
    print(f'pandas_merge df2 shape = {df2.shape}')

    print('pandas DF merge start ...')
    start_time = time.time()
    df_merged = pd.merge(df1,df2, left_on=df1.idX, right_on=df2.idX, how='inner')
    print(f'pandas_merge df_merged shape = {df_merged.shape}')
    print(f'>>>Time costs(s) - pandas_merge_df:  {time.time() - start_time}')

@ray.remote
def modin_merge_df(csv_file_path1:str, csv_file_path2:str):
    df1 = mpd.read_csv(csv_file_path1)
    print(f'modin_merge df1 shape = {df1.shape}')
    df2 = mpd.read_csv(csv_file_path2)
    print(f'modin_merge df2 shape = {df2.shape}')

    print('modin DF merge start ...')
    start_time = time.time()
    df_merged = mpd.merge(df1,df2, on='idX', how='inner')
    print(f'modin_merge df_merged shape = {df_merged.shape}')
    print(f'>>>Time costs(s) - modin_merge_df:  {time.time() - start_time}')

if __name__ == '__main__':
    print('''This cluster consists of
    {} nodes in total
    {} CPU resources in total
    '''.format(len(ray.nodes()), ray.cluster_resources()['CPU']))

    pandas_merge_df('1wID.csv', '500w50f.csv')
    ray.get(modin_merge_df.remote('/home/term/wanlu/1wID.csv', '/home/term/wanlu/500w50f.csv'))

Describe the problem

Hi modin team,  I'm comparing the performance of  merge()  with modin & pandas.
mpd.merge()  report memory errors  when merging 10000 rows * 1 col  data frame with 5 million rows * 50 col data frame on 3 ray nodes, while pandas works fine as you can see from the output log.     I  ran the script on the head node.   And by top command I can see memory  & swap usage increase to  nearly 100% (head node).    But the other 2 nodes  loads seems not high.
When mpd.merge()  of 10000 rows * 1 col  data frame with 1 million rows * 50 col data frame ,   it works fine and with no memory error.    And in  this case,  to my surprise, pandas runs faster than modin.
I think modin should work well as least as Pandas,  and faster,  right?   I've found many goods with modin.  Many thanks.
Source code / logs

[term@dev-ctb-xs-196-65 wanlu]$ python3 minray.py
This cluster consists of
3 nodes in total
24.0 CPU resources in total
pandas_merge df1 shape = (10000, 1)
pandas_merge df2 shape = (5000000, 51)
pandas DF merge start ...
pandas_merge df_merged shape = (10000, 53)

Time costs(s) - pandas_merge_df:  6.965200901031494
(modin_merge_df pid=1062) modin_merge df1 shape = (10000, 1)
(modin_merge_df pid=1062) modin_merge df2 shape = (5000000, 51)
(modin_merge_df pid=1062) modin DF merge start ...
2022-05-30 15:27:53,805 WARNING worker.py:1382 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: 10a47da8b2e06334833f8d662c87aac5abe16ad518000000 Worker ID: 6088f2b078879412460290eb3198dc1cb80482ce9293bb5ab24a9014 Node ID: c7163f4413b122015ee32a308507a1612fcca45486c61759e8bd2b96 Worker IP address: <IP_ADDRESS> Worker port: 10194 Worker PID: 1309
2022-05-30 15:27:57,592 WARNING worker.py:1382 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: 5a4bb0f1c88e007a2b219d5690578c4a1d4001de18000000 Worker ID: 39ef4a6ea7cbb913d52bd525f109d3babb3c900762f5637f482c1189 Node ID: c7163f4413b122015ee32a308507a1612fcca45486c61759e8bd2b96 Worker IP address: <IP_ADDRESS> Worker port: 10192 Worker PID: 1116
2022-05-30 15:28:05,204 WARNING worker.py:1382 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: b9914c515693f9ba21122f58a97c0e82b25dad5318000000 Worker ID: ccc4ab1bad8bc0d5c2e319378a8d1d7e0a213943171e42a62d99a016 Node ID: c7163f4413b122015ee32a308507a1612fcca45486c61759e8bd2b96 Worker IP address: <IP_ADDRESS> Worker port: 10190 Worker PID: 1113
Traceback (most recent call last):
File "minray.py", line 170, in
ray.get(modin_merge_df.remote('/home/term/wanlu/1wID.csv', '/home/term/wanlu/500w50f.csv'))
File "/data/program/python3.8/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/data/program/python3.8/lib/python3.8/site-packages/ray/worker.py", line 1809, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(MemoryError): ray::modin_merge_df() (pid=1062, ip=<IP_ADDRESS>)
File "minray.py", line 136, in modin_merge_df
df_merged = mpd.merge(df1,df2, on='idX', how='inner')
File "/home/term/.local/lib/python3.8/site-packages/modin/pandas/general.py", line 89, in merge
return left.merge(
File "/home/term/.local/lib/python3.8/site-packages/modin/pandas/dataframe.py", line 1404, in merge
query_compiler=self._query_compiler.merge(
File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 458, in merge
self._modin_frame.apply_full_axis(1, map_func)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 114, in run_f_on_minimally_updated_metadata
result = f(self, *args, **kwargs)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1897, in apply_full_axis
return self.broadcast_apply_full_axis(
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 114, in run_f_on_minimally_updated_metadata
result = f(self, *args, **kwargs)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 2324, in broadcast_apply_full_axis
new_axes = [
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 2325, in
self._compute_axis_labels(i, new_partitions)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 420, in _compute_axis_labels
return self._partition_mgr_cls.get_indices(
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/partition_manager.py", line 837, in get_indices
new_idx = cls.get_objects_from_partitions(new_idx)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition_manager.py", line 112, in get_objects_from_partitions
return ray.get([partition.oid for partition in partitions])
ray.exceptions.RayTaskError(MemoryError): ray::_apply_func() (pid=1884, ip=<IP_ADDRESS>)
At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::deploy_ray_func() (pid=1888, ip=<IP_ADDRESS>)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
result = func(*args)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
result = func(dataframe, **kwargs)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1384, in _tree_reduce_func
series_result = func(df, *args, **kwargs)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 455, in map_func
return pandas.merge(left, right, **kwargs)
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 107, in merge
op = _MergeOperation(
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 700, in init
) = self._get_merge_keys()
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1155, in _get_merge_keys
self.right = self.right._drop_labels_or_levels(right_drop)
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 1910, in _drop_labels_or_levels
dropped = self.copy()
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6032, in copy
data = self._mgr.copy(deep=deep)
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 615, in copy
res._consolidate_inplace()
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1685, in _consolidate_inplace
self.blocks = tuple(_consolidate(self.blocks))
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2084, in _consolidate
merged_blocks = _merge_blocks(
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2111, in _merge_blocks
new_values = np.vstack([b.values for b in blocks])  # type: ignore[misc]
File "<array_function internals>", line 180, in vstack
File "/home/term/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 282, in vstack
return _nx.concatenate(arrs, 0)
File "<array_function internals>", line 180, in concatenate
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.86 GiB for an array with shape (50, 5000000) and data type float64
(modin_merge_df pid=1062) 2022-05-30 15:28:05,964       ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::_apply_func() (pid=1884, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   At least one of the input arguments for this task could not be computed:
(modin_merge_df pid=1062) ray.exceptions.RayTaskError: ray::deploy_ray_func() (pid=1888, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
(modin_merge_df pid=1062)     result = func(*args)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
(modin_merge_df pid=1062)     result = func(dataframe, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1384, in _tree_reduce_func
(modin_merge_df pid=1062)     series_result = func(df, *args, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 455, in map_func
(modin_merge_df pid=1062)     return pandas.merge(left, right, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 107, in merge
(modin_merge_df pid=1062)     op = _MergeOperation(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 700, in init
(modin_merge_df pid=1062)     ) = self._get_merge_keys()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1155, in _get_merge_keys
(modin_merge_df pid=1062)     self.right = self.right._drop_labels_or_levels(right_drop)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 1910, in _drop_labels_or_levels
(modin_merge_df pid=1062)     dropped = self.copy()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6032, in copy
(modin_merge_df pid=1062)     data = self._mgr.copy(deep=deep)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 615, in copy
(modin_merge_df pid=1062)     res._consolidate_inplace()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1685, in _consolidate_inplace
(modin_merge_df pid=1062)     self.blocks = tuple(_consolidate(self.blocks))
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2084, in _consolidate
(modin_merge_df pid=1062)     merged_blocks = _merge_blocks(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2111, in _merge_blocks
(modin_merge_df pid=1062)     new_values = np.vstack([b.values for b in blocks])  # type: ignore[misc]
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in vstack
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 282, in vstack
(modin_merge_df pid=1062)     return _nx.concatenate(arrs, 0)
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in concatenate
(modin_merge_df pid=1062) numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.86 GiB for an array with shape (50, 5000000) and data type float64
(modin_merge_df pid=1062) 2022-05-30 15:28:05,966       ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::_apply_func() (pid=1884, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   At least one of the input arguments for this task could not be computed:
(modin_merge_df pid=1062) ray.exceptions.RayTaskError: ray::deploy_ray_func() (pid=1888, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
(modin_merge_df pid=1062)     result = func(*args)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
(modin_merge_df pid=1062)     result = func(dataframe, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1384, in _tree_reduce_func
(modin_merge_df pid=1062)     series_result = func(df, *args, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 455, in map_func
(modin_merge_df pid=1062)     return pandas.merge(left, right, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 107, in merge
(modin_merge_df pid=1062)     op = _MergeOperation(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 700, in init
(modin_merge_df pid=1062)     ) = self._get_merge_keys()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1155, in _get_merge_keys
(modin_merge_df pid=1062)     self.right = self.right._drop_labels_or_levels(right_drop)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 1910, in _drop_labels_or_levels
(modin_merge_df pid=1062)     dropped = self.copy()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6032, in copy
(modin_merge_df pid=1062)     data = self._mgr.copy(deep=deep)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 615, in copy
(modin_merge_df pid=1062)     res._consolidate_inplace()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1685, in _consolidate_inplace
(modin_merge_df pid=1062)     self.blocks = tuple(_consolidate(self.blocks))
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2084, in _consolidate
(modin_merge_df pid=1062)     merged_blocks = _merge_blocks(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2111, in _merge_blocks
(modin_merge_df pid=1062)     new_values = np.vstack([b.values for b in blocks])  # type: ignore[misc]
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in vstack
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 282, in vstack
(modin_merge_df pid=1062)     return _nx.concatenate(arrs, 0)
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in concatenate
(modin_merge_df pid=1062) numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.86 GiB for an array with shape (50, 5000000) and data type float64
(modin_merge_df pid=1062) 2022-05-30 15:28:05,966       ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::_apply_func() (pid=1884, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   At least one of the input arguments for this task could not be computed:
(modin_merge_df pid=1062) ray.exceptions.RayTaskError: ray::deploy_ray_func() (pid=1888, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
(modin_merge_df pid=1062)     result = func(*args)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
(modin_merge_df pid=1062)     result = func(dataframe, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1384, in _tree_reduce_func
(modin_merge_df pid=1062)     series_result = func(df, *args, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 455, in map_func
(modin_merge_df pid=1062)     return pandas.merge(left, right, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 107, in merge
(modin_merge_df pid=1062)     op = _MergeOperation(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 700, in init
(modin_merge_df pid=1062)     ) = self._get_merge_keys()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1155, in _get_merge_keys
(modin_merge_df pid=1062)     self.right = self.right._drop_labels_or_levels(right_drop)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 1910, in _drop_labels_or_levels
(modin_merge_df pid=1062)     dropped = self.copy()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6032, in copy
(modin_merge_df pid=1062)     data = self._mgr.copy(deep=deep)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 615, in copy
(modin_merge_df pid=1062)     res._consolidate_inplace()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1685, in _consolidate_inplace
(modin_merge_df pid=1062)     self.blocks = tuple(_consolidate(self.blocks))
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2084, in _consolidate
(modin_merge_df pid=1062)     merged_blocks = _merge_blocks(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2111, in _merge_blocks
(modin_merge_df pid=1062)     new_values = np.vstack([b.values for b in blocks])  # type: ignore[misc]
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in vstack
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 282, in vstack
(modin_merge_df pid=1062)     return _nx.concatenate(arrs, 0)
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in concatenate
(modin_merge_df pid=1062) numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.86 GiB for an array with shape (50, 5000000) and data type float64
(modin_merge_df pid=1062) 2022-05-30 15:28:05,967       ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::deploy_ray_func() (pid=1884, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
(modin_merge_df pid=1062)     result = func(*args)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
(modin_merge_df pid=1062)     result = func(dataframe, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1384, in _tree_reduce_func
(modin_merge_df pid=1062)     series_result = func(df, *args, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 455, in map_func
(modin_merge_df pid=1062)     return pandas.merge(left, right, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 107, in merge
(modin_merge_df pid=1062)     op = _MergeOperation(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 700, in init
(modin_merge_df pid=1062)     ) = self._get_merge_keys()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1155, in _get_merge_keys
(modin_merge_df pid=1062)     self.right = self.right._drop_labels_or_levels(right_drop)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 1910, in _drop_labels_or_levels
(modin_merge_df pid=1062)     dropped = self.copy()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6032, in copy
(modin_merge_df pid=1062)     data = self._mgr.copy(deep=deep)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 615, in copy
(modin_merge_df pid=1062)     res._consolidate_inplace()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1685, in _consolidate_inplace
(modin_merge_df pid=1062)     self.blocks = tuple(_consolidate(self.blocks))
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2084, in _consolidate
(modin_merge_df pid=1062)     merged_blocks = _merge_blocks(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2111, in _merge_blocks
(modin_merge_df pid=1062)     new_values = np.vstack([b.values for b in blocks])  # type: ignore[misc]
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in vstack
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 282, in vstack
(modin_merge_df pid=1062)     return _nx.concatenate(arrs, 0)
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in concatenate
(modin_merge_df pid=1062) numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.86 GiB for an array with shape (50, 5000000) and data type float64
(modin_merge_df pid=1062) 2022-05-30 15:28:05,967       ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::deploy_ray_func() (pid=1884, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
(modin_merge_df pid=1062)     result = func(*args)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
(modin_merge_df pid=1062)     result = func(dataframe, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1384, in _tree_reduce_func
(modin_merge_df pid=1062)     series_result = func(df, *args, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 455, in map_func
(modin_merge_df pid=1062)     return pandas.merge(left, right, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 107, in merge
(modin_merge_df pid=1062)     op = _MergeOperation(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 700, in init
(modin_merge_df pid=1062)     ) = self._get_merge_keys()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1155, in _get_merge_keys
(modin_merge_df pid=1062)     self.right = self.right._drop_labels_or_levels(right_drop)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 1910, in _drop_labels_or_levels
(modin_merge_df pid=1062)     dropped = self.copy()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6032, in copy
(modin_merge_df pid=1062)     data = self._mgr.copy(deep=deep)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 615, in copy
(modin_merge_df pid=1062)     res._consolidate_inplace()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1685, in _consolidate_inplace
(modin_merge_df pid=1062)     self.blocks = tuple(_consolidate(self.blocks))
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2084, in _consolidate
(modin_merge_df pid=1062)     merged_blocks = _merge_blocks(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2111, in _merge_blocks
(modin_merge_df pid=1062)     new_values = np.vstack([b.values for b in blocks])  # type: ignore[misc]
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in vstack
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 282, in vstack
(modin_merge_df pid=1062)     return _nx.concatenate(arrs, 0)
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in concatenate
(modin_merge_df pid=1062) numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.86 GiB for an array with shape (50, 5000000) and data type float64
(modin_merge_df pid=1062) 2022-05-30 15:28:05,967       ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::deploy_ray_func() (pid=1884, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
(modin_merge_df pid=1062)     result = func(*args)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
(modin_merge_df pid=1062)     result = func(dataframe, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1384, in _tree_reduce_func
(modin_merge_df pid=1062)     series_result = func(df, *args, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 455, in map_func
(modin_merge_df pid=1062)     return pandas.merge(left, right, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 107, in merge
(modin_merge_df pid=1062)     op = _MergeOperation(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 700, in init
(modin_merge_df pid=1062)     ) = self._get_merge_keys()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1155, in _get_merge_keys
(modin_merge_df pid=1062)     self.right = self.right._drop_labels_or_levels(right_drop)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 1910, in _drop_labels_or_levels
(modin_merge_df pid=1062)     dropped = self.copy()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6032, in copy
(modin_merge_df pid=1062)     data = self._mgr.copy(deep=deep)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 615, in copy
(modin_merge_df pid=1062)     res._consolidate_inplace()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1685, in _consolidate_inplace
(modin_merge_df pid=1062)     self.blocks = tuple(_consolidate(self.blocks))
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2084, in _consolidate
(modin_merge_df pid=1062)     merged_blocks = _merge_blocks(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2111, in _merge_blocks
(modin_merge_df pid=1062)     new_values = np.vstack([b.values for b in blocks])  # type: ignore[misc]
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in vstack
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 282, in vstack
(modin_merge_df pid=1062)     return _nx.concatenate(arrs, 0)
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in concatenate
(modin_merge_df pid=1062) numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.86 GiB for an array with shape (50, 5000000) and data type float64
(modin_merge_df pid=1062) 2022-05-30 15:28:05,968       ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::deploy_ray_func() (pid=1888, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
(modin_merge_df pid=1062)     result = func(*args)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
(modin_merge_df pid=1062)     result = func(dataframe, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1384, in _tree_reduce_func
(modin_merge_df pid=1062)     series_result = func(df, *args, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 455, in map_func
(modin_merge_df pid=1062)     return pandas.merge(left, right, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 107, in merge
(modin_merge_df pid=1062)     op = _MergeOperation(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 700, in init
(modin_merge_df pid=1062)     ) = self._get_merge_keys()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1155, in _get_merge_keys
(modin_merge_df pid=1062)     self.right = self.right._drop_labels_or_levels(right_drop)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 1910, in _drop_labels_or_levels
(modin_merge_df pid=1062)     dropped = self.copy()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6032, in copy
(modin_merge_df pid=1062)     data = self._mgr.copy(deep=deep)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 615, in copy
(modin_merge_df pid=1062)     res._consolidate_inplace()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1685, in _consolidate_inplace
(modin_merge_df pid=1062)     self.blocks = tuple(_consolidate(self.blocks))
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2084, in _consolidate
(modin_merge_df pid=1062)     merged_blocks = _merge_blocks(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2111, in _merge_blocks
(modin_merge_df pid=1062)     new_values = np.vstack([b.values for b in blocks])  # type: ignore[misc]
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in vstack
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 282, in vstack
(modin_merge_df pid=1062)     return _nx.concatenate(arrs, 0)
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in concatenate
(modin_merge_df pid=1062) numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.86 GiB for an array with shape (50, 5000000) and data type float64
(modin_merge_df pid=1062) 2022-05-30 15:28:05,968       ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::deploy_ray_func() (pid=1888, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
(modin_merge_df pid=1062)     result = func(*args)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
(modin_merge_df pid=1062)     result = func(dataframe, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1384, in _tree_reduce_func
(modin_merge_df pid=1062)     series_result = func(df, *args, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 455, in map_func
(modin_merge_df pid=1062)     return pandas.merge(left, right, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 107, in merge
(modin_merge_df pid=1062)     op = _MergeOperation(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 700, in init
(modin_merge_df pid=1062)     ) = self._get_merge_keys()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1155, in _get_merge_keys
(modin_merge_df pid=1062)     self.right = self.right._drop_labels_or_levels(right_drop)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 1910, in _drop_labels_or_levels
(modin_merge_df pid=1062)     dropped = self.copy()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6032, in copy
(modin_merge_df pid=1062)     data = self._mgr.copy(deep=deep)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 615, in copy
(modin_merge_df pid=1062)     res._consolidate_inplace()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1685, in _consolidate_inplace
(modin_merge_df pid=1062)     self.blocks = tuple(_consolidate(self.blocks))
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2084, in _consolidate
(modin_merge_df pid=1062)     merged_blocks = _merge_blocks(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2111, in _merge_blocks
(modin_merge_df pid=1062)     new_values = np.vstack([b.values for b in blocks])  # type: ignore[misc]
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in vstack
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 282, in vstack
(modin_merge_df pid=1062)     return _nx.concatenate(arrs, 0)
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in concatenate
(modin_merge_df pid=1062) numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.86 GiB for an array with shape (50, 5000000) and data type float64
(modin_merge_df pid=1062) 2022-05-30 15:28:05,969       ERROR worker.py:92 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): ray::deploy_ray_func() (pid=1888, ip=<IP_ADDRESS>)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
(modin_merge_df pid=1062)     result = func(*args)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
(modin_merge_df pid=1062)     result = func(dataframe, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1384, in _tree_reduce_func
(modin_merge_df pid=1062)     series_result = func(df, *args, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 455, in map_func
(modin_merge_df pid=1062)     return pandas.merge(left, right, **kwargs)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 107, in merge
(modin_merge_df pid=1062)     op = _MergeOperation(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 700, in init
(modin_merge_df pid=1062)     ) = self._get_merge_keys()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1155, in _get_merge_keys
(modin_merge_df pid=1062)     self.right = self.right._drop_labels_or_levels(right_drop)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 1910, in _drop_labels_or_levels
(modin_merge_df pid=1062)     dropped = self.copy()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6032, in copy
(modin_merge_df pid=1062)     data = self._mgr.copy(deep=deep)
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 615, in copy
(modin_merge_df pid=1062)     res._consolidate_inplace()
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1685, in _consolidate_inplace
(modin_merge_df pid=1062)     self.blocks = tuple(_consolidate(self.blocks))
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2084, in _consolidate
(modin_merge_df pid=1062)     merged_blocks = _merge_blocks(
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2111, in _merge_blocks
(modin_merge_df pid=1062)     new_values = np.vstack([b.values for b in blocks])  # type: ignore[misc]
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in vstack
(modin_merge_df pid=1062)   File "/home/term/.local/lib/python3.8/site-packages/numpy/core/shape_base.py", line 282, in vstack
(modin_merge_df pid=1062)     return _nx.concatenate(arrs, 0)
(modin_merge_df pid=1062)   File "<array_function internals>", line 180, in concatenate
(modin_merge_df pid=1062) numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.86 GiB for an array with shape (50, 5000000) and data type float64

@lionstory thank you for reporting your issue. I have tried to reproduce the Memory error with some dataframes of the dimensions you gave, but I'm not getting it. Are you able to provide a complete script, including code to generate the data or files with the data, that I could run on my own machine? I don't have the CSVs you are using to get the data for the merge.
Modin merge is known to have poor performance in many cases. See #4293 and #656. It doesn't look like those should apply in this case, where we are doing an inner join with on, but maybe they will be helpful:
df_merged = mpd.merge(df1,df2, on='idX', how='inner')

I think this is not a performance related issue. It is rather related to available memory resources and the way of Modin usage. I corrected the provided script to be pythonic and as I see from it @lionstory is doing a ray remote operation inside of which the ray remote operations are performed by Modin internals for merge and read_csv. @lionstory, you shouldn't be doing ray operations yourself, Modin will take care of it. read_csv and merge will be executed in parallel. Could you just remove @ray.remote decorator for modin_merge_df and run the function as a regular python function. That would look like this.
def pandas_merge_df(csv_file_path1:str, csv_file_path2:str):
    df1 = pd.read_csv(csv_file_path1)
    print(f'pandas_merge df1 shape = {df1.shape}')
    df2 = pd.read_csv(csv_file_path2)
    print(f'pandas_merge df2 shape = {df2.shape}')

    print('pandas DF merge start ...')
    start_time = time.time()
    df_merged = pd.merge(df1,df2, left_on=df1.idX, right_on=df2.idX, how='inner')
    print(f'pandas_merge df_merged shape = {df_merged.shape}')
    print(f'>>>Time costs(s) - pandas_merge_df:  {time.time() - start_time}')

def modin_merge_df(csv_file_path1:str, csv_file_path2:str):
    df1 = mpd.read_csv(csv_file_path1)
    print(f'modin_merge df1 shape = {df1.shape}')
    df2 = mpd.read_csv(csv_file_path2)
    print(f'modin_merge df2 shape = {df2.shape}')

    print('modin DF merge start ...')
    start_time = time.time()
    df_merged = mpd.merge(df1,df2, on='idX', how='inner')
    print(f'modin_merge df_merged shape = {df_merged.shape}')
    print(f'>>>Time costs(s) - modin_merge_df:  {time.time() - start_time}')

if __name__ == '__main__':
    print('''This cluster consists of
    {} nodes in total
    {} CPU resources in total
    '''.format(len(ray.nodes()), ray.cluster_resources()['CPU']))

    pandas_merge_df('1wID.csv', '500w50f.csv')
    modin_merge_df('/home/term/wanlu/1wID.csv', '/home/term/wanlu/500w50f.csv')

Thanks responses  above.    @YarShev  I  remove Ray.remote/get,   and reran the codes,   the output run as below.  And the mem & swap  both increased to 100%  by top command.
[term@dev-ctb-xs-196-65 wanlu]$ python3  minray.py
This cluster consists of
3 nodes in total
24.0 CPU resources in total
pandas_merge df1 shape = (10000, 1)
pandas_merge df2 shape = (5000000, 51)
pandas DF merge start ...
pandas_merge df_merged shape = (10000, 53)

Time costs(s) - pandas_merge_df:  7.0381903648376465
modin_merge df1 shape = (10000, 1)
modin_merge df2 shape = (5000000, 51)
modin DF merge start ...
(raylet) Spilled 4580 MiB, 61 objects, write throughput 25 MiB/s.
2022-06-02 16:21:21,092        WARNING worker.py:1382 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: b293882dde835a4597ee440c1499be6cf97bb2331a000000 Worker ID: f9514bcd73c835cc8d3dadf2d26b6cb538d5f66d131d5c2a1f11e122 Node ID: c7163f4413b122015ee32a308507a1612fcca45486c61759e8bd2b96 Worker IP address: <IP_ADDRESS> Worker port: 10224 Worker PID: 11455
2022-06-02 16:21:26,847 WARNING worker.py:1382 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: e177633da1bed53aa8a704944f4c0482e40e40641a000000 Worker ID: ed73c1ecb0db55859fc394901e07d6d261647f34971460542ffeee5f Node ID: c7163f4413b122015ee32a308507a1612fcca45486c61759e8bd2b96 Worker IP address: <IP_ADDRESS> Worker port: 10222 Worker PID: 11314
2022-06-02 16:21:43,815 WARNING worker.py:1382 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: fcdcc85242320b54127779664fd7b679a79f52d01a000000 Worker ID: 22e9c1630d92742ca2b34fc9367016be42688fa7af4dec7dd9130a7e Node ID: c7163f4413b122015ee32a308507a1612fcca45486c61759e8bd2b96 Worker IP address: <IP_ADDRESS> Worker port: 10226 Worker PID: 12004
2022-06-02 16:21:48,044 WARNING worker.py:1382 -- A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: f969278a8a916934fa0af750898da760cab72e1b1a000000 Worker ID: f29674a1f6548a0ddda39a891637db23b2ea8994b2ef9f9d1788365a Node ID: c7163f4413b122015ee32a308507a1612fcca45486c61759e8bd2b96 Worker IP address: <IP_ADDRESS> Worker port: 10228 Worker PID: 12005
Traceback (most recent call last):
File "minray.py", line 170, in
modin_merge_df('/home/term/wanlu/1wID.csv', '/home/term/wanlu/500w50f.csv')
File "minray.py", line 136, in modin_merge_df
df_merged = mpd.merge(df1,df2, on='idX', how='inner')
File "/home/term/.local/lib/python3.8/site-packages/modin/pandas/general.py", line 89, in merge
return left.merge(
File "/home/term/.local/lib/python3.8/site-packages/modin/pandas/dataframe.py", line 1404, in merge
query_compiler=self._query_compiler.merge(
File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 458, in merge
self._modin_frame.apply_full_axis(1, map_func)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 114, in run_f_on_minimally_updated_metadata
result = f(self, *args, **kwargs)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1897, in apply_full_axis
return self.broadcast_apply_full_axis(
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 114, in run_f_on_minimally_updated_metadata
result = f(self, *args, **kwargs)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 2324, in broadcast_apply_full_axis
new_axes = [
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 2325, in
self._compute_axis_labels(i, new_partitions)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 420, in _compute_axis_labels
return self._partition_mgr_cls.get_indices(
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/partition_manager.py", line 837, in get_indices
new_idx = cls.get_objects_from_partitions(new_idx)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/partition_manager.py", line 112, in get_objects_from_partitions
return ray.get([partition.oid for partition in partitions])
File "/data/program/python3.8/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/data/program/python3.8/lib/python3.8/site-packages/ray/worker.py", line 1809, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(MemoryError): ray::_apply_func() (pid=12146, ip=<IP_ADDRESS>)
At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::deploy_ray_func() (pid=12005, ip=<IP_ADDRESS>)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/execution/ray/implementations/pandas_on_ray/partitioning/virtual_partition.py", line 497, in deploy_ray_func
result = func(*args)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/partitioning/axis_partition.py", line 160, in deploy_axis_func
result = func(dataframe, **kwargs)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/dataframe/pandas/dataframe/dataframe.py", line 1384, in _tree_reduce_func
series_result = func(df, *args, **kwargs)
File "/home/term/.local/lib/python3.8/site-packages/modin/core/storage_formats/pandas/query_compiler.py", line 455, in map_func
return pandas.merge(left, right, **kwargs)
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 107, in merge
op = _MergeOperation(
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 700, in init
) = self._get_merge_keys()
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/reshape/merge.py", line 1155, in _get_merge_keys
self.right = self.right._drop_labels_or_levels(right_drop)
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 1910, in _drop_labels_or_levels
dropped = self.copy()
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 6032, in copy
data = self._mgr.copy(deep=deep)
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 615, in copy
res._consolidate_inplace()
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1685, in _consolidate_inplace
self.blocks = tuple(_consolidate(self.blocks))
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2084, in _consolidate
merged_blocks = _merge_blocks(
File "/home/term/.local/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 2118, in _merge_blocks
new_values = new_values[argsort]
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 1.86 GiB for an array with shape (50, 5000000) and data type float64

@lionstory, I am still tended to think this is a memory related issue. First, I would recommend you to take a look at the optimization notes section regarding the merge operation to understand the underlying logic and the performance and memory implications. If that doesn't help, could you run the script on a cluster with a bit more memory resources?

@lionstory because we can't verify whether this issue is still relevant, and it's been over 3 months since @YarShev asked you for more information, I'm going to close this issue. Please comment here if you have more questions or would like any help from the Modin contributors. We're happy to help.


RemoteInspection: clean up CMakeLists.txt slightly (NFC)

Inline single use variable, inline CPP definition, and reflow some text.

CC: @Azoy @etcwilde

@swift-ci please test


ProductCatalog release name is limited to 7 characters

This PR fixes issue #306
The port names of all v1beta3 components in the testData folder were changed,
so that the release name is no longer contained.
The release name can now be 32 characters.

Tests are failing (locally executed tests had no errors...)
https://reports.cucumber.io/reports/bb7313a2-8781-4b5a-9f57-1ab9e9d410e2

Maybe it is related to the preinstalled components which come from the repo and not from the oda-canvas github repo.

I rerun the tests without having changed anything and now the tests are green:
https://reports.cucumber.io/reports/f4ffcdb1-e267-4e47-8ecd-1a116e641f8e

Found further locations:

feature-definition-and-test-kit/testData/productcatalog-enhanced-v1beta3/templates/deployment-promotionmanagementapi.yaml line 43
feature-definition-and-test-kit/testData/productcatalog-enhanced-v1beta3/templates/service-promotionmanagementapi.yaml line 12
feature-definition-and-test-kit/testData/productcatalog-v1alpha4/templates/deployment-metricsapi.yaml line 29
feature-definition-and-test-kit/testData/productcatalog-v1alpha4/templates/deployment-mongodb.yaml line 23
feature-definition-and-test-kit/testData/productcatalog-v1alpha4/templates/deployment-partyroleapi.yaml
feature-definition-and-test-kit/testData/productcatalog-v1alpha4/templates/deployment-productcatalogapi.yaml
feature-definition-and-test-kit/testData/productcatalog-v1alpha4/templates/service-mongodb.yaml
feature-definition-and-test-kit/testData/productcatalog-v1alpha4/templates/service-partyroleapi.yaml
feature-definition-and-test-kit/testData/productcatalog-v1alpha4/templates/service-productcatalogapi.yaml
feature-definition-and-test-kit/testData/productcatalog-v1alpha4/templates/service-registerallevents.yaml

Also in feature-definition-and-test-kit/testData/productcatalog-v1beta2/templates and feature-definition-and-test-kit/testData/productinventory-v1beta3/templates

I only changed the yamls for the v1beta3 components, because these are the ones which are further developed.
v1alpha4 and v1beta2 are left unchanged.


Add React Native to CI examples

This PR:

[X] Adds React Native example to CI workflow.

thanks!


[cinder-csi-plugin] Example of Cinder CSI PVC Recovery

What this PR does / why we need it:
documentation example
Which issue this PR fixes(if applicable):
fixes #1141

/ok-to-test

@eumel8 please help to update your release note, thanks

@eumel8 please help to update your release note, thanks

@jichenjc could you point me to release note or changelog?

add
NONE

at the end of your first comment (use edit) then you should fix the problem

add
NONE

at the end of your first comment (use edit) then you should fix the problem

done

@eumel8 Thanks for the PR. Have you tested the same in your env? I saw couple of bugs before volume doesnt get attached to diff node , so just checking

yes, of course, it works for me as described

/lgtm
I think we might need add more detailed steps (in case someone else use this) overtime but so far it's a good start

/approve


Failed to generate native debug information for optimized C code with debug info about dead functions

I've been attempting to debug some C code recently but unfortunately I keep running into an error that indicates "failed to emit DWARF debug information" when passing the -g flag to the wasmtime CLI. The reproduction of this is:
// foo.c
__attribute__((export_name("foo")))
int foo() { return 3; }

int bar(int a) {
  int b[50];
  b[0] = a;
  b[29] = a;
  return a;
}

$  clang -O3 foo.c -o foo.wasm -mexec-model=reactor -g
$ wasmtime -g ./foo.wasm
Error: failed to run main module `./foo.wasm`

Caused by:
    0: failed to emit DWARF debug information
    1: The end offset of a location list entry must not be before the beginning.

Inspecting the results of llvm-dwarfdump I see:

foo.wasm:       file format WASM

.debug_info contents:
0x00000000: Compile Unit: length = 0x00000093, format = DWARF32, version = 0x0004, abbr_offset = 0x0000, addr_size = 0x04 (next unit at 0x00000097)

0x0000000b: DW_TAG_compile_unit
              DW_AT_producer    ("clang version 12.0.0 (https://github.com/llvm/llvm-project d28af7c654d8db0b68c175db5ce212d74fb5e9bc)")
              DW_AT_language    (DW_LANG_C99)
              DW_AT_name        ("foo.c")
              DW_AT_stmt_list   (0x00000000)
              DW_AT_comp_dir    ("/home/acrichto/code/wasmtime")
              DW_AT_low_pc      (0x00000000)
              DW_AT_ranges      (0x00000000
                 [0x0000000e, 0x00000012))

0x00000026:   DW_TAG_subprogram
                DW_AT_low_pc    (0x0000000e)
                DW_AT_high_pc   (0x00000012)
                DW_AT_frame_base        (DW_OP_WASM_location 0x3 0x0, DW_OP_stack_value)
                DW_AT_GNU_all_call_sites        (true)
                DW_AT_name      ("foo")
                DW_AT_decl_file ("/home/acrichto/code/wasmtime/foo.c")
                DW_AT_decl_line (2)
                DW_AT_type      (0x0000007c "int")
                DW_AT_external  (true)

0x00000041:   DW_TAG_subprogram
                DW_AT_low_pc    (dead code)
                DW_AT_high_pc   (0x00000004)
                DW_AT_frame_base        (DW_OP_WASM_location 0x3 0x0, DW_OP_stack_value)
                DW_AT_GNU_all_call_sites        (true)
                DW_AT_name      ("bar")
                DW_AT_decl_file ("/home/acrichto/code/wasmtime/foo.c")
                DW_AT_decl_line (4)
                DW_AT_prototyped        (true)
                DW_AT_type      (0x0000007c "int")
                DW_AT_external  (true)

0x0000005c:     DW_TAG_formal_parameter
                  DW_AT_location        (DW_OP_WASM_location 0x0 0x0, DW_OP_stack_value)
                  DW_AT_name    ("a")
                  DW_AT_decl_file       ("/home/acrichto/code/wasmtime/foo.c")
                  DW_AT_decl_line       (4)
                  DW_AT_type    (0x0000007c "int")

0x0000006c:     DW_TAG_variable
                  DW_AT_location        (0x00000000:
                     [0xfffffffe, 0x100000002): DW_OP_WASM_location 0x0 0x0, DW_OP_stack_value, DW_OP_piece 0x4, DW_OP_piece 0x70, DW_OP_WASM_location 0x0 0x0, DW_OP_stack_value, DW_OP_piece 0x4)
                  DW_AT_name    ("b")
                  DW_AT_decl_file       ("/home/acrichto/code/wasmtime/foo.c")
                  DW_AT_decl_line       (5)
                  DW_AT_type    (0x00000083 "int[50]")

0x0000007b:     NULL

0x0000007c:   DW_TAG_base_type
                DW_AT_name      ("int")
                DW_AT_encoding  (DW_ATE_signed)
                DW_AT_byte_size (0x04)

0x00000083:   DW_TAG_array_type
                DW_AT_type      (0x0000007c "int")

0x00000088:     DW_TAG_subrange_type
                  DW_AT_type    (0x0000008f "__ARRAY_SIZE_TYPE__")
                  DW_AT_count   (0x32)

0x0000008e:     NULL

0x0000008f:   DW_TAG_base_type
                DW_AT_name      ("__ARRAY_SIZE_TYPE__")
                DW_AT_byte_size (0x08)
                DW_AT_encoding  (DW_ATE_unsigned)

0x00000096:   NULL

With some digging the issue appears to be this:
0x0000006c:     DW_TAG_variable
                  DW_AT_location        (0x00000000:
                     [0xfffffffe, 0x100000002): DW_OP_WASM_location 0x0 0x0, DW_OP_stack_value, DW_OP_piece 0x4, DW_OP_piece 0x70, DW_OP_WASM_location 0x0 0x0, DW_OP_stack_value, DW_OP_piece 0x4)

where gimli is doing addition in the 32-bit address space for ranges which means that from gimli's perspective the start address of this variable is 0xfffffffe and the end address is 0x2, hence the error from gimli itself.
This appears to come about with DWARF debug information for dead code in a program. It appears that LLD specifically uses the value of -2 for representing relocations against dead functions that don't actually show up in the output. The DW_AT_location attribute here I believe is encoded as a offset/length which means that the offset receives the relocation of -2 and when gimli adds the length it gets the error that the range is messed up.
I believe the real fix here is to basically avoid generating native DWARF debugging information for dead code, or code that comes at the address of -1. llvm-dwarfdump already itself prints:
0x00000041:   DW_TAG_subprogram
                DW_AT_low_pc    (dead code)

and for something like that we should probably just skip over that and not copy over any other attributes related to this subprogram (function). I unfortunately don't know the dwarf code well enough in Wasmtime to know of a great place to put this.

Fixed in https://github.com/bytecodealliance/wasmtime/pull/3498


Support RM3 Pro Plus

There is a new version out there, that causes this bug: https://github.com/lprhodes/homebridge-broadlink-rm/issues/208
It can be fixed by adding devtype == 0x279d to the list of devices

I've added support for this dev type in the most recent version


Inconsistency between comments and code implementation

I noticed a possible panic due to inconsistency between comments and code implementation in arrow-rs/arrow-array/src/array/union_array.rs. The details can be found in the following code.
I think this function should check whether index is out of bounds as your comment read before it is used as self.type_ids[index], which could avoid unnecessary panic.
   /// # Panics
    ///
    /// Panics if `index` is greater than the length of the array.
    pub fn type_id(&self, index: usize) -> i8 {
        self.type_ids[index]
    }

By the way, I think it would be more complete if  the comment is modified to Panics if index is greater than  or equal to the length of the array. instead of Panics if index is greater than the length of the array.
    /// # Panics
    ///
    /// Panics if `index` is greater than the length of the array.
    pub fn value_offset(&self, index: usize) -> usize {
        assert!(index < self.len());

label_issue.py automatically added labels {'arrow'} from #5456


Test Issue - Potential Markdown Error

While reviewing content for ignite I noticed this section of the session delivery resources
plain text:
Important
[https://aka.ms/aitour/brk540/prompty-updates](GitHub Models and adding Prompty code)
markdown version (how it looks in the readme):

[!IMPORTANT]
[https://aka.ms/aitour/brk540/prompty-updates](GitHub Models and adding Prompty code)

I was unsure if this was intentional or a markdown issue, so I wanted to bring this to everyone's attention

Done


MWPW-153603 Remove stage mapping

Remove stage mapping (revert #205 )

Resolves: MWPW-153603
Related: MWPW-153692
Test URLs:
CTA link:

Before: https://main--bacom--adobecom.hlx.page/in/products/marketo/attribution?martech=off
After: https://methomas-remove-mapping--bacom--adobecom.hlx.page/in/products/marketo/attribution?martech=off

Scroll down to "Grow leads without growing your marketing spend." => Read report.
Should link to: https://business.adobe.com/resources/reports/the-total-economic-impact-of-bizible.html
Not the stage link: https://business.stage.adobe.com/resources/reports/the-total-economic-impact-of-bizible.html
PDF Viewer:

Before: https://main--bacom--adobecom.hlx.page/resources/sdk/how-to-achieve-personalization-at-scale-in-financial-services?martech=off
After: https://methomas-remove-mapping--bacom--adobecom.hlx.page/resources/sdk/how-to-achieve-personalization-at-scale-in-financial-services?martech=off

PDF still won't load in the "after" page because the pdf viewer key doesn't work on feature branches, but you can see in the network tab that it's correctly getting the prod link: https://business.adobe.com/content/dam/dx/us/en/resources/sdk/how-to-achieve-personalization-at-scale-in-financial-services/how-to-achieve-personalization-at-scale-in-financial-services-v2.pdf

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.43%. Comparing base (69e3c0b) to head (5efd813).

Additional details and impacted files
@@           Coverage Diff           @@
##            stage     #218   +/-   ##
=======================================
  Coverage   95.43%   95.43%
=======================================
  Files          11       11
  Lines         899      899
=======================================
  Hits          858      858
  Misses         41       41

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@Dli3 signed off via slack


How to open PDF file with Pdf.js viewer after some PHP codes executing? (without ?file=URL_PDF)

I don't know my question title is enough good or not.
I am tried to build pdf viewer in my website with PDf.js. I know, normally it's simple as: http://example.com/pdfjs/web/viewer.html?file=PDF_NAME.pdf
But my claim is - Pdf name dont show in the address bar. I was created view.php from viewer.html codes. And I want like this:
http://example.com/pdfjs/web/view.php?file=1
So in my php page I check database and fetch fileurl from ID=1.
and finally I want to set Pdf.js viewer for to open my pdf from Fileurl. But I cant. I am lost much time for this, googling for hours, but I cant unfortunately.
It works now like -> view.php?file=PDF_NAME.pdf
But I want to like -> view.php?file="fileId"
What can I do for getting like this result? How load PDF with javascript after my php codes?
Or with .htaccess is it like this possible? (sorry for my low quality English)

Web browser and its version:  Chrome (last)
PDF.js version: v2.1.266

.
` // my view.php file
// <?php (php start.)
if(isset($_GET['file'])){
$file = mysqli_real_escape_string($db, @$_GET['file']);
}else{
$file="";
}
if(is_numeric($file) AND $file!=""){
$query = mysqli_query($db, "select * FROM pdf_files WHERE id='$file' ");
if(mysqli_num_rows($query) == 1){
$result = mysqli_fetch_assoc($sorgu);
$filepath = $result['filepath'];
}
// ?> (php closed)
//then html codes starting (from pdfjs viewer.html) `

It's unfortunately somewhat difficult to understand what you're actually trying to ask here, but please understand that that it's not really possible to for us to provide PHP integration support here.
Furthermore, please see https://github.com/mozilla/pdf.js/blob/master/.github/CONTRIBUTING.md (emphasis mine):

If you are developing a custom solution, first check the examples at https://github.com/mozilla/pdf.js#learning and search existing issues. If this does not help, please prepare a short well-documented example that demonstrates the problem and make it accessible online on your website, JS Bin, GitHub, etc. before opening a new issue or contacting us on the IRC channel -- keep in mind that just code snippets won't help us troubleshoot the problem.

Yes, this is out of scope for the issue tracker since this isn't a bug in PDF.js, which is what we're tracking here. For integration support, please use either IRC or open forums like StackOverflow.


Fix Eth:Tx.decode for transaction with s length < 64 chars

Sometimes transaction can have "s" that has 62 char length, instead of 64 chars.
In the "ethereumjs-monorepo"(which is used in web3.js) used the same solution for creation signature
https://github.com/ethereumjs/ethereumjs-monorepo/blob/f39b0b2c826031a3f56b30f154d80c85bebfd611/packages/util/src/signature.ts#L59
const signature = Buffer.concat([setLengthLeft(r, 32), setLengthLeft(s, 32)], 64)

I've added the example transaction in tests:
Raw Transaction:
0xf86b820e8485012a05f200831e848094ffe811714ab35360b67ee195ace7c10d93f89d8c80844e71d92d8194a07b8f34a8fb85d850b3be4fc0330382e125e4216df5598c6d2c3bc47954684cf99f35ef53ee007c2f705eca91448b5c86e81d10f659ad868409bac8197bba9814

Codecov Report

Merging #148 (8368df0) into main (c100d53) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #148      +/-   ##
==========================================
- Coverage   99.74%   99.71%   -0.03%
==========================================
  Files          67       67
  Lines        3901     3911      +10
==========================================
+ Hits         3891     3900       +9
- Misses         10       11       +1

Impacted Files
Coverage Δ

lib/eth/tx/legacy.rb
100.00% <100.00%> (ø)

spec/eth/tx_spec.rb
100.00% <100.00%> (ø)

lib/eth/client/ipc.rb
93.33% <0.00%> (-6.67%)
:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

fix looks good, we would need to apply to for all transaction types though!
also, please run rufo on the code!


fixed flaky test in DataDogBackendClientTest

Requirements for Contributing to this repository

Fill out the template below. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
The pull request must only fix one issue at the time.
The pull request must update the test suite to demonstrate the changed functionality.
After you create the pull request, all status checks must be pass before a maintainer reviews your contribution. For more details, please see CONTRIBUTING.

What does this PR do?

The PR fixed the flaky test explained in issue#32.
Description of the Change

The cause of test failure is due to comparing two JSON strings. However the order of fields stored in JSONObject is non-deterministic. And you can get different results per run when you call the toString() method on a JSONObject. A simple fix is to use a JSONParser to parser the string back to JSONObject, and compare them instead.
Alternate Designs

Possible Drawbacks

Almost none. It adds non-significant runtime for the test because of the parse().
Verification Process

After the fix, use the reproduce procedure from issue#32, and the test will be passed.
Additional Notes

Review checklist (to be filled by reviewers)

[ ] Feature or bug fix MUST have appropriate tests (unit, integration, etc...)
[ ] Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
[ ] PR must have one changelog/ label attached. If applicable it should have the backward-incompatible label attached.
[ ] PR should not have do-not-merge/ label attached.
[ ] If Applicable, issue must have kind/ and severity/ labels attached at least.

Thanks!

Hi, so do I need to sign my commits to have this pr merged?

Hi, so do I need to sign my commits to have this pr merged?

Yes, I forgot about that, sorry. Could you squash the two commits and sign them? Thanks!

Updates: I'm merging this even without the commit signed 🙂 Thanks for your contribution!


update to WASI 0.2.1

Ran update-wasi.sh
Manually copied wasi:http 0.2.1, since the script did not update that
Manually updated WIT and rust refs to use 0.2.1
Rebuilt the adapters using build-adapter.sh
Added more error context because I've accidentally updated wasi:config to 0.2.1, which is still in draft - it was really difficult to find the issue without this. Happy to drop the commit and/or split into a separate PR
Updated dependencies

FWIW, I was unable to use wasi-virt for my use case (https://github.com/bytecodealliance/wasm-tools/issues/1811), but this issue was already reported in May (https://github.com/bytecodealliance/wasm-tools/issues/1565), so it could not have been introduced by this PR

Thank you for working through this @rvolosatovs this is great, I appreciate it was a tough upgrade path. The extra error context will definitely help in future too.
Upgrading to WAC instead of wasm-tools compose might be sensible for WASI Virt at this point, as I'm not sure what the maintenance status is for wasm-tools compose (WAC wasn't around when WASI Virt was first written).


[BUG] Tilt - Update permissions to avoid using sudo

Description
Fix Tilt permissions issue
Issue
Main build is broken as seen here.

Type of change
Please mark the relevant option(s):

[ ] New feature, functionality or library
[x] Bug fix
[ ] Code health or cleanup
[ ] Major breaking change
[ ] Documentation
[ ] Other

List of changes

Add ENV PATH $PATH:$HOME/.local/bin to dev and prod images

There's 1% of me that thinks that it should be a oneliner
ENV PATH $PATH:$GOPATH/bin:$HOME/.local/bin

but LGTM 🚢

@deblasis Bingo! You're 100% correct. But we are utilizing official golang image that already has this dir in a PATH envar! :)

@deblasis I think I'll just go ahead and remove line 9.


When using bash shell terminal on Windows, activating a virtual env messes up PATH

Environment data

VS Code version: Code 1.44.2 (ff91584, 2020-04-16T16:36:23.138Z)
Extension version (available under the Extensions sidebar): 2020.4.76186
OS and version: Windows_NT x64 10.0.18362
Python version (& distribution if applicable, e.g. Anaconda): Python 3.7.5 (tags/v3.7.5:5c02a39a0b, Oct 15 2019, 00:11:34) [MSC v.1916 64 bit (AMD64)] on win32
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
Relevant/affected Python packages and their versions: N/A
Relevant/affected Python-related VS Code extensions and their versions: N/A
Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977): XXX
Value of the python.languageServer setting: XXX

Expected behaviour
Activating virtual environment should work properly when using
Actual behaviour
After activating venv from a bash terminal the PATH is malformed and simple commands don't work.
Steps to reproduce:
See https://github.com/microsoft/vscode/issues/96924 which was closed as caused-by-extension even though it happens when the Python extension is disabled.
Also note, the same doesn't happen when running from a standalone bash terminal.
# /c/code/example
$ ls
main.py venv

# /c/code/example
$ echo $PATH
/mingw64/bin:/usr/bin:/c/Users/my-user/bin:/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/bin:/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.2/libnvvp:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/cmd:/c/WINDOWS/System32/OpenSSH:/c/Program Files/Docker/Docker/resources/bin:/c/ProgramData/DockerDesktop/version-bin:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/IncrediBuild:/c/Program Files/NVIDIA Corporation/Nsight Compute 2019.5.0:/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/c/Users/my-user/AppData/Local/Programs/Python/Python37/Scripts:/c/Users/my-user/AppData/Local/Programs/Python/Python37:/c/Users/my-user/AppData/Local/Microsoft/WindowsApps:/c/Users/my-user/AppData/Local/Programs/Microsoft VS Code/bin:/c/Utils/google-cloud-sdk/bin

# /c/code/example
$ source venv/Scripts/activate
(venv)

# /c/code/example
$ ls
bash: ls: command not found
(venv)

# /c/code/example
$ echo $PATH
C:\code\example\venv/Scripts:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\bin;C:\Program
Files\NVIDIA GPU Computing Toolkit\CUDA\v10.2\libnvvp;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\Program Files\Git\cmd;C:\WINDOWS\System32\OpenSSH;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Program Files\Microsoft
SQL Server\130\Tools\Binn;C:\Program Files (x86)\IncrediBuild;C:\Program Files\NVIDIA Corporation\Nsight Compute 2019.5.0;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Users\my-user\AppData\Local\Programs\Python\Python37\Scripts;C:\Users\my-user\AppData\Local\Programs\Python\Python37;C:\Users\my-user\AppData\Local\Microsoft\WindowsApps;;C:\Users\my-user\AppData\Local\Programs\Microsoft VS Code\bin

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

XXXX

@lanzkron, thanks for letting us know about this.  Note that VS Code (and consequently the Python extension) does not support git-bash as a shell.  So unexpected/unwanted behavior can happen sometimes.  That said, the difference between bash and git-bash is basically zero, so we would still like to get this sorted out for you.
As to the response from the VS Code team, there is a chance this is related to the Python extension.  What do you have set for "terminal.shell.integrated.windows"?
Please clarify what you meant by "even though it happens when the Python extension is disabled".  It sounds like you are not talking about the venv activation the Python extension does, but rather just the behavior in the VS Code terminal window when you run the activation command yourself.  Is that right?  (If so, this might now be related to the Python extension after all.)
Also, what happens when the Python extension tries to activate the venv (i.e. "python.pythonPath" set to your venv)?

@lanzkron, thanks for letting us know about this. Note that VS Code (and consequently the Python extension) does not support git-bash as a shell.

I'm talking about the terminal window in VSCode, git-bash is one of the three built in options VSCode suggests as the default shell.

So unexpected/unwanted behavior can happen sometimes. That said, the difference between bash and git-bash is basically zero, so we would still like to get this sorted out for you.

Thanks!

As to the response from the VS Code team, there is a chance this is related to the Python extension. What do you have set for "terminal.shell.integrated.windows"?

I can't find "terminal.shell.integrated.windows" I assume you mean "terminal.integrated.shell.windows".
`"terminal.integrated.shell.windows": "C:\\Program Files\\Git\\bin\\bash.exe",`

Please clarify what you meant by "even though it happens when the Python extension is disabled". It sounds like you are not talking about the venv activation the Python extension does, but rather just the behavior in the VS Code terminal window when you run the activation command yourself. Is that right?

Yes, when the virtual environment is activated $PATH gets messed up. If the Python extension is activated then this happens automatically. If Python isn't enabled then this only happens after the venv is activated manually (source venv/Scripts/activate).

(If so, this might now be related to the Python extension after all.)

Is now here  a typo for not? If so, that's what I thought which is why I originally opened the defect on VSCode core and not on the extension. If it's not a typo then I'm sorry I don't understand what you're trying to say.

Also, what happens when the Python extension tries to activate the venv (i.e. "python.pythonPath" set to your venv)?

 `"python.pythonPath": "venv\\Scripts\\python.exe"`

@ericsnowcurrently I answered your questions in my previous post. However while collecting the information for this I see that the problem no longer reproduces for me...
I don't know if something was fixed by some update (VScode/pip) or just a hiccough in my system.
If you think it's worthwhile trying to investigate the issue please do so, otherwise please feel free to close it.
Thanks!

Thanks for answering my questions.  Sorry for any confusion.  Yeah, I meant to say "not" rather than "now". :smile:
Given your answers, the problem appears to have nothing to do with the Python extension.  It is either a problem with VS Code or with venv.  Note that both VS Code and the Python extension published new releases yesterday.  I expect that is why you aren't seeing the problem now.  So that would imply that VS Code was the source of the problem.  If it had been venv then the problem would not have cleared up until you had a different version, created a new environment, and tried using that one.
At this point I think we're okay closing this issue.  Thanks again!


Added workflow, license, social media, RSS badges to README

Added workflow, license, social media, RSS badges to README.md
Purpose of this pull request
This pull request (PR) ...
Issue number:

Select the type of change:

[ ] Minor Changes - Typos, formatting, slight revisions
[ ] Update Content - Revisions and updating sections
[ ] New Content - New features, sections, pages, tutorials
[x] Site and Tools - Updates, maintenance, and new packages for the site, Gatsby, React, etc

@SumoLogic/open-source-collection-team , are there any other badges I should add?


Add displayName to User/UserProfile

This makes displayName settable for new User creation

Thank you for your PR and sorry it took so long to get back to it. We have recently released a 1.x version of this SDK which completely updates the structure and build process for our SDK.
I believe our changes no longer require this PR to be included so we are going to close it. If you feel the new version does not resolve the issues you were seeing, prompting this PR, please let us know and we will be happy to re-investigate.


Support for ES2018 Regular Expressions

I'm getting a Warning for a Regular Expression but it actually works.
Javascript code:
function getIframesSources(html) {
	return html.match(/(?<=<iframe.* src\s*=\s*").*?(?=")/gmi);
}

Warning thrown by Closure Compiler v20180716:
WARNING - Malformed Regular Expression: Malformed parenthetical: (?<=<iframe.* src\s*=\s*").*?(?=")
Working RegEx test: https://regex101.com/r/4YyzKP/1

The regular expression in question contains a positive lookbehind assertion (?<=...), which is expected to be part of the next version of the standard (ES2018).

@MatthewMerrill will be working on this in the next few weeks.

Bear in mind, though, that closure-compiler does not attempt to make new regular expression features work on old browsers. For now, the best you can hope for is that the compiler will stop considering the regular expression to be an error and start issuing you a warning that it may not work at runtime.
We'll be able to make that warning go away when we can support ES_2018 level output, but I'm guessing that's a year away.

@tjgq @brad4d thanks, I didn't know this kind of RegEx was so new. I end up changing it in order to support old browsers. I will change the issue title.

Sorry, I don't quite remember the context of why I assigned myself (maybe by mistake?). In any case I'm not working on this so un-assigning.

We've done all we plan to do for supporting the RegExp features added in ES_2018.
closure-compiler will recognize them and not choke on them.
However, if your output language is earlier than ES_2018, then the compiler will issue a suppressible error to tell you that it cannot transpile the new RegExp features to make them work where they are not natively supported.

Why is closure-compiler not including newer RegExp formats? I think there should be at least some kind of polyfill or fallback. On still somewhat common browsers, now vscode fails to open due to a new RegExp included in a .ts file. This is very ugly.


⚠ Mobitopup has degraded performance

In d6318ed, Mobitopup (https://napcard.voz.tech) experienced degraded performance:

HTTP code: 200
Response time: 3858 ms

Resolved: Mobitopup performance has improved in 59a5abb after 14 minutes.


MM-9639 Obey max post length as provided by server

There'll still be a followup PR for the post expand/collapse feature, but the CSS for it is still being worked on.
Ticket Link
https://mattermost.atlassian.net/browse/MM-9639

Looks like it needs to update the snapshots.

@hmhealey we good to merge this?

@jwilander Yep, it's good to merge


Link OpenSSF announcement to blog page

The OpenSSF announcement/article should be posted on the blog page under the “IN AND AROUND THE WEB” section here: https://comcast.github.io/blog
Date of article: May 9, 2022
Title: OpenSSF Announces 15 New Members To Further Strengthen Open Source Software Supply Chain Security
Link to article: https://openssf.org/press-release/2022/05/09/openssf-announces-15-new-members-to-further-strengthen-open-source-software-supply-chain-security/

Can I do this?

Hi, don't do this one please.  I'm an intern with the OSPO and will be working on this issue soon, thank you.

where can i go to do my twilioquset mission?

@o2ku can you please elaborate on the twilioquset mission? I'm not sure what that relates to. Thank you!