Upgraded koalas and Spark, the same datasets but out of memoy

Dataset: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/HG7NV7
We used the 2005, 2006, 2007 dataset. The size of these datasets are about 2GB.
In the same configuration，when we handle these datasets with koalas 0.32 and spark 2.4.4, the resut is nomal.

When we handle these datasets with koalas 1.1.0 and spark 3.0.0, the result is abnormal.

Specific error messages:
ERROR:root:Exception while sending command.
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 241, in _collect_as_arrow
results = list(_load_from_socket((port, auth_secret), ArrowCollectSerializer()))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 60, in load_stream
for batch in self.serializer.load_stream(stream):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 98, in load_stream
reader = pa.ipc.open_stream(stream)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 146, in open_stream
return RecordBatchStreamReader(source)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 62, in init
self._open(source)
File "pyarrow/ipc.pxi", line 360, in pyarrow.lib._RecordBatchStreamReader._open
File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1207, in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1033, in send_command
response = connection.send_command(command)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1212, in send_command
"Error while receiving", e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py:134: UserWarning: toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.pyspark.enabled' is set to true, but has reached the error below and can not continue. Note that 'spark.sql.execution.arrow.pyspark.fallback.enabled' does not have an effect on failures in the middle of computation.
An error occurred while calling o1304.getResult
warnings.warn(msg)
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (<IP_ADDRESS>:50804)
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 241, in _collect_as_arrow
results = list(_load_from_socket((port, auth_secret), ArrowCollectSerializer()))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 60, in load_stream
for batch in self.serializer.load_stream(stream):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 98, in load_stream
reader = pa.ipc.open_stream(stream)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 146, in open_stream
return RecordBatchStreamReader(source)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 62, in init
self._open(source)
File "pyarrow/ipc.pxi", line 360, in pyarrow.lib._RecordBatchStreamReader._open
File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 224, in catch_format_error
r = method(self, *args, **kwargs)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 702, in call
printer.pretty(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 394, in pretty
return _repr_pprint(obj, self, cycle)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 684, in _repr_pprint
output = repr(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9965, in repr
pdf = self._get_or_create_repr_pandas_cache(max_display_count)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9957, in _get_or_create_repr_pandas_cache
self._repr_pandas_cache = {n: self.head(n + 1)._to_internal_pandas()}
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9953, in _to_internal_pandas
return self._internal.to_pandas_frame
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/utils.py", line 477, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/internal.py", line 743, in to_pandas_frame
pdf = sdf.toPandas()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 108, in toPandas
batches = self.toDF(*tmp_column_names)._collect_as_arrow()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 244, in _collect_as_arrow
jsocket_auth_server.getResult()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1305, in call
answer, self.gateway_client, self.target_id, self.name)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/utils.py", line 131, in deco
return f(*a, **kw)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o1304.getResult
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
stb = value.render_traceback()
AttributeError: 'Py4JError' object has no attribute 'render_traceback'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (<IP_ADDRESS>:50804)
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 241, in _collect_as_arrow
results = list(_load_from_socket((port, auth_secret), ArrowCollectSerializer()))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 60, in load_stream
for batch in self.serializer.load_stream(stream):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 98, in load_stream
reader = pa.ipc.open_stream(stream)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 146, in open_stream
return RecordBatchStreamReader(source)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 62, in init
self._open(source)
File "pyarrow/ipc.pxi", line 360, in pyarrow.lib._RecordBatchStreamReader._open
File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 224, in catch_format_error
r = method(self, *args, **kwargs)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 702, in call
printer.pretty(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 394, in pretty
return _repr_pprint(obj, self, cycle)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 684, in _repr_pprint
output = repr(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9965, in repr
pdf = self._get_or_create_repr_pandas_cache(max_display_count)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9957, in _get_or_create_repr_pandas_cache
self._repr_pandas_cache = {n: self.head(n + 1)._to_internal_pandas()}
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9953, in _to_internal_pandas
return self._internal.to_pandas_frame
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/utils.py", line 477, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/internal.py", line 743, in to_pandas_frame
pdf = sdf.toPandas()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 108, in toPandas
batches = self.toDF(*tmp_column_names)._collect_as_arrow()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 244, in _collect_as_arrow
jsocket_auth_server.getResult()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1305, in call
answer, self.gateway_client, self.target_id, self.name)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/utils.py", line 131, in deco
return f(*a, **kw)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o1304.getResult
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
stb = value.render_traceback()
AttributeError: 'Py4JError' object has no attribute 'render_traceback'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (<IP_ADDRESS>:50804)
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 241, in _collect_as_arrow
results = list(_load_from_socket((port, auth_secret), ArrowCollectSerializer()))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 60, in load_stream
for batch in self.serializer.load_stream(stream):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 98, in load_stream
reader = pa.ipc.open_stream(stream)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 146, in open_stream
return RecordBatchStreamReader(source)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 62, in init
self._open(source)
File "pyarrow/ipc.pxi", line 360, in pyarrow.lib._RecordBatchStreamReader._open
File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 224, in catch_format_error
r = method(self, *args, **kwargs)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 702, in call
printer.pretty(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 394, in pretty
return _repr_pprint(obj, self, cycle)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 684, in _repr_pprint
output = repr(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9965, in repr
pdf = self._get_or_create_repr_pandas_cache(max_display_count)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9957, in _get_or_create_repr_pandas_cache
self._repr_pandas_cache = {n: self.head(n + 1)._to_internal_pandas()}
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9953, in _to_internal_pandas
return self._internal.to_pandas_frame
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/utils.py", line 477, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/internal.py", line 743, in to_pandas_frame
pdf = sdf.toPandas()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 108, in toPandas
batches = self.toDF(*tmp_column_names)._collect_as_arrow()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 244, in _collect_as_arrow
jsocket_auth_server.getResult()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1305, in call
answer, self.gateway_client, self.target_id, self.name)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/utils.py", line 131, in deco
return f(*a, **kw)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o1304.getResult
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
stb = value.render_traceback()
AttributeError: 'Py4JError' object has no attribute 'render_traceback'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (<IP_ADDRESS>:50804)
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 241, in _collect_as_arrow
results = list(_load_from_socket((port, auth_secret), ArrowCollectSerializer()))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 60, in load_stream
for batch in self.serializer.load_stream(stream):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 98, in load_stream
reader = pa.ipc.open_stream(stream)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 146, in open_stream
return RecordBatchStreamReader(source)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 62, in init
self._open(source)
File "pyarrow/ipc.pxi", line 360, in pyarrow.lib._RecordBatchStreamReader._open
File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 224, in catch_format_error
r = method(self, *args, **kwargs)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 702, in call
printer.pretty(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 394, in pretty
return _repr_pprint(obj, self, cycle)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 684, in _repr_pprint
output = repr(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9965, in repr
pdf = self._get_or_create_repr_pandas_cache(max_display_count)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9957, in _get_or_create_repr_pandas_cache
self._repr_pandas_cache = {n: self.head(n + 1)._to_internal_pandas()}
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9953, in _to_internal_pandas
return self._internal.to_pandas_frame
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/utils.py", line 477, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/internal.py", line 743, in to_pandas_frame
pdf = sdf.toPandas()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 108, in toPandas
batches = self.toDF(*tmp_column_names)._collect_as_arrow()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 244, in _collect_as_arrow
jsocket_auth_server.getResult()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1305, in call
answer, self.gateway_client, self.target_id, self.name)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/utils.py", line 131, in deco
return f(*a, **kw)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o1304.getResult
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
stb = value.render_traceback()
AttributeError: 'Py4JError' object has no attribute 'render_traceback'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (<IP_ADDRESS>:50804)
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 241, in _collect_as_arrow
results = list(_load_from_socket((port, auth_secret), ArrowCollectSerializer()))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 60, in load_stream
for batch in self.serializer.load_stream(stream):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 98, in load_stream
reader = pa.ipc.open_stream(stream)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 146, in open_stream
return RecordBatchStreamReader(source)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 62, in init
self._open(source)
File "pyarrow/ipc.pxi", line 360, in pyarrow.lib._RecordBatchStreamReader._open
File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 224, in catch_format_error
r = method(self, *args, **kwargs)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 702, in call
printer.pretty(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 394, in pretty
return _repr_pprint(obj, self, cycle)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 684, in _repr_pprint
output = repr(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9965, in repr
pdf = self._get_or_create_repr_pandas_cache(max_display_count)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9957, in _get_or_create_repr_pandas_cache
self._repr_pandas_cache = {n: self.head(n + 1)._to_internal_pandas()}
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9953, in _to_internal_pandas
return self._internal.to_pandas_frame
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/utils.py", line 477, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/internal.py", line 743, in to_pandas_frame
pdf = sdf.toPandas()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 108, in toPandas
batches = self.toDF(*tmp_column_names)._collect_as_arrow()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 244, in _collect_as_arrow
jsocket_auth_server.getResult()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1305, in call
answer, self.gateway_client, self.target_id, self.name)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/utils.py", line 131, in deco
return f(*a, **kw)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o1304.getResult
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
stb = value.render_traceback()
AttributeError: 'Py4JError' object has no attribute 'render_traceback'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (<IP_ADDRESS>:50804)
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 241, in _collect_as_arrow
results = list(_load_from_socket((port, auth_secret), ArrowCollectSerializer()))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 60, in load_stream
for batch in self.serializer.load_stream(stream):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 98, in load_stream
reader = pa.ipc.open_stream(stream)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 146, in open_stream
return RecordBatchStreamReader(source)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 62, in init
self._open(source)
File "pyarrow/ipc.pxi", line 360, in pyarrow.lib._RecordBatchStreamReader._open
File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 224, in catch_format_error
r = method(self, *args, **kwargs)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 702, in call
printer.pretty(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 394, in pretty
return _repr_pprint(obj, self, cycle)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 684, in _repr_pprint
output = repr(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9965, in repr
pdf = self._get_or_create_repr_pandas_cache(max_display_count)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9957, in _get_or_create_repr_pandas_cache
self._repr_pandas_cache = {n: self.head(n + 1)._to_internal_pandas()}
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9953, in _to_internal_pandas
return self._internal.to_pandas_frame
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/utils.py", line 477, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/internal.py", line 743, in to_pandas_frame
pdf = sdf.toPandas()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 108, in toPandas
batches = self.toDF(*tmp_column_names)._collect_as_arrow()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 244, in _collect_as_arrow
jsocket_auth_server.getResult()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1305, in call
answer, self.gateway_client, self.target_id, self.name)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/utils.py", line 131, in deco
return f(*a, **kw)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o1304.getResult
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
stb = value.render_traceback()
AttributeError: 'Py4JError' object has no attribute 'render_traceback'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (<IP_ADDRESS>:50804)
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 241, in _collect_as_arrow
results = list(_load_from_socket((port, auth_secret), ArrowCollectSerializer()))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 60, in load_stream
for batch in self.serializer.load_stream(stream):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py", line 98, in load_stream
reader = pa.ipc.open_stream(stream)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 146, in open_stream
return RecordBatchStreamReader(source)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py", line 62, in init
self._open(source)
File "pyarrow/ipc.pxi", line 360, in pyarrow.lib._RecordBatchStreamReader._open
File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Tried reading schema message, was null or length 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 224, in catch_format_error
r = method(self, *args, **kwargs)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py", line 702, in call
printer.pretty(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 394, in pretty
return _repr_pprint(obj, self, cycle)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py", line 684, in _repr_pprint
output = repr(obj)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9965, in repr
pdf = self._get_or_create_repr_pandas_cache(max_display_count)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9957, in _get_or_create_repr_pandas_cache
self._repr_pandas_cache = {n: self.head(n + 1)._to_internal_pandas()}
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py", line 9953, in _to_internal_pandas
return self._internal.to_pandas_frame
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/utils.py", line 477, in wrapped_lazy_property
setattr(self, attr_name, fn(self))
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/internal.py", line 743, in to_pandas_frame
pdf = sdf.toPandas()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 108, in toPandas
batches = self.toDF(*tmp_column_names)._collect_as_arrow()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py", line 244, in _collect_as_arrow
jsocket_auth_server.getResult()
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1305, in call
answer, self.gateway_client, self.target_id, self.name)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/utils.py", line 131, in deco
return f(*a, **kw)
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/protocol.py", line 336, in get_return_value
format(target_id, ".", name))
py4j.protocol.Py4JError: An error occurred while calling o1304.getResult
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2044, in showtraceback
stb = value.render_traceback()
AttributeError: 'Py4JError' object has no attribute 'render_traceback'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
ArrowInvalid                              Traceback (most recent call last)
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py in _collect_as_arrow(self)
240         try:
--> 241             results = list(_load_from_socket((port, auth_secret), ArrowCollectSerializer()))
242         finally:
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py in load_stream(self, stream)
59         # load the batches
---> 60         for batch in self.serializer.load_stream(stream):
61             yield batch
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/serializers.py in load_stream(self, stream)
97         import pyarrow as pa
---> 98         reader = pa.ipc.open_stream(stream)
99         for batch in reader:
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py in open_stream(source)
145     """
--> 146     return RecordBatchStreamReader(source)
147
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.py in init(self, source)
61     def init(self, source):
---> 62         self._open(source)
63
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/ipc.pxi in pyarrow.lib._RecordBatchStreamReader._open()
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
ArrowInvalid: Tried reading schema message, was null or length 0
During handling of the above exception, another exception occurred:
Py4JError                                 Traceback (most recent call last)
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py in call(self, obj)
700                 type_pprinters=self.type_printers,
701                 deferred_pprinters=self.deferred_printers)
--> 702             printer.pretty(obj)
703             printer.flush()
704             return stream.getvalue()
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py in pretty(self, obj)
392                         if cls is not object
393                                 and callable(cls.dict.get('repr')):
--> 394                             return _repr_pprint(obj, self, cycle)
395
396             return _default_pprint(obj, self, cycle)
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/lib/pretty.py in repr_pprint(obj, p, cycle)
682     """A pprint that just redirects to the normal repr function."""
683     # Find newlines and replace them with p.break()
--> 684     output = repr(obj)
685     lines = output.splitlines()
686     with p.group():
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py in repr(self)
9963             return self._to_internal_pandas().to_string()
9964
-> 9965         pdf = self._get_or_create_repr_pandas_cache(max_display_count)
9966         pdf_length = len(pdf)
9967         pdf = pdf.iloc[:max_display_count]
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py in _get_or_create_repr_pandas_cache(self, n)
9955     def _get_or_create_repr_pandas_cache(self, n):
9956         if not hasattr(self, "_repr_pandas_cache") or n not in self._repr_pandas_cache:
-> 9957             self._repr_pandas_cache = {n: self.head(n + 1)._to_internal_pandas()}
9958         return self._repr_pandas_cache[n]
9959
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py in _to_internal_pandas(self)
9951         This method is for internal use only.
9952         """
-> 9953         return self._internal.to_pandas_frame
9954
9955     def _get_or_create_repr_pandas_cache(self, n):
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/utils.py in wrapped_lazy_property(self)
475     def wrapped_lazy_property(self):
476         if not hasattr(self, attr_name):
--> 477             setattr(self, attr_name, fn(self))
478         return getattr(self, attr_name)
479
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/internal.py in to_pandas_frame(self)
741         """ Return as pandas DataFrame. """
742         sdf = self.to_internal_spark_frame
--> 743         pdf = sdf.toPandas()
744         if len(pdf) == 0 and len(sdf.schema) > 0:
745             pdf = pdf.astype(
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py in toPandas(self)
106                     # Rename columns to avoid duplicated column names.
107                     tmp_column_names = ['col_{}'.format(i) for i in range(len(self.columns))]
--> 108                     batches = self.toDF(*tmp_column_names)._collect_as_arrow()
109                     if len(batches) > 0:
110                         table = pyarrow.Table.from_batches(batches)
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/pandas/conversion.py in _collect_as_arrow(self)
242         finally:
243             # Join serving thread and raise any exceptions from collectAsArrowToPython
--> 244             jsocket_auth_server.getResult()
245
246         # Separate RecordBatches from batch order indices in results
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py in call(self, *args)
1303         answer = self.gateway_client.send_command(command)
1304         return_value = get_return_value(
-> 1305             answer, self.gateway_client, self.target_id, self.name)
1306
1307         for temp_arg in temp_args:
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/utils.py in deco(*a, **kw)
129     def deco(*a, **kw):
130         try:
--> 131             return f(*a, **kw)
132         except py4j.protocol.Py4JJavaError as e:
133             converted = convert_exception(e.java_exception)
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
334             raise Py4JError(
335                 "An error occurred while calling {0}{1}{2}".
--> 336                 format(target_id, ".", name))
337     else:
338         type = answer[1]
Py4JError: An error occurred while calling o1304.getResult
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (<IP_ADDRESS>:50804)
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 977, in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/dengxinhuan/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py", line 1115, in start
self.socket.connect((self.address, self.port))
ConnectionRefusedError: [Errno 111] Connection refused
IndexError                                Traceback (most recent call last)
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py in _get_connection(self)
976         try:
--> 977             connection = self.deque.pop()
978         except IndexError:
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
ConnectionRefusedError                    Traceback (most recent call last)
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py in start(self)
1114         try:
-> 1115             self.socket.connect((self.address, self.port))
1116             self.stream = self.socket.makefile("rb")
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Py4JNetworkError                          Traceback (most recent call last)
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/IPython/core/formatters.py in call(self, obj)
343             method = get_real_method(obj, self.print_method)
344             if method is not None:
--> 345                 return method()
346             return None
347         else:
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/frame.py in repr_html(self)
9979
9980     def repr_html(self):
-> 9981         max_display_count = get_option("display.max_rows")
9982         # pandas 0.25.1 has a regression about HTML representation so 'bold_rows'
9983         # has to be set as False explicitly. See https://github.com/pandas-dev/pandas/issues/28204
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/config.py in get_option(key, default)
299     _options_dict[key].validate(default)
300
--> 301     return json.loads(default_session().conf.get(_key_format(key), default=json.dumps(default)))
302
303
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/databricks/koalas/utils.py in default_session(conf)
379     # configuration. This is needed with Spark 3.0+.
380     builder.config("spark.sql.analyzer.failAmbiguousSelfJoin", False)
--> 381     session = builder.getOrCreate()
382
383     if not should_use_legacy_ipc:
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/pyspark/sql/session.py in getOrCreate(self)
189                     session = SparkSession(sc)
190                 for key, value in self._options.items():
--> 191                     session._jsparkSession.sessionState().conf().setConfString(key, value)
192                 return session
193
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py in call(self, *args)
1301             proto.END_COMMAND_PART
1302
-> 1303         answer = self.gateway_client.send_command(command)
1304         return_value = get_return_value(
1305             answer, self.gateway_client, self.target_id, self.name)
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py in send_command(self, command, retry, binary)
1029          if binary is True.
1030         """
-> 1031         connection = self._get_connection()
1032         try:
1033             response = connection.send_command(command)
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py in _get_connection(self)
977             connection = self.deque.pop()
978         except IndexError:
--> 979             connection = self._create_connection()
980         return connection
981
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py in _create_connection(self)
983         connection = GatewayConnection(
984             self.gateway_parameters, self.gateway_property)
--> 985         connection.start()
986         return connection
987
~/miniconda3/envs/blackhole_qatest0507/lib/python3.6/site-packages/py4j/java_gateway.py in start(self)
1125                 "server ({0}:{1})".format(self.address, self.port)
1126             logger.exception(msg)
-> 1127             raise Py4JNetworkError(msg, e)
1128
1129     def _authenticate_connection(self):
Py4JNetworkError: An error occurred while trying to connect to the Java server (<IP_ADDRESS>:50804)
Configuration information:

Looks like your Spark version was also upgraded to Spark 3.0. Does the same error happens even with Spark 2.4?

It may be the influence of resources. When the current resource is determined to be exclusive, the problem does not recur.


add pure C and C++ versions of the enumopencl sample

This change switches the existing enumopencl sample to use pure C, compilable with a C compiler (no C++).
It also adds an equivalent C++ sample that makes extensive use of the OpenCL C++ bindings.
The output from both samples is identical.

Thanks @keryell , I think I fixed all of these, except I didn't combine the previous \n with the current \t just because it seemed a little too clever for this simple sample.


Update PortRanger.sh

Ports numbers are between 0 and 65535 so instead of using + we can just specify a min and max number of digits to match.

if you prefer it :) #2ef80f4


EdgeConnection

Review EdgeConnections. Fix constructors and sort out PredefinedTypes logic.
https://github.com/strusoft/femdesign-api/blob/71ef4cce6d0aba528d1d74cefdfcbdfaa780c8ad/src/Shells/ShellEdgeConnection.cs#L54-L61
https://github.com/strusoft/femdesign-api/blob/71ef4cce6d0aba528d1d74cefdfcbdfaa780c8ad/src/Shells/ShellEdgeConnection.cs#L26-L41

Relates to #303 and #302

@andosca Should we consider this done now that I have implemented 302 and 303?


Stale scheduled jobs (whose responsibility, what to do about it?)

Hi!
I'm testing manually the resiliency of the new cron (2023.1+).
I came across a "(cron)scheduled job" that is scheduled in the past. Other jobs are picked up from that registry and queued, but this job doesn't.
Question 1: does the job exist? (or is it an artifact)
Question 2: is it RQ's responsibility? (and for some reason it's not picking it up?)
Question 3: can we do something about it? (e.g enqueue it if we see that it's X time old, and the queue is empty and there's 0 active jobs in that queue? (all things we can know about with certainty))
Thanks!

Fixed, added Job scheduling jobs


added Base64 support

Already tested.
Gives users ability to use Base64, so they can load images from the web, for example.
Also cleaned up here and there.

Now it should be finally ready. Forgot to remove some unneeded software...


example app upgraded to latest version of react, react-native

Also updated the example app UI to display the selected date on the UI instead of console log

fixes issue https://github.com/mmazzarolo/react-native-modal-datetime-picker/issues/182

Thanks again @gbhasha !


4/19 Publish

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

2017-03-09-profile/docs-ref-autogen/TOC.yml
:white_check_mark:Succeeded
View (azure-cli-2017-03-09-profile)

2017-03-09-profile/docs-ref-autogen/ext/aem/index.yml
:white_check_mark:Succeeded
View (azure-cli-2017-03-09-profile)

2017-03-09-profile/docs-ref-autogen/ext/aem/vm.yml
:white_check_mark:Succeeded

2017-03-09-profile/docs-ref-autogen/ext/aem/vm/aem.yml
:white_check_mark:Succeeded

latest/docs-ref-autogen/TOC.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/ext/aem/index.yml
:white_check_mark:Succeeded
View (azure-cli-latest)

latest/docs-ref-autogen/ext/aem/vm.yml
:white_check_mark:Succeeded

latest/docs-ref-autogen/ext/aem/vm/aem.yml
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


enrollCustom does not emit events

Describe the bug
The enrollCustom function does not emit events so the subgraph state doesn't accurately reflect custom representations. This causes offchain agents to be unable to retrieve the correct assets.
To Reproduce
Steps to reproduce the behavior:

Setup new asset
Call enrollCustom to register the new asset as the new representation in the lookup tables
Try to transfer the new representation with xcall
Router subscriber will error with "Unable to get asset" on execute
XCall stalls forever

Additional context
See transferId: 0x90410c48109664efa8a73d4ad01fdedb5ba3876ad6f34d49e5806623cfc0a388 for an example using madWETH from Rinkeby -> Goerli.
The relevant DD logs

I don't think we need to change this, we should be able to get all the info we need from our own events in setupAsset.

no longer needed as we are emitting the local asset in setupAsset


unused_ddic: add skipNames config

Should work the same way as for unused_types/unused_variables

can the existing exclude field be used? https://rules.abaplint.org/unused_ddic/ it supports regex patterns
eg if domain FOO is reported unused, set exclude: ["foo.doma"]

it works! (note: refer to the file #mbtools#mytab.tabl and not the object /mbtools/mytab).


Plugin for PkgPage

would be great to have a plugin to generate files for PkgPage and other files for the package in one template.
cc: @tlienart

Sounds like a good idea to me; happy to help with this if the devs here are interested

I certainly wouldn't object, seems useful.


MySQL 8: Error executing statement parameters: Incorrect arguments to mysqld_stmt_execute

Can be reproduced with e.g.
db = assert(require"DBI".Connect("MySQL", "prosody", "prosody", "secret", "<IP_ADDRESS>", 3306))
s = assert(db:prepare("SELECT 1 LIMIT ? OFFSET ?"))
assert(s:execute(1, 1))

A MySQL to test against can be brought up easily using:
docker run -d --rm --name mysql \
  -e MYSQL_ROOT_PASSWORD=secret \
  -e MYSQL_USER=prosody \
  -e MYSQL_PASSWORD=secret \
  -e MYSQL_DATABASE=prosody \
  -v mysql:/var/lib/mysql \
  -p 3306:3306 \
  mysql:8.0.23

Theory: MySQL expects integers in the LIMIT slot but LuaDBI is pushing floats, since before Lua 5.3, all numbers are.

Hi,
are there any updates on this? I encounter the same issue, and it prevents some people in the Monal client from sending messages.

I'm using a custom XMPP client that connects to a Prosody server and am affected by this same issue.

Is everyone affected using Lua 5.3?

I'm running Prosody with Lua 5.1:
prosodyctl about
Prosody 0.11.8
# Lua environment
Lua version:                    Lua 5.1
Lua module search paths:
  /usr/lib64/prosody/?.lua
  /usr/local/share/lua/5.1/?.lua
  /usr/local/share/lua/5.1/?/init.lua
  /usr/local/lib64/lua/5.1/?.lua
  /usr/local/lib64/lua/5.1/?/init.lua
  /usr/share/lua/5.1/?.lua
  /usr/share/lua/5.1/?/init.lua
Lua C module search paths:
  /usr/lib64/prosody/?.so
  /usr/local/lib64/lua/5.1/?.so
  /usr/lib64/lua/5.1/?.so
  /usr/local/lib64/lua/5.1/loadall.so
LuaRocks:               Not installed
# Network
Backend: libevent epoll
# Lua module versions
lfs:            LuaFileSystem 1.7.0
libevent:       2.1.8-stable
luaevent:       0.4.4
lxp:            LuaExpat 1.3.0
socket:         LuaSocket 3.0-rc1
ssl:            0.6

5.2 for me

# Prosody directories
Data directory:     /var/lib/prosody
Config directory:   /etc/prosody
Source directory:   /usr/lib/prosody
Plugin directories:
  /usr/lib/prosody/modules/

# Lua environment
Lua version:                    Lua 5.2

Lua module search paths:
  /usr/lib/prosody/?.lua
  /usr/local/share/lua/5.2/?.lua
  /usr/local/share/lua/5.2/?/init.lua
  /usr/local/lib/lua/5.2/?.lua
  /usr/local/lib/lua/5.2/?/init.lua
  /usr/share/lua/5.2/?.lua
  /usr/share/lua/5.2/?/init.lua

Lua C module search paths:
  /usr/lib/prosody/?.so
  /usr/local/lib/lua/5.2/?.so
  /usr/lib/x86_64-linux-gnu/lua/5.2/?.so
  /usr/lib/lua/5.2/?.so
  /usr/local/lib/lua/5.2/loadall.so

LuaRocks:               Not installed

# Network

Backend: select

# Lua module versions
DBI:            0.7
lfs:            LuaFileSystem 1.6.3
libevent:       2.1.8-stable
luaevent:       0.4.6
lxp:            LuaExpat 1.3.0
socket:         LuaSocket 3.0-rc1
ssl:            0.7```

seems to be caused by an update to MySQL, does this need to be patched in luadbi as well?
https://github.com/sidorares/node-mysql2/issues/1239
there are a couple workarounds posted here

@mwild1

Is everyone affected using Lua 5.3?

I think what I meant was that all numbers in Lua 5.2 are floats, so floats is what gets pushed, but MySQL wants integers here. Was my guess anyway.

https://github.com/sidorares/node-mysql2/issues/1239#issuecomment-760314979 suggests that MySQL expects these numbers to be strings. And indeed:
s = assert(db:prepare("SELECT 1 LIMIT ? OFFSET ?"))
assert(s:execute(1, 1)) --[[
stdin:1: Error executing statement parameters: Incorrect arguments to mysqld_stmt_execute
stack traceback:
	[C]: in function 'assert'
	stdin:1: in main chunk
	[C]: in ?
]]
assert(s:execute("1", "1")) --> true

I don't even!

One solution would be to check whether *num == floor(*num) at https://github.com/mwild1/luadbi/blob/master/dbd/mysql/statement.c#L242 and push the appropriate mysql_type based on that. But probably it won't be a good solution in every case. There is not enough info to always guess the intended/correct type.

Related: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270844

https://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-22.html#mysqld-8-0-22-optimizer

It was possible (although unsupported) to insert a floating point value into a LIMIT or OFFSET clause by means of a parameter (and possibly a user variable). This is now explicitly disallowed; such values must be integers.

So: Due to how the Lua C API is implemented, we have no way of knowing within the binding if the value is an integer or a float. And because we're a binding, we also don't have a way to know if a specific parameter is inside a LIMIT or OFFSET clause.
TL;DR: we're basically hosed?

lua_tointeger() ? Or am I missing something?

So in Lua 5.3+ we can probably push this burden onto the user, because Lua does differentiate between integers and non-integers itself now. So we just need to ensure we pass to MySQL as integers if(lua_isinteger(param)).
We then either drop support for Lua 5.2 and earlier, or just live with them not being able to use LIMIT/OFFSET in recent MySQL versions, or we go an extra mile and make some kind of compatibility API with a wrapper type for integers.
The second option seems the most reasonable to me.

So we just need to ensure we pass to MySQL as integers if(lua_isinteger(param)).

I swear I spent hours looking for exactly this and never found it :) This is exactly what is needed, at least for 5.3 and newer.

or we go an extra mile and make some kind of compatibility API with a wrapper type for integers.

I've been thinking about exactly that, for the simple reason that it may be useful in other unexpected contexts, such as emulating booleans, nil/NULL impedence mismatches - or possibly future cases unforseen. The overhead shouldn't be huge, if you don't need it don't use it.
Something like: DBI.forceinteger( 123 ) - which returns a Userdata that the underlying DBI layer knows how to interpret.
But IMHO we can, and should, move on full 5.3 integer support and worry about that compatibility API later. For correctness alone implement it in the other drivers as well.

Is a PR being prepared for this?
FreeBSD stopped including mysql 5.x in ports, hence I'm not even in the situation of compiling luadbi with it, so I could have mysql support for prosody.

For what it's worth, I can confirm MariaDB 10.6 is not impacted by this.
(I'm fully aware that doesn't mean much to a lot of people who have their reasons to use mainline MySQL.)

Finally managed to test #75 against MySQL 8.0.36 with Lua 5.4 and it worked flawlessly.
I'm reluctant to close this issue without further confirmation. Can someone who originally reported trouble, ie with Prosody, test and see if the problem is fixed?


Update for intellij 2023.2.0

Thanks for all the hard work. Seems I updated intellij to 2023.2.0 before checking if this plugin had ben updated.
When you get a chance to make it compatible, thanks

fixed by https://github.com/patricklx/intellij-emberjs-experimental/commit/a4b185df750091d5734a22ac789a324c2d8bd77f

Thanks, didnt expect so much code to bump a version. Or were there some features added?

You talking about this  https://youtrack.jetbrains.com/issue/IDEA-162901/Provide-LSP-API-for-external-plugin-developers#focus=Comments-27-7613792.0-0? Cause it looks like this release. Tried to update plugins, dont see this yet. Let me know when I can update. Trying to use the new template code and getting no syntax highlighting as the plugin is disabled for 2023.2.0

ah, right. but its still marked as experimental...
sure, it's gonna take a while, a few hours to 1 day until intellij approves the update

It's now available: https://plugins.jetbrains.com/plugin/15499-ember-experimental-js/versions/stable/370882🚀

Great, I have syntax highlighting again.
FYI, not sure if this is you or Intellij, but hash when imported and used always reports as an unused import

you mean this?
import { hash } from "@ember/helper";

which is working for me when used.
can you provide an example?

Hmmm, another person is seeing on the same way. Note we are js still, trying to get to typescript but having some addons clashing typescript versions when trying to add it to the project.

i created an issue for it.
https://github.com/patricklx/intellij-emberjs-experimental/issues/45
i think i have a better implementation for it now. will publish it soon

@cah-brian-gantzler is this fixed for you now?


group

Can you create a function to make a groupt of routers?
like
profile/show
profile/get
profile/edit
...
.
.

You want all those routes to share the same action? I'm not sure why you would want to do that as it will just make your code harder to follow, but if you want to do it like this then just change all the get, post, put, patch, delete functions to accept an array for the route, convert non array values to an array and loop though each route. Here's an example to get you started.
private function set($method, $route, $action) {
if (!is_array($route))
$route = [$route];
foreach ($route as $r)
$this->routes[$method][$r] = $action;
}
public function get($route, $action) {
$this->set('GET', $route, $action);
}
public function post($route, $action) {
$this->set('POST', $route, $action);
}
public function put($route, $action) {
$this->set('PUT', $route, $action);
}
public function patch($route, $action) {
$this->set('PATCH', $route, $action);
}

You want all those routes to share the same action? I'm not sure why you would want to do that as it will just make your code harder to follow, but here you go:
private function set($method, $route, $action) {
     if (!is_array($route))
          $route = [$route];
     foreach ($route as $r)
          $this->routes[$method][$r] = $action;
}

public function get($route, $action) {
     $this->set('GET', $route, $action);
}

public function post($route, $action) {
     $this->set('POST', $route, $action);
}

public function put($route, $action) {
     $this->set('PUT', $route, $action);
}

public function patch($route, $action) {
     $this->set('PATCH', $route, $action);
}

public function delete($route, $action) {
     $this->set('DELETE', $route, $action);
}

public function any($route, $action) {
     $this->set('GET', $route, $action);
     $this->set('POST', $route, $action);
     $this->set('PUT', $route, $action);
     $this->set('PATCH', $route, $action);
     $this->set('DELETE', $route, $action);
}

Now you can either specify a single route:
$Router->get('/example', 'views/example.php');
Or you can specify many:
$Router->get(
     [
          '/example',
          '/another-example',
          '/home/example',
          '/dynamic/$example'
     ],
     'views/example.php'
);


consider making Invoke virtual virtual on AuthenticationMiddleware

In cases such as some HMAC authentication algorithms, we need to inspect the stream during the Authentication Middleware.  In this case, it would be nice if we could go ahead and read the stream in only in the cases where it's necessary, replacing the body with a buffered stream.  This would prevent having to add buffered stream middleware in blanketly to every request, when it's only necessary in certain situations (e.g. certain Authroization header is present).

Hi @rposener, is you scenario achievable through customize a middleware and added before any authentication middleware?


AWS::Lambda::Function VpcConfig : Members must satisfy regular expression pattern: ^sg-[0-9a-zA-Z]*$] in vpcConfig.securityGroupIds

Is this feature request related to a new rule or cfn-lint capabilities?
rules
Describe the feature you'd like to request
CFT template fails with the below error message if security group id doesn't match the regular expression pattern ^sg-[0-9a-zA-Z]*$] in AWS::Lambda::Function VpcConfig.securityGroupIds
CREATE_FAILED: Resource handler returned message: Validation error detected: Value '<>' at 'vpcConfig.securityGroupIds' failed to satisfy constraint: Member must satisfy constraint: [Member must have length less than or equal to 102, Member must have length greater than or equal to 0, Members must satisfy regular expression pattern: ^sg-[0-9a-zA-Z]*$] (Service : Lambda, Status Code: 400)
Developers tend to provide security group names instead of security group ids which results in failure during CFN execution as it doesn't match the expected regex pattern
Describe the solution you'd like
Update CFN Lint to validate and report it as "Error" if the SG provided doesn't matches regex ^sg-[0-9a-zA-Z]*$].
Additional context
No work around.  It is one of the top occurring common issues when provisioning a Lambda function.
Is this something that you'd be interested in working on?

[ ] 👋 I may be able to implement this feature request

Would this feature include a breaking change?

[ ] ⚠ This feature might incur a breaking change

Please keep submitting these types of requests.  I will work as quick as I can to fix them.  Thank you!

I've been doing some work on this for v1.  I'm going to start tagging security group properties with format of AWS::EC2::SecurityGroup.GroupId this will allow me to more uniformly apply the pattern instead of having it copied it everywhere.  The interesting bit here is I have found 4-5 patterns some that allow upper case, some that don't, etc.

This is fixed in v1 and v0.


Don't normalize name when writing Pipfile

Signed-off-by: Frost Ming<EMAIL_ADDRESS>The fix
Don't canonicalize the name user passes in. Just make sure the name is safe. With that said, a package with dots in its name like fsc.export won't be normalized to fsc-export in Pipfile
Closes #3324
The impact
This PR doesn't fix the wrong package name which already exists in user's Pipfile. Taking the following examples:

Item
Content

Original name
requests-html

Canonical name(PEP-503)
requests-html

User command
$ pipenv install Requests_HTML

Entry in Pipfile
requests-html = "*"

Item
Content

Original name
fsc.export

Canonical name(PEP-503)
fsc-export

User command
$ pipenv install Fsc.Export

Entry in Pipfile
"fsc.export" = "*"

User command
$ pipenv install fsc-export

Entry in Pipfile
Installation fails, nothing is added

The checklist

[x] Associated issue
[x] A news fragment in the news/ directory to describe this fix with the extension .bugfix, .feature, .behavior, .doc. .vendor. or .trivial (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

Approach gets a +1 from me


Conver to md 1 to 5

The most changed post is "GLOBAL CODE RETREAT 2011 IN BERLIN" I have remove uppercase letters from title, I also have remove further reading and includes as links on the text because I didn't find a way to make link to a list in same file.
The rest of post are almost identical. Just change html tag by md  and add/remove line breaks.

I think this PR is ready to merge or close.


Fast refresh seems to crash values returned by input

This is visible in codesandbox here:
https://codesandbox.io/s/leva-ui-45bkg?file=/src/App.js
Click on new box, and then change something in the code.

This seems to be fixed now.


10013 clean up code

Added some more descriptive docstrings.
Fixed several PEP8 issues.
Added ability to run coverage if something exists in env "COVERAGE".
Added 'badges' to the readme.
Did not fix the JavaScript since we will be replacing it.

Changes Unknown when pulling 2ea5f72049585ad84266e86eed27c54e77781c0e on 10013-CleanUpCode into ** on master**.

Changes Unknown when pulling 739281d8860ead8aafa7a6fa595584cf6f6f9dd4 on 10013-CleanUpCode into ** on master**.


Print location of HTMLErrorReport when finished.

Would it be possible to print out the location of the HTMLErrorReport.html when finished and if it is generated.
I got this feature request a couple times in Fladle. 🤣  https://github.com/runningcode/fladle/issues/64

Would you be able to provide some more information? Is this a feature which would fit on your roadmap and something you have time for or would you prefer external contributions?
Should this feature be behind a flag, or enabled for all?

I fixed this in #612, it'll be in the next Flank release.

Would it be possible to print the fully qualified path? This makes it
easier to click on the HTML report from the terminal output.
bootstraponline<EMAIL_ADDRESS>schrieb am Fr., 11. Okt. 2019,
11:51:

[image: image]
https://user-images.githubusercontent.com/1173057/66669484-7a78a380-ec0c-11e9-9461-4067219be557.png
I fixed this in #612 https://github.com/TestArmada/flank/pull/612,
it'll be in the next Flank release.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/TestArmada/flank/issues/592?email_source=notifications&email_token=AACRGNLFOEEQGOONCIQTL43QOCVJLA5CNFSM4IQEBQKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBASRTI#issuecomment-541141197,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACRGNLPJUTKF5V57BSQ3D3QOCVJLANCNFSM4IQEBQKA
.

Would it be possible to print the fully qualified path?

Sure, I'll update to print the full path.

You on fire! :)
bootstraponline<EMAIL_ADDRESS>schrieb am Fr., 11. Okt. 2019,
12:02:

Would it be possible to print the fully qualified path?
Sure, I'll update to print the full path.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/TestArmada/flank/issues/592?email_source=notifications&email_token=AACRGNOUDI6PLOVLSJQOJWLQOCWSLA5CNFSM4IQEBQKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBATPCY#issuecomment-541144971,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACRGNM2ZRVNDPIHMYZV2IDQOCWSLANCNFSM4IQEBQKA
.


Stream encryption API

Something that i would like to see in this library is an API for encrypting a stream of messages.
This is for example useful if a file is too large to be encrypted at once.
It is impotent for such an API that messages can not be modified , dropped or reordered. This means the complete stream needs to be authenticated.
Its also a requirement, that the same message, if encrypted repeatedly, produces different ciphertexts.
Libsodium provides such a functionality based on XChaCha20Poly1305.
https://download.libsodium.org/doc/secret-key_cryptography/secretstream
This functionality can also easily implemented for orion. (I already have a working, hazardous implementation for it )
The main problem that i see is that this algorithm isn't standardized.
If you want this functionality  i can submit a pull request for this

Something that i would like to see in this library is an API for encrypting a stream of messages.
This is for example useful if a file is too large to be encrypted at once.

This is a great idea! I'm curious whether you plan on using such a feature anywhere specific?

The main problem that i see is that this algorithm isn't standardized.

XChaCha20Poly1305 is in the process of being standardised and is used quite a few other places as well. I see no issue with basing a streaming encryption API upon this. However, we could just as well add support for streaming ChaCha20Poly1305 encryption, should the non-standardisation cause concern to anyone.
The libsodium secretstream is a good baseline for implementing it in orion.

If you want this functionality i can submit a pull request for this

Yes, this definitely seems worthwhile to provide. I have looked into it before, but not thought about specific requirements for such an implementation, which means we might need to re-iterate over its design a few times. Since you mentioned you already have a working implementation, you could open it as WIP while we work through it?


add domain politeknik-sampit.ac.id

add domain politeknik-sampit.ac.id
Website : https://politeknik-sampit.ac.id/
Address : Jalan Tjilik Riwut, Komplek Perumahan Wengga Metropolitan Barat No.KM 3, Baamang Tengah, Kec. Baamang, Kabupaten Kotawaringin Timur, Kalimantan Tengah 74312 , Indonesia

@nurisakbar First stage of verification passed, official domain for Politeknik Sampit is politeknik-sampit.ac.id as far as I verified. This is just an informational note. Pull request is still in the review. I appreciate your patience while waiting for the final decision.

Hello @philipto , thank you very much for taking the time to review our request

@nurisakbar Thank you for your patience while waiting for a review and processing your request. Pull request merged.


Report spelling errors as info rather than warnings

In the vscode extension settings, if I set the log level to "info", it continues to report spelling errors as warnings.

Interesting, could you share a screenshot? I'm not sure how to replicate this behaviour.

In this extension, The setting is set to "info", but still returns a "warning" (yellow).

In the code spell checker extension, I get the report as "info" (blue)

I was originally using the code spell checker extension, but went looking for something with ess false positives, and your extension seems to do great at that. Having spelling errors marked as "info" instead of as a "warning", makes it easier to separate programing warnings from spelling mistakes.
This issue is also consistent under the "problems" tab of vscode.

Ah gotcha. So Typos: Log Level refers to the logging that appears in the Output -> Typos pane.
It sounds like you'd like to configure the diagnostic severity which isn't currently supported but is something that I'd be happy to add.

Ok, that makes sense. Thank you.

@AspieSoft I've released v0.1.7 which has the new setting typos.diagnosticSeverity. You can use this to configure diagnostics to display as Information instead of Warning.

I'm really sorry for bothering you all on this closed issue but I didn't find a better place where to ask 😞
Question: the diagnostics levels cannot be customized on NeoVim via nvim-lspconfig, only on VSCode, right?
Asking because I kind of tried everything but didn't managed to make the diagnostics being reported as anything different from errors.
Sorry for the dumb question and once again for the bother.
P.S: thank you really a lot for all you amazing work! 🙇♂

Hey @fusillicode it should be possible on any editor than can provide initialisation options to the lsp server. I'm sorry I don't know how to do that with neovim but if you find a way let us know 😊

@tekumara thanks really a lot for the quick feedback! 🙇♂
If I make it to work with Neovim I'll come back with the solution and maybe open a PR to document the thing! 👍
Thanks again for everything! ♾ 🙇♂

@nghialm269 thanks for the feedback but unfortunately it's not working on my side https://github.com/fusillicode/dotfiles/blob/39fa7e51d89924091703c9ebc68897e8cec5a468/nvim/lua/mason-tools.lua#L49-L53 🥲

@fusillicode: here is the mini config file from nvim-lspconfig:
local on_windows = vim.loop.os_uname().version:match("Windows")

local function join_paths(...)
	local path_sep = on_windows and "\\" or "/"
	local result = table.concat({ ... }, path_sep)
	return result
end

vim.cmd([[set runtimepath=$VIMRUNTIME]])

local temp_dir = vim.loop.os_getenv("TEMP") or "/tmp"

vim.cmd("set packpath=" .. join_paths(temp_dir, "nvim", "site"))

local package_root = join_paths(temp_dir, "nvim", "site", "pack")
local lspconfig_path = join_paths(package_root, "test", "start", "nvim-lspconfig")

if vim.fn.isdirectory(lspconfig_path) ~= 1 then
	vim.fn.system({ "git", "clone", "https://github.com/neovim/nvim-lspconfig", lspconfig_path })
end

vim.lsp.set_log_level("trace")
require("vim.lsp.log").set_format_func(vim.inspect)
local nvim_lsp = require("lspconfig")
local on_attach = function(_, bufnr)
	local function buf_set_option(...)
		vim.api.nvim_buf_set_option(bufnr, ...)
	end

	buf_set_option("omnifunc", "v:lua.vim.lsp.omnifunc")

	-- Mappings.
	local opts = { buffer = bufnr, noremap = true, silent = true }
	vim.keymap.set("n", "gD", vim.lsp.buf.declaration, opts)
	vim.keymap.set("n", "gd", vim.lsp.buf.definition, opts)
	vim.keymap.set("n", "K", vim.lsp.buf.hover, opts)
	vim.keymap.set("n", "gi", vim.lsp.buf.implementation, opts)
	vim.keymap.set("n", "<C-k>", vim.lsp.buf.signature_help, opts)
	vim.keymap.set("n", "<space>wa", vim.lsp.buf.add_workspace_folder, opts)
	vim.keymap.set("n", "<space>wr", vim.lsp.buf.remove_workspace_folder, opts)
	vim.keymap.set("n", "<space>wl", function()
		print(vim.inspect(vim.lsp.buf.list_workspace_folders()))
	end, opts)
	vim.keymap.set("n", "<space>D", vim.lsp.buf.type_definition, opts)
	vim.keymap.set("n", "<space>rn", vim.lsp.buf.rename, opts)
	vim.keymap.set("n", "gr", vim.lsp.buf.references, opts)
	vim.keymap.set("n", "<space>e", vim.diagnostic.open_float, opts)
	vim.keymap.set("n", "[d", vim.diagnostic.goto_prev, opts)
	vim.keymap.set("n", "]d", vim.diagnostic.goto_next, opts)
	vim.keymap.set("n", "<space>q", vim.diagnostic.setloclist, opts)
end

-- Add the server that troubles you here
local name = "typos_lsp"
local cmd = { '/home/nghialm/.local/share/nvim/mason/bin/typos-lsp' } -- needed for elixirls, lua_ls, omnisharp
if not name then
	print("You have not defined a server name, please edit minimal_init.lua")
end
if not nvim_lsp[name].document_config.default_config.cmd and not cmd then
	print([[You have not defined a server default cmd for a server
    that requires it please edit minimal_init.lua]])
end

nvim_lsp[name].setup({
	cmd = cmd,
	on_attach = on_attach,
	init_options = {
		diagnosticSeverity = 'Warning',
	},
})

-- test typo symbo

save to mini.lua, change the path to typos-lsp and run neovim with: nvim --clean -u mini.lua mini.lua
you should see the warning typo in the last line (symbo)

@nghialm269 thanks so much for coming back so quickly and with that extensive example! ♾ 🙇♂
I think now I understand the issue 🙏
It's just a matter of fixing my setup 🥲
Thanks again!

To anyone who will bump into this, my issue was fixed thanks to this.
More specifically the diagnosticSeverity must be passed to the LSP (typos) as part of init_options, not settings (https://github.com/neovim/nvim-lspconfig/blob/39546f730bdff8eccf7cec344cfce694f19ac908/CONTRIBUTING.md?plain=1#L39).
I managed to do this ☝ like this.
♾ thanks once again to @nghialm269 🙏 🙇♂


French translation update

Thank you @nicobubulle for providing these translation updates, Will merge in now for the next release.


[now-rust][wip] Support now-dev

As of right now, now-rust doesn't work very well with now-dev (at all, really). There are several things that need to happen for it to work semi-well.

[x] don't compile with --release during dev
[ ] watch relevant rust files for changes to re-compile
[ ] stop overwriting .rs entrypoints (#3)
[ ] windows support (#9)

Not sure if it's in the scope of this PR, but now-dev also fails on macOS, because of yum (and I assume would fail on any os/distro that doesn't have yum).
$ now dev
> Now CLI 16.3.1 dev (beta) — https://zeit.co/feedback/dev
> Ready! Available at http://localhost:3000
> GET /api/exists/Cargo.toml
> Building now-rust:api/exists/Cargo.toml
downloading the rust toolchain
info: downloading installer
info: profile set to 'default'
info: updating existing rustup installation

Rust is installed now. Great!

To get started you need Cargo's bin directory ($HOME/.cargo/bin) in your PATH
environment variable. Next time you log in this will be done
automatically.

To configure your current shell run source $HOME/.cargo/env
installing openssl-devel...
failed to `yum install -y openssl-devel`
Error: spawn yum ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:264:19)
    at onErrorNT (internal/child_process.js:456:16)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  message: 'spawn yum ENOENT',
  errno: 'ENOENT',
  code: 'ENOENT',
  syscall: 'spawn yum',
  path: 'yum',
  spawnargs: [ 'install', '-y', 'openssl-devel' ],
  stdout: null,
  stderr: null,
  failed: true,
  signal: null,
  cmd: 'yum install -y openssl-devel',
  timedOut: false,
  killed: false
}

@amaurymartiny that should have been resolved quite a while ago. Are you still using @now/rust in your now.json file?

@mike-engel I'm actually experiencing the same issue that @amaurymartiny is experiencing. Here's my now.json file:
{
  "functions": {
    "api/**/*.rs": {
      "runtime"<EMAIL_ADDRESS>    }
  },
  "routes": [{ "src": "/", "dest": "/api/image.rs" }]
}


Getting only an empty screen when include the ace editor component

I am running my app in an electron environment packaged by webpack.
I pasted the code sample in the example :
<script>
  import AceEditor from "svelte-ace-editor";

  if(process.browser) {
  require("brace");
  // require('brace/ext/language_tools')
  // require('brace/ext/language_tools')
  // require('brace/mode/pgsql')
  // require('brace/mode/mysql')
  // require('brace/mode/sql')
  // require('brace/mode/sqlserver')
  require("brace/theme/clouds_midnight");
}
  let value = "";
  let lang = "sql";
  let options = {};

  function editorInit(editor) {
    // @todo something with editor or something after init
  }
  function onEditorChange(newValue) {
    value = newValue.detail;
  }
</script>

<div>
  <AceEditor
    bind:value
    theme="clouds_midnight"
    {lang}
    {options}
    width="100%"
    height="512"
    on:init={editorInit}
    on:input={onEditorChange} />
</div>

But in my electron app, i just get an empty screen with an empty div that takes the height 512 and full width of screen.
how do i fix this ? please help..

It seem you had changed the code in onMount from checking process.browser to always being true.
but you haven't published the latest code on npm.
can you please publish this ?


Request the next version available in crates.io

Problem description
Request the next version available in crates.io(0.20.1 ?)

Ow! I see this is kinda a open problem https://github.com/tree-sitter/tree-sitter/discussions/1768
Let me know if there is anything i can do from my end.
I can see this https://github.com/tree-sitter/tree-sitter-java/blob/master/.github/workflows/publish_crate.yml , but i am not sure if I should try to invoke it (or even if I can if I try, I might night have those access)

New release is out: https://github.com/tree-sitter/tree-sitter-java/releases/tag/v0.20.2


Carousel Autoslide didn't work like it supposed to

Hey, I'm new to swiper JS. I followed the documentation about autoplay = {{delay: 5000}} but it didn't work.

Well, it'd been fixed, no problemo.


Add thumbnails relationship to driveItem

See
https://learn.microsoft.com/en-us/graph/api/resources/driveitem?view=graph-rest-1.0
https://learn.microsoft.com/en-us/graph/api/resources/thumbnailset?view=graph-rest-1.0
https://learn.microsoft.com/en-us/graph/api/resources/thumbnail?view=graph-rest-1.0
Tbh I'm not sure about the thumbnailSet definition, while I copied the exact definition from the thumbnailSet documentation - this page https://learn.microsoft.com/en-us/graph/api/driveitem-list-thumbnails?view=graph-rest-1.0&tabs=http#size-options lists more options and they are partially contradicting (at least the large thumbnail is said to be 1920x1920 or 800 longest on the other page)

Thoughts on this one?

The idea in general is a good one.
I would like to collect requirements on a broader scope for that topic.
My questions are:

How could we implement that seriuosly? (postprocessing step writes thumbnails and stores info in file metadata)?
How could existing clients make use of that feature?

I would like to follow up on this:

for starters I would implement it just like audio metadata. Extract it in the search service, write it into arbitrary file metadata

only write the original size into the metadata, we can scale to any size anyhow

web can make use of it by checking whether a thumbnail is available and only then make a request for the thumbnail image (saves some inexpensive requests and avoids error requests in the browser console)

Any chance to make progress here any time soon?

@butonic @micbar I know that this doesn't have high prio at the moment... but since it can be useful for web and it's already defined in ms graph I don't see an issue bringing it to the libre graph api as well. What do you think?

ping @butonic @micbar

The idea is to save web requests if thumbnails cannot be generated. Web will only request thumbnails if the driveItem contains a thumbnail. On the server side the thumbnail metadata is created asynchronously: if we can create a thumbnail we persist the sizes in the file metadata so we can efficiently list files.
This can be done by the search service, which already reads all file to extract metadata for the audio, video and photo facets.
The thumbnailer could also be put into the postprocessing to precreate thumbnails, but that is a different topic.
In any case the thumbnail metadata is stored in the file metadata.
We can also emit an event when the postprocessing has created the thumbnail. Then web can just wait for a thumbnail ready event.
For now this is an investment into the future where we want to use the graph api to navigate the tree.


allow-notify-from may be empty too

@marknl I think you need to target the 'develop' branch with your PR.  That's where mine was recently merged to.

I touch the same code in https://github.com/sensson/puppet-powerdns/pull/22/files#diff-0ac866172efd29d0ddc2aa75ae6f182dR9

@marknl I've changed target of this merge request. Could you look at the conflicts @alexjfisher mentioned?
This will also make sure the tests in Travis will pass again as that was solved in #21.
I will make sure we'll release a new version of this module shortly after.

Fixed! Next time i'll use better branch names. ;)

Awesome, thanks @marknl!


Error: Library not loaded: @rpath/libpython3.11.dylib

The above file, post-installation of Pipeless following the guidelines, is unavailable in the .pipeless directory. The only files in the directory are: device_id, ibonnxruntime.1.16.0.dylib, libonnxruntime.dylib.
My system has Python installed already; further, I downgraded it to 3.11 for compatibility with Pipeless. What is the solution to this?

Hi @paakhim10 ,
Could you share some logs or more context about the error?
Also, Pipeless is compatible with Python 3.12 in macOS. I assume you are using macOS since you mention the dylib files.

Hi @miguelaeh! Thanks so much for responding.
pipeless_log.txt
This is the terminal log file. For context, the prior Python packages are Conda installed, I am not sure if it interferes with Pipeless installation.

I've now tried installing it using Docker. Despite that, the package issue still occurs.
For resolution, I've replaced the keyword pipeless with docker run miguelaeh/pipeless. For example: if I want to execute pipeless --help, I execute docker run miguelaeh/pipeless --help. This method does not throw the error of the unloaded library.
For creating a project using this method, I did the following: docker run miguelaeh/pipeless init project_name --template empty
This does not allow me to navigate to the project director, i.e., cd project_name throws an error of no such directory existing.
Could you offer an alternate solution to this? I am not sure how to proceed with building the project.

Hi @paakhim10 ,
I think it is not related to Python packages but with the Python installation. libpython3.11 is basically the main Python interpreter library.
You can try to build it from source as described in the following link. In this way Pipeless will take whatever Python you have installed in your system: https://www.pipeless.ai/docs/docs/v1/getting-started/installation#macos

I had tried that earlier because of the same reasoning, but that's when I was encountering the error. I've now run the project using the docker image, and the dylib error is resolved. Thank you, @miguelaeh!

Hi, @miguelaeh
The docker image method is encountering errors with the server connection. Could you look into the following code and explain why the dylib error occurs?
curl https://raw.githubusercontent.com/pipeless-ai/pipeless/main/install.sh | bash
This installs Pipeless successfully with the following warning:
warning: trait EventTypeis never used  --> src/events.rs:8:7   | 8 | trait EventType {}   |       ^^^^^^^^^   |   = note:#[warn(dead_code)]` on by default
warning: pipeless-ai (lib) generated 1 warning
Finished release profile [optimized] target(s) in 1m 13s
Pipeless was properly built
strip: changes being made to the file will invalidate the code signature in: /private/var/folders/tn/5qwnwx_j20b_5qhwx5vg07840000gn/T/tmp.O6rTKnzFSS/pipeless/target/release/pipeless
[cctools-port]: generating fake signature for '/private/var/folders/tn/5qwnwx_j20b_5qhwx5vg07840000gn/T/tmp.O6rTKnzFSS/pipeless/target/release/pipeless.strip'
✅ Pipeless installed successfully!
Execute the following commands to load the Pipeless binary:
export PATH="${PATH}:${HOME}/.pipeless"

[cctools-port]: generating fake signature for '/Users/paakhim10/.pipeless/pipeless'`
On executing pipeless:
(base) paakhim10@Paakhis-MacBook-Air ~ % pipeless dyld[21163]: Library not loaded: @rpath/libpython3.11.dylib   Referenced from: <18C153A2-4CC2-3D23-A4DD-75BAEC6E9BED> /Users/paakhim10/.pipeless/pipeless   Reason: tried: '/Users/paakhim10/.pipeless/libpython3.11.dylib' (no such file), '/Users/paakhim10/.pipeless/libpython3.11.dylib' (no such file) zsh: abort      pipeless
I've tried building it through different methods and used the docker container images for building my project as well, but I can not get any example to run on my system. Thank you so much!


Fixed detection if user logged in already or not updates screen

Fixed the screen updating the options to LOGOUT if user is logged in already or LOGIN and REG if user not logged in already.
Needed to add a synch and await to axios.

@Cjunk could you please resolve these conflicts, not sure which one ended up working for you in the end


pocketbase: 0.18.10 -> 0.19.2

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://github.com/pocketbase/pocketbase/releases.
meta.description for pocketbase is: Open Source realtime backend in 1 file
meta.homepage for pocketbase is: https://github.com/pocketbase/pocketbase
Updates performed

Golang update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.19.2 with grep in /nix/store/59kh2kzpbgr9jrilw58a48sz6mdn8rkg-pocketbase-0.19.2
found 0.19.2 in filename of file in /nix/store/59kh2kzpbgr9jrilw58a48sz6mdn8rkg-pocketbase-0.19.2

Rebuild report (if merged into master) (click to expand)

1 total rebuild path(s)

1 package rebuild(s)

First fifty rebuilds by attrpath
pocketbase

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/59kh2kzpbgr9jrilw58a48sz6mdn8rkg-pocketbase-0.19.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A pocketbase https://github.com/r-ryantm/nixpkgs/archive/15ccb5d7c395f6525e183332c63f18f750283ad7.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/15ccb5d7c395f6525e183332c63f18f750283ad7#pocketbase

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/59kh2kzpbgr9jrilw58a48sz6mdn8rkg-pocketbase-0.19.2
ls -la /nix/store/59kh2kzpbgr9jrilw58a48sz6mdn8rkg-pocketbase-0.19.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

pocketbase

Maintainer pings
cc @dit7ya @thilobillerbeck for testing.

Result of nixpkgs-review pr 265238 run on aarch64-darwin 1

1 package built:

pocketbase