[SPARK-23030][SQL][PYTHON] Use Arrow stream format for creating from and collecting Pandas DataFrames

What changes were proposed in this pull request?
This changes the calls of toPandas() and createDataFrame() to use the Arrow stream format, when Arrow is enabled.  Previously, Arrow data was written to byte arrays where each chunk is an output of the Arrow file format.  This was mainly due to constraints at the time, and caused some overhead by writing the schema/footer on each chunk of data and then having to read multiple Arrow file inputs and concat them together.
Using the Arrow stream format has improved these by increasing performance, lower memory overhead, and simplified the code.  Here are the details of this change:
toPandas()
Before:
Spark internal rows are converted to Arrow file format, each group of records is a complete Arrow file which contains the schema and other metadata.  Next a collect is done and an Array of Arrow files is the result.  After that each Arrow file is sent to Python driver which then loads each file and concats them to a single Arrow DataFrame.
After:
Spark internal rows are converted to ArrowRecordBatches directly, which is the simplest Arrow component for IPC data transfers.  The driver JVM then immediately starts serving data to Python as an Arrow stream, sending the schema first. It then starts Spark jobs with a custom handler such that when a partition is received the ArrowRecordBatches can be sent to python as soon as possible.  This improves performance, simplifies memory usage on executors, and improves the average memory usage on the JVM driver.  Since the order of partitions must be preserved, the order of partitions received is stored and then sent to Python to re-order the batches correctly.
createDataFrame()
Before:
A Pandas DataFrame is split into parts and each part is made into an Arrow file.  Then each file is prefixed by the buffer size and written to a temp file.  The temp file is read and each Arrow file is parallelized as a byte array.
After:
A Pandas DataFrame is split into parts, then an Arrow stream is written to a temp file where each part is an ArrowRecordBatch.  The temp file is read as a stream and the Arrow messages are examined.  If the message is an ArrowRecordBatch, the data is saved as a byte array.  After reading the file, each ArrowRecordBatch is parallelized as a byte array.  This has slightly more processing than before because we must look each Arrow message to extract the record batches, but performance ends up a litle better.  It is cleaner in the sense that IPC from Python to JVM is done over a single Arrow stream.
How was this patch tested?
Added new unit tests for the additions to ArrowConverters in Scala, existing tests for Python.
Performance Tests - toPandas
Tests run on a 4 node standalone cluster with 32 cores total, 14.04.1-Ubuntu and OpenJDK 8
measured wall clock time to execute toPandas() and took the average best time of 5 runs/5 loops each.
Test code
df = spark.range(1 << 25, numPartitions=32).toDF("id").withColumn("x1", rand()).withColumn("x2", rand()).withColumn("x3", rand()).withColumn("x4", rand())
for i in range(5):
	start = time.time()
	_ = df.toPandas()
	elapsed = time.time() - start

Current Master
This PR

5.803557
4.342533

5.409119
4.399408

5.493509
4.468471

5.433107
4.36524

5.488757
4.373791

Avg Master
Avg This PR

5.5256098
4.3898886

Speedup of 1.258712989
Performance Tests - createDataFrame
Tests run on a 4 node standalone cluster with 32 cores total, 14.04.1-Ubuntu and OpenJDK 8
measured wall clock time to execute createDataFrame() and get the first record. Took the average best time of 5 runs/5 loops each.
Test code
def run():
	pdf = pd.DataFrame(np.random.rand(10000000, 10))
	spark.createDataFrame(pdf).first()

for i in range(6):
	start = time.time()
	run()
	elapsed = time.time() - start
	gc.collect()
	print("Run %d: %f" % (i, elapsed))

Current Master
This PR

6.234608
5.665641

6.32144
5.3475

6.527859
5.370803

6.95089
5.479151

6.235046
5.529167

Avg Master
Avg This PR

6.4539686
5.4784524

Speedup of 1.178064192
Memory Improvements
toPandas()
The most significant improvement is reduction of the upper bound space complexity in the JVM driver.  Before, the entire dataset was collected in the JVM first before sending it to Python.  With this change, as soon as a partition is collected, the result handler immediately sends it to Python, so the upper bound is the size of the largest partition.  Also, using the Arrow stream format is more efficient because the schema is written once per stream, followed by record batches.  The schema is now only send from driver JVM to Python.  Before, multiple Arrow file formats were used that each contained the schema.  This duplicated schema was created in the executors, sent to the driver JVM, and then Python where all but the first one received are discarded.
I verified the upper bound limit by running a test that would collect data that would exceed the amount of driver JVM memory available.  Using these settings on a standalone cluster:
spark.driver.memory 1g
spark.executor.memory 5g
spark.sql.execution.arrow.enabled true
spark.sql.execution.arrow.fallback.enabled false
spark.sql.execution.arrow.maxRecordsPerBatch 0
spark.driver.maxResultSize 2g

Test code:
from pyspark.sql.functions import rand
df = spark.range(1 << 25, numPartitions=32).toDF("id").withColumn("x1", rand()).withColumn("x2", rand()).withColumn("x3", rand())
df.toPandas()

This makes total data size of 33554432×8×4 =<PHONE_NUMBER>
With the current master, it fails with OOM but passes using this PR.
createDataFrame()
No significant change in memory except that using the stream format instead of separate file formats avoids duplicated the schema, similar to toPandas above.  The process of reading the stream and parallelizing the batches does cause the record batch message metadata to be copied, but it's size is insignificant.

This is a WIP because I had to hack up some of the message processing code in Arrow.  This should be done in Arrow, and then it can be cleaned up here.  I will make these changes for version 0.10.0 and complete this once we have upgraded.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3956/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/67/
Test PASSed.

Test build #91733 has started for PR 21546 at commit a5a1fbe.

Test build #91733 has finished for PR 21546 at commit a5a1fbe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91733/
Test PASSed.

cc @ueshin

Test build #91862 has started for PR 21546 at commit 4af58f9.

Thanks @viirya and @HyukjinKwon !

cc @icexelloss

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4041/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/150/
Test PASSed.

Test build #91862 has finished for PR 21546 at commit 4af58f9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ArrowStreamSerializer(Serializer):

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91862/
Test PASSed.

Btw, do you have benchmark number or something yet?

++1

Thanks @ueshin and @icexelloss !  I do have some benchmarks that I'll post soon..

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/423/
Test PASSed.

Test build #92233 has finished for PR 21546 at commit b971e42.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92233/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/523/
Test PASSed.

Test build #92395 has started for PR 21546 at commit fe3319b.

UPDATE
I changed toPandas to write out of order partitions to python as they come in, followed by a list of indices to represent the correct batch order.  In python, the batches are then put in the correct order to make the Arrow Table / Pandas DataFrame.  This is slightly more complicated than before because we are sending extra info to Python, but it significantly reduces the upper-bound space complexity in the driver JVM from all data to the size of the largest partition.  It also seems to be a little faster, so I re-ran the performance tests, which I'll post now.

Performance Tests - toPandas
Tests run on a 4 node standalone cluster with 32 cores total, 14.04.1-Ubuntu and OpenJDK 8
measured wall clock time to execute toPandas() and took the average best time of 5 runs/5 loops each.
Test code
df = spark.range(1 << 25, numPartitions=32).toDF("id").withColumn("x1", rand()).withColumn("x2", rand()).withColumn("x3", rand()).withColumn("x4", rand())
for i in range(5):
	start = time.time()
	_ = df.toPandas()
	elapsed = time.time() - start

Current Master
This PR

5.803557
4.342533

5.409119
4.399408

5.493509
4.468471

5.433107
4.36524

5.488757
4.373791

Avg Master
Avg This PR

5.5256098
4.3898886

Speedup of 1.258712989

Memory Improvements
toPandas()
The most significant improvement is reduction of the upper bound space complexity in the JVM driver.  Before, the entire dataset was collected in the JVM first before sending it to Python.  With this change, as soon as a partition is collected, the result handler immediately sends it to Python, so the upper bound is the size of the largest partition.  Also, using the Arrow stream format is more efficient because the schema is written once per stream, followed by record batches.  The schema is now only send from driver JVM to Python.  Before, multiple Arrow file formats were used that each contained the schema.  This duplicated schema was created in the executors, sent to the driver JVM, and then Python where all but the first one received are discarded.
I verified the upper bound limit by running a test that would collect data that would exceed the amount of driver JVM memory available.  Using these settings on a standalone cluster:
spark.driver.memory              1g
spark.executor.memory            5g
spark.sql.execution.arrow.enabled  true
spark.sql.execution.arrow.fallback.enabled  false
spark.sql.execution.arrow.maxRecordsPerBatch 0
spark.driver.maxResultSize 2g

Test code:
from pyspark.sql.functions import rand
df = spark.range(1 << 25, numPartitions=32).toDF("id").withColumn("x1", rand()).withColumn("x2", rand()).withColumn("x3", rand())
df.toPandas()

This makes total data size of 33554432×8×4 =<PHONE_NUMBER>
With the current master, it fails with OOM but passes using this PR.
createDataFrame()
No significant change in memory except that using the stream format instead of separate file formats avoids duplicated the schema, similar to toPandas above.  The process of reading the stream and parallelizing the batches does cause the record batch message metadata to be copied, but it's size is insignificant.

Test build #92395 has finished for PR 21546 at commit fe3319b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class BatchOrderSerializer(Serializer):

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92395/
Test PASSed.

Hey @BryanCutler, btw, mind i fI ask move the benchmarks into the PR description?

Test build #92432 has started for PR 21546 at commit f6d6620.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/551/
Test PASSed.

Test build #92432 has finished for PR 21546 at commit f6d6620.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92432/
Test PASSed.

Hey @BryanCutler, btw, mind if I ask move the benchmarks into the PR description?

Yeah, I guess it doesn't hurt.  I'll copy them there.

Test build #92487 has started for PR 21546 at commit 8b814b7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/593/
Test PASSed.

Test build #92487 has finished for PR 21546 at commit 8b814b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92487/
Test PASSed.

Test build #92543 has started for PR 21546 at commit e25acd2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/628/
Test PASSed.

Test build #92543 has finished for PR 21546 at commit e25acd2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92543/
Test PASSed.

Test build #93137 has started for PR 21546 at commit 1308169.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1025/
Test PASSed.

I removed the WIP because it does not require anything from Arrow 0.10.0, this would only help clean some thing up. @HyukjinKwon, @ueshin what are your thoughts on merging this as is and then doing a followup to clean once Arrow 0.10.0 is released?  There is a chance that we might not get the Arrow upgrade in before Spark 2.4.0, so it would still be nice to get this in.  Thanks!

Test build #93137 has finished for PR 21546 at commit 1308169.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93137/
Test PASSed.

No objection. For me, I need few days to take another look. From a cursory look, seems fine going ahead considering this is an experimental feature and still in progress.
Please go ahead otherwise will take a look and go ahead in few days.

@BryanCutler, this takes longer then I thought. Will complete my review till this week. For clarification, still no objection about merging it in orthogonally with my review.

@BryanCutler, this takes longer then I thought. Will complete my review till this weekend. For clarification, still no objection about merging it in orthogonally with my review.

No problem, thanks @HyukjinKwon !

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1242/
Test PASSed.

Test build #93455 has started for PR 21546 at commit f19f204.

Test build #93455 has finished for PR 21546 at commit f19f204.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93455/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1248/
Test PASSed.

Test build #93465 has started for PR 21546 at commit 3224625.

SGTM except the discussion going on in https://github.com/apache/spark/pull/21546#discussion_r204324646

Test build #93465 has finished for PR 21546 at commit 3224625.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93465/
Test PASSed.

so... can we still target this for 2.4? sounds like very good to have and complimentary to hydrogen

Yea.. I hope so. only one comment left.

Test build #95055 has started for PR 21546 at commit 92b8e26.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2399/
Test PASSed.

@HyukjinKwon @felixcheung , I reverted the out-of-order batching for now (we can look at that in another PR) and cleaned up things after the Arrow 0.10.0 upgrade.  If this looks good to you guys, I'd like to rerun some of the benchmarks double-check before merging. Thanks!

Test build #95055 has finished for PR 21546 at commit 92b8e26.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95055/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2423/
Test PASSed.

Test build #95083 has started for PR 21546 at commit 89d7836.

Test build #95083 has finished for PR 21546 at commit 89d7836.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class BarrierTaskContext(TaskContext):
class BarrierTaskInfo(object):
case class StateStoreCustomSumMetric(name: String, desc: String) extends StateStoreCustomMetric
sealed trait StreamingAggregationStateManager extends Serializable
abstract class StreamingAggregationStateManagerBaseImpl(
class StreamingAggregationStateManagerImplV1(
class StreamingAggregationStateManagerImplV2(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95083/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2433/
Test PASSed.

Test build #95093 has started for PR 21546 at commit 89d7836.

Test build #95093 has finished for PR 21546 at commit 89d7836.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class BarrierTaskContext(TaskContext):
class BarrierTaskInfo(object):
case class StateStoreCustomSumMetric(name: String, desc: String) extends StateStoreCustomMetric
sealed trait StreamingAggregationStateManager extends Serializable
abstract class StreamingAggregationStateManagerBaseImpl(
class StreamingAggregationStateManagerImplV1(
class StreamingAggregationStateManagerImplV2(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95093/
Test PASSed.

LGTM otherwise.

Test build #95137 has finished for PR 21546 at commit 5549644.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95137/
Test FAILed.

retest this please

Test build #95145 has started for PR 21546 at commit 5549644.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2480/
Test PASSed.

Test build #95145 has finished for PR 21546 at commit 5549644.

This patch passes all tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95145/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2578/
Test PASSed.

Test build #95296 has started for PR 21546 at commit 2fe46f8.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2588/
Test PASSed.

Test build #95310 has started for PR 21546 at commit 2fe46f8.

Test build #95310 has finished for PR 21546 at commit 2fe46f8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95310/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2599/
Test PASSed.

Test build #95325 has started for PR 21546 at commit 2fe46f8.

Test build #95325 has finished for PR 21546 at commit 2fe46f8.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95325/
Test FAILed.

Merged build finished. Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2610/
Test PASSed.

Test build #95340 has started for PR 21546 at commit 2fe46f8.

Test build #95340 has finished for PR 21546 at commit 2fe46f8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95340/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2629/
Test PASSed.

Test build #95363 has started for PR 21546 at commit 2fe46f8.

Test build #95363 has finished for PR 21546 at commit 2fe46f8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95363/
Test FAILed.

Test build #95390 has started for PR 21546 at commit ffb47cb.

It's not a bug. Here's what's going on now:
He proposed a way to write out regardless of partition order, I suggested to revert that part back (see https://github.com/apache/spark/pull/21546#discussion_r204324646) since the code path looks being complicated and looks that can orthogonally proceed, which can be also applied to other PySpark code path.
This one still has some improvements (see https://github.com/apache/spark/pull/21546#issuecomment-416373315).

@gatorsmile , this is just the format for Arrow IPC between the JVM and Python process and although it used the Arrow File format, there is nothing persisted. There is no real reason to keep both formats, the stream format is better for our purposes and it's already what is being used for pandas_udfs, so there is unlikely a bug in the Arrow format itself. As with any change, a bug is possible but this has been tested pretty thouroughly and trying to keep the old code would get really messy and complicated.

@BryanCutler The worst case is to turn off spark.sql.execution.arrow.enabled, if the new code path has a bug, right?

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/2655/
Test PASSed.

Test build #95397 has started for PR 21546 at commit ffb47cb.

@HyukjinKwon I redid the benchmarks for toPandas with the current code and updated the description. It's not a huge speedup now, but definitely does improve some. I'll also followup with another PR with the out-of-order batches to improve this even further. Let me know if this looks ok to you (pending tests). Thanks!

Test build #95390 has finished for PR 21546 at commit ffb47cb.

This patch fails from timeout after a configured wait of `400m`.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ArrowEvalPython(udfs: Seq[PythonUDF], output: Seq[Attribute], child: LogicalPlan)
case class BatchEvalPython(udfs: Seq[PythonUDF], output: Seq[Attribute], child: LogicalPlan)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95390/
Test FAILed.

Merged build finished. Test FAILed.

Test build #95397 has finished for PR 21546 at commit ffb47cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ArrowEvalPython(udfs: Seq[PythonUDF], output: Seq[Attribute], child: LogicalPlan)
case class BatchEvalPython(udfs: Seq[PythonUDF], output: Seq[Attribute], child: LogicalPlan)

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95397/
Test PASSed.

Merged to master.

Thanks @HyukjinKwon and others who reviewed!

@gatorsmile, I'm working on test coverage report and already almost finished. Only left job is to setup a Jenkins job as I talked. cc @shaneknapp

For now, you can ruin https://github.com/apache/spark/blob/master/python/run-tests-with-coverage script to check the coverage.


Questionnaire a

What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports)
What did you learn from the changes you have made?
Was there another approach you were thinking about making? If so, what approach(es) were you thinking of?
Were there any challenges? If so, what issue(s) did you face? How did you overcome it?
How were these changes tested?
A reference to a related issue in your repository (if applicable)
Checklist

[ ] I can confirm that my changes are working as intended

A2: PASS
Next time I would take it a step further and describe how to split strata's


Can't craft items (or can't see recipes in JEI)

The title says it all. I can't craft any of the items from the mods (or can't see the recipes in JEI, which I assume means the craft don't work). Let me know if I'm in the wrong.

what version of the mod/minecraft are you using?

fixed in 1.1.1


Implement TypesIntersect(a, b *types.Type) bool

fixes: #3203

fyi: @aboodman. @rafael-atticlabs, let me know if you'd like someone else to review.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@d1244c1). Click here to learn what that means.
The diff coverage is 95.83%.

Impacted Files
Coverage Δ

go/types/intersect.go
95.83% <95.83%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1244c1...45ca907. Read the comment docs.

PTAL @rafael-atticlabs

How is this different from IsSubtype?

@arv
TypesIntersect(Struct { a: Number, b: String }, Struct { a: Number, c: Bool }) == true

Can we use another word to describe this. Type intersection is already a well defined term in type theory.

Note that this doesn't return a new type... it just returns whether the two
types do intersect or not. Is this not type intersection from theory?
On Fri, Feb 24, 2017 at 7:37 AM Erik Arvidsson<EMAIL_ADDRESS>wrote:

Can we use another word to describe this. Type intersection is already a
well defined term in type theory.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/attic-labs/noms/pull/3223#issuecomment-282321733, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAE6BHIfiCtCgKC-QNimUah1SFfLF9EDks5rfvk0gaJpZM4MKmgO
.

You can always get the intersection of two different types but there might not be any values that fit in the type intersection.
type T = number & string;
const n: T = ???;

There is no value that can be a number and a string....
type T2 = {b: string} & {a: number};
const n2: T2 = {a: 42, b: 'a'}

Given this it is not clear what it means that {a: number, b: string} & {b: string, c: boolean} returns true? Maybe the question we are asking is if there is a common field in two types?

I think the terminology needs some tweaking. The question we are asking is the type may help get us towards the other type. In other words if there is a common (non trivial) super type
containsCommonSuperType
mayContribute{To,Towards}

Per meatspace discussion with @arv, let's change the name of this to be ContainCommonSupertype(a, b *types.Type) bool

I like that name a lot!
On Fri, Feb 24, 2017 at 10:18 AM, Rafael Weinstein<EMAIL_ADDRESS>
wrote:

Per meatspace discussion with @arv https://github.com/arv, let's change
the name of this to be ContainCommonSupertype(a, b *types.Type) bool
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/attic-labs/noms/pull/3223#issuecomment-282364199, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAE6BK8id_UT8f-w3VL26bK_RUhvY_Hhks5rfx8EgaJpZM4MKmgO
.

Except for it it's not grammatical. Should be ContainsCommonSupertype
(plural)
On Fri, Feb 24, 2017 at 10:30 AM, Aaron Boodman<EMAIL_ADDRESS>wrote:

I like that name a lot!
On Fri, Feb 24, 2017 at 10:18 AM, Rafael Weinstein <
<EMAIL_ADDRESS>wrote:

Per meatspace discussion with @arv https://github.com/arv, let's
change the name of this to be ContainCommonSupertype(a, b *types.Type)
bool
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/attic-labs/noms/pull/3223#issuecomment-282364199,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAE6BK8id_UT8f-w3VL26bK_RUhvY_Hhks5rfx8EgaJpZM4MKmgO
.

@aboodman: Sounds good, but I think ContainCommonSupertype is correct since it refers to two arguments (i.e. a and b contain common supertypes)

there are two things you are asking the question about. e.g. "a and b contain a common supertype".

 My mistake.


HTTP Error 500.19 - Internal Server Error

I have followed your instructions as carefully as I was able; some of the instructions are a little confusing.
However, I have it "installed" on a Windows 2012 R2 Standard Server with IIS 8.5. And I'm using PassCore 3.0.0. I have installed from the zip file created 13 days ago.
But I'm still getting the error: HTTP Error 500.19 - Internal Server Error
Where it lists the following;
Module	   IIS Web Core
Notification	   Unknown
Handler	   Not yet determined
Error Code	   0x8007000d
Config Error
Config File	   \?\C:\PassCore\web.config
Config Source:
-1:
0:
I'm uncertain on how to proceed to resolve this issue.
Thank you.

@rhart211 How did you fix this?


Magic wand icon

Describe the requested icon
I think a magic wand icon would be really useful. For places like adding filters in photos or transforming something magical.
Screenshots
If applicable, add screenshots to help illustrate the icon appearance.

Additional context
Add any other context here.

magic-wand  added in v1.3.1!


added the 'phone' class and added the the colour

Colour code : #00316A

Can you put the css class as phone -> phone-number

If you do like that we can easily find what is that. If it is phone contributors will think "it is the phone icon"


Show full progress bar if --tty option is passed

I use observr to run my specs when files change. It runs rspec --tty --format Fuubar but ruby-progressbar still thinks there is no TTY, so it shows the dumbed-down progress bar.
This change makes it so passing the --tty option to rspec will force ruby-progressbar to think there is a TTY.

It could be argued that this patch (adding tty? => true on output) belongs in rspec-core, but I think the utility of it is further removed from that project and has a lesser chance of actually getting merged.

I'm unsubscribing from this PR to get it off my list. If you have any questions, please @-mention me.

@seven1m I pinged the rspec-core team about this.  In the meantime, I'm going to add your patch because I think the functionality you're describing is a reasonable assumption.

@seven1m I committed a fix for this.  It's different from your solution because I didn't want to modify the object that RSpec gives us.  I'm going to release a beta version of fuubar in a second.  Give it a try and let me know if it works for you.
Thanks for reporting this!

Great, thanks!!!

@seven1m for the record I'm not 100% sure I'm leaving this in here.  It's kind of hacky, but we can try it with the beta and see how it goes. :)

I might have done something wrong, but I'm getting cannot load such file -- fuubar/output (LoadError) with 2.1.0.beta2 😢
/usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require': cannot load such file -- fuubar/output (LoadError)
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/fuubar-2.1.0.beta2/lib/fuubar.rb:4:in `<top (required)>'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
	from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration.rb:1295:in `block in requires='
	from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration.rb:1295:in `each'
	from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration.rb:1295:in `requires='
	from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration_options.rb:109:in `block in process_options_into'
	from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration_options.rb:108:in `each'
	from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration_options.rb:108:in `process_options_into'
	from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration_options.rb:21:in `configure'
	from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:105:in `setup'
	from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:92:in `run'
	from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:78:in `run'
	from /usr/local/Cellar/rbenv/0.4.0/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:45:in `invoke'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.4/exe/rspec:4:in `<top (required)>'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `load'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `block in load'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:268:in `load'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/spring-commands-rspec-1.0.4/lib/spring/commands/rspec.rb:18:in `call'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_require.rb:54:in `require'
	from -e:1:in `<main>'

Not sure if it matters, but here is the command I'm running:
bundle exec spring rspec --color --tty  --require /usr/local/opt/rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/fuubar-2.1.0.beta2/lib/fuubar.rb --format Fuubar my_spec.rb

The reason I'm requiring the library manually is because the gem is not in my bundle (and I don't want it to be).

Nevermind! I'm dumb. Works great!


Add Dockerfile, compose.yaml, and an unbound example config

This adds a Dockerfile for unbound exporter, which we can publish in the future.  An example docker compose.yml is included to demonstrate and test using it with unbound, along with a sample configuration file for unbound showing how to set up the remote-control.
The unbound example config file is based on the one inside the mvance/docker image that's used here.
A small integration test runs against the docker-compose setup to smoke-test unbound_exporter.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (cbed007) 5.23% compared to head (bf12da6) 5.23%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@          Coverage Diff          @@
##            main     #66   +/-   ##
=====================================
  Coverage   5.23%   5.23%
=====================================
  Files          1       1
  Lines        191     191
=====================================
  Hits          10      10
  Misses       181     181

Flag
Coverage Δ

unbound_exporter_tests
5.23% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Looks good so far 👍🏻

I was thinking about adding publishing to this PR but I don't think there's any reason to block on that, so I've marked this ready to review.


Manual review for DRINK extension update

Update our extension on tokens-list.json

pr #6065

can you pls fix the formatting (number of spaces)?

done

done man
Pada tanggal Jum, 3 Des 2021 pukul 22.04 keone @.***>
menulis:

can you pls fix the formatting (number of spaces)?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/solana-labs/token-list/issues/6091#issuecomment-985592153,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AWQPMK47OTJHMH62SV4Y2YTUPDL7JANCNFSM5JJ2WPJA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

merged


1.15.0 Doesnot update online status

title says enough. but as i can see, earlier versions had some alert like "poe.trade settings saved", but for now clicking Save poe.trade settings seems does nothing

i am getting this aswell

Please update to Procurement 1.20.0, and create a new ticket if this is still an issue.


Is there an easy way to add a root node to the json file?

Hi,
Currently, the converter is outputting json as:
[
  {
    "name": "John Doe",
    "age": "30"
  },
  {
   "name": "Alex Len",
    "age": "27"
  },
  {
   "name": "Debbie John",
    "age": "36"
  }

]

But, is there a way to add a root node through configuration variables, so that the json looks like this:
[
  {
    "user": [
      {
        "name": "John Doe",
        "age": "30"
      },
      {
        "name": "Alex Len",
        "age": "27"
      },
      {
        "name": "Debbie John",
        "age": "36"
      }
    ]
  }
]

I would love to be able to do that too.

Isn't this easy with a post-processing step?  Seems better to leave it up to the client rather than try to solve it with configuration.

Yeah it can be a post-proccessing step. That's what I did when I wanted it this way. But having it is an option in the config would have saved all that PP. Not a deal breaker but I was curious if there was something that I was missing.

There is always a tension between simplicity and feature count.
I prefer small, simple packages that you chain together as opposed to each
package being able to execute each feature requested. Small and simple
usually means stable, solid, easy to fix. Feature rich packages are always
more buggy, difficult to fix, extend, etc. For each feature added, there is
a non-linear number of test cases added.
IOW: I completely agree with Max's response.
On Tue, Apr 25, 2017 at 2:45 PM, Anurag Sharma<EMAIL_ADDRESS>wrote:

Yeah it can be a post-proccessing step. That's what I did when I wanted it
this way. But having it is an option in the config would have saved all
that PP. Not a deal breaker but I was curious if there was something that I
was missing.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/stevetarver/excel-as-json/issues/15#issuecomment-297159435,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADvmTk8J9Bvadf05mH_ttvxae-9nwXd-ks5rzltfgaJpZM4MqlAU
.


Fixed bug, state set to None in Annealer.init. Reworked tests to catch bug.

Fixed bug in Annealer.init, where state was initialized to None when load_state was passed to initializer. Reworked test suite to catch the bug.

Thanks for catching this bug @Gunnstein - this PR works great, I appreciate the tests!


Speeding up convex decomposition

Hi,
One quick question: I've tried test_path_decomp_3d.cpp and it takes ~1305 ms to do the convex decomposition (with the example environment provided).  test_path_decomp_2d.cpp  takes 944 ms.
Is there a way to speed this up? I'd like to achieve runtimes < 10 ms
Thanks a lot!

Did you compile it with Release flag? it should not take that long....

No problem:) If you are looking for extremely fast implementation, I think downsample the point cloud will definitely speed up the process, and also the 2D implementation is much faster than 3D.


Refactor menu's css seems missing

Does this issue occur when all extensions are disabled?: Yes, but with vscode-remote enabled.

Version: 1.85.0-insider (user setup)
Commit: 05cd2640ec8a106a4ee99cb38e6ee34fbec04f11
Date: 2023-11-21T05:35:09.651Z
Electron: 25.9.4
ElectronBuildId: 25127168
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.19045
Remote-SSH: v0.107.0

Steps to Reproduce:

Create a typescript file.
Select a function and click on refactor.

Do you still get this after you update to the newest Insiders?

Yeah, the bug is still there.
The latest insider:

Version: 1.85.0-insider (user setup)
Commit:<PHONE_NUMBER>ef1159f5e973539d68c69ed1b519f2
Date: 2023-11-22T14:33:51.941Z
Electron: 25.9.4
ElectronBuildId: 25127168
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.19045

I've been unable to reproduce this - do you have a reliable repro for this @fourdim ?

At least it is reproducible on my side.

Disable all the extensions.
Create a file called a.ts
Copy the following code to the file:

export const aa = () => {
    console.log('aa');
};

Move the cursor to lo|g

export const aa = () => {
    console.lo|g('aa');
};

Right click to open the menu and then click on the refactor.

repro:

Linux
refactor (via menu) before hitting the lightbulb, using the quick fix menu, or using ctrl + shift + r
Observe focused, but incorrectly rendered menu

The same issue is in my environment when using "Refactor" on a selected text in a TypeScript file. I'm able to reproduce it reliably.
Note it happens only if the setting window.titleBarStyle has the value custom. Value native works well.
It could be a regression in 1.85 as I didn't the issue in 1.84.

Tested with (both environments are affected):

Windows 11, VSCode 1.85
Ubuntu 22.04, VSCode 1.85

This is happening in stable now, the refactor menu is broken in many ways. For TypeScript it looks like this:

For Dart/Flutter, the menu just doesn't appear when you click "Refactor" in the context menu, but if you use the shortcut key it appears, and then it subsequently appears when you use the menu.

exploring this right now - looks like it's only happening in Windows and Linux, was unable to repro on Mac.

wonder if it's related to @bpasero 's change in the contextViewService https://github.com/microsoft/vscode/commit/af63c41918a0608c2385d79efc8b1b5b3a033339
I'm seeing shadow-root where it's not supposed to be.


Is adding an "ongoing" property to catch up on the daily count a good idea?

Hi, I'm very excited about this tool. It is convenient and helpful. Thank you so much.
I'm trying to draw a GitHub contribution heatmap in my blog site and I want to keep this renewed daily. And I found this API doesn't fetch the newest data. I viewed the JSON file structure and recognized that it is organized on a weekly basis as the data structure offered by GitHub API(https://github.com/users/{username}/contributions). So it doesn't contain the contribution data of the ongoing week that exists though.
The JSON file has two properties(total and contributions) as same as the GitHub API. Is adding an "ongoing" property to catch up on the daily count a good idea? This property will not destroy the current data structure and provide an alter for those who want to renew their contribution count daily.

Glad you've found it useful.

And I found this API doesn't fetch the newest data.

Yes, values are cached for 24 hours. If you want to blow through the cache you'll need to do that in your browser.

So it doesn't contain the contribution data of the ongoing week that exists though.

Looks like a bug, it's not intentional.

Thanks for your reply.

Looks like a bug, it's not intentional.

Haha, I thought before that it was to make each week's data equal in length to prevent unexpected behavior.
The structure of raw data is organized by the name of the day of the week rather not the time. So the data of the ongoing week will be cut off here I guess.
Object.values(data)
        .sort((a, b) => (a.date < b.date ? -1 : 1))
        .slice(0, limit * 7)
        .forEach((day, idx) => {
            weekIndex = Math.floor(idx / 7);
            if (!contributions[weekIndex]) contributions.push([]);
            contributions[weekIndex].push(day);
        });

I'm new to programming. Thanks for your help.

Indeed, limit defaults to -1 and so that slice call turns into .slice(0, -7), hence the cut off. Oops
PRs welcome, if you wanted to fix it. Else I can take a look at some point.

That's great. Maybe I can have a try.
And just appending an array that is shorter then seven in the contribution property is recommended?

Fixed in #7


Print the log messages of failed tests for shaded-test

Motivation:
We currently do not print the log messages of failed tests, which is
sometimes a problem when a test failed in shaded-test but not in a
non-shaded one.
Modifications:

Print the log message of failed tests for shaded-test as well

Result:
More build visibility

Thanks, @minwoox !


Missing code during insertlocal for BGI/Ethornell

I am having an issue with encoding during insertlocal. I do not have this issue when I try to encode the files with BGIKit's ScriptEncoder, but I'm still trying to figure out what I'm doing wrong here. I first used extractlocal to spit out the script.xlsx file, then ran it again after editing the names in names.xml, then I ran insertlocal with that script.xlsx using the syntax VNTextPatch insertlocal \orig script.xlsx \patch where C:\Orig contains one of the original script files, ev0010.bgi. The command works in that it spits out ev0010.bgi into C:\Patch, but when I open it in EthernelEditor, there are some commands missing.
Below shows the original file (left) and the encoded file from insertlocal (right) in EthornellEditor. Scene and BGM commands disappear during the encoding process, and the IS_ECG102_s01 command gets duplicated at the beginning, replacing _FadeBGM. The game files are from Ima Sugu Onii-chan ni Imouto da tte Iitai!.

The above code change should fix it.

It did fix it, thank you.


testsuite: ztress: Remove idle thread NULL check

This commit removes the NULL check on the idle thread array
(idle_tid) because it is statically declared and will never be NULL.
This fixes the "comparison will always evaluate as ‘false’" warning
generated by the GCC 12.
Signed-off-by: Stephanos Ioannidis<EMAIL_ADDRESS>

added a backport label for v3.1-branch, as this also will report same build warning on v3.1-branch with zephyr sdk 0.15.0.


How to use pixel training instead of low-dimensional state?

You need to use a wrapper.
For example see:
https://github.com/emansim/acktr/blob/master/rgb_env.py
or
https://github.com/deepmind/dm_control/blob/a8112e730ed109c7b21a296f5cb1402bfb0bbcee/dm_control/suite/wrappers/pixels.py#L32
for dm_control suite.

Hi Ilya,
Thank you very much for answering my question.
That is what I tried (using rgb_env.py from emansim's github). However, I got this following error:
#######
WARNING: All rewards are clipped or normalized so you need to use a monitor (see envs.py) or visdom plot to get true rewards
#######
[2018-04-26 18:10:26,435] Making new env: Reacher-v1
Traceback (most recent call last):
File "main.py", line 205, in
main()
File "main.py", line 111, in main
obs = envs.reset()
File "/home/knn1989/PycharmProjects/Mar_9_2018/baselines-master/baselines/common/vec_env/dummy_vec_env.py", line 21, in reset
results = [env.reset() for env in self.envs]
File "/home/knn1989/PycharmProjects/Mar_9_2018/baselines-master/baselines/common/vec_env/dummy_vec_env.py", line 21, in
results = [env.reset() for env in self.envs]
File "/home/knn1989/PycharmProjects/Mar_9_2018/gym/gym/core.py", line 104, in reset
return self._reset()
File "/home/knn1989/PycharmProjects/Mar_9_2018/gym/gym/core.py", line 312, in _reset
return self._observation(observation)
File "/home/knn1989/PycharmProjects/Mar_9_2018/gym/gym/core.py", line 322, in _observation
raise NotImplementedError
NotImplementedError

It's probably because they changed something in gym. See how the interface for envs is implemented now.

Great! In this case I'm closing the issue.


⚠ 366.ru has degraded performance

In 6a50df7, 366.ru (https://366.ru) experienced degraded performance:

HTTP code: 200
Response time: 1348 ms

Resolved: 366.ru performance has improved in d20e7fa.


Api Key is invalid

Hey,
I'm trying to use the Adobe Lightroom API,  but I'm getting a response to a POST request.

{
"error_code": "403003",
"message": "Api Key is invalid"
} '

Token generated from client_id and secret_key, but in "Scopes" field "openid" by default, any other value leads to Authorization error
In a POST request, I use Access Token and Client Id
Where am I wrong?

Hey did you ever solve this?  I have the same issue

As do I, It seems like these APIs may be abandoned

@hirenoble Is this API still supported?


Fix MediaUtil for devices with density of 1.0

Turns out that calling scaling with a density of 1.0 causes Android to
just return the original bitmap. But we called recycle() on the
original bitmap, which results in an error when the scaled
bitmap (which is “eq” to the original bitmap) is used.
Remove the call to recycle() as it likely isn’t needed. Don’t bother
attempting to scale a bitmap when the density is 1.0
Change-Id: Ib9b52baa29c99b2ccd77fbef299522446552ab60

@halatmit Please review. I'll likely update the commit message and the comments in the code to be more explanatory. But this is the gist of the actual change.

LGTM


Update media-services-concepts.md

@Juliako,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Optimize to_checksum_address function

While developing Safe Tx Service we found out that a lot of time was spent on converting addresses from bytes to ChecksumAddress. Biggest part of this time was overhead added on eth_utils library to keccak256 function.
So we decided to use pysha3 library (way faster than pycryptodome) directly. That's why we implemented new methods:

fast_keccak
fast_keccak_hex
fast_to_checksum_address
fast_bytes_to_checksum_address
fast_is_checksum_address

This is also very relevant for the EthereumAddressV2Field, as it stores addresses as bytes on database and returns them as ChecksumAddress
Related to:

https://github.com/ethereum/eth-utils/issues/95
https://github.com/ethereum/eth-hash/issues/35

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

80 of 82 (97.56%)  changed or added relevant lines in 15 files are covered.
22 unchanged lines in 6 files lost coverage.
Overall coverage decreased (-0.6%) to 90.359%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

gnosis/eth/utils.py
28
30
93.33%

Files with Coverage Reduction
New Missed Lines
%

gnosis/eth/oracles/oracles.py
2
89.27%

gnosis/eth/typing.py
2
83.33%

gnosis/protocol/gnosis_protocol_api.py
2
89.25%

gnosis/safe/safe.py
2
87.85%

gnosis/eth/ethereum_client.py
3
85.94%

gnosis/eth/clients/sourcify.py
11
62.5%

Totals

Change from base Build<PHONE_NUMBER>:
-0.6%

Covered Lines:
2943

Relevant Lines:
3257

💛  - Coveralls

Related to https://github.com/safe-global/safe-pm/issues/89


There is an extra '…' next to the Download action on the toolbar for one ADLS Gen2 blob container

Storage Explorer Version: 1.27.0-dev
Build Number: 20221025.2
Branch: main
Platform/OS: Windows 10/Linux Ubuntu 22.04/MacOS Monterey 12.6 (Apple M1 Pro)
Architecture: ia32/x64
How Found: From running test cases
Regression From: Previous release (1.26.1)
Steps to Reproduce

Expand one ADLS Gen2 storage account -> Blob Containers.
Open one blob container -> Observe the toolbar.
Check there is no '...' next to the 'Download' action.

Expected Experience
There is no '...' next to the 'Download' action.
Actual Experience
There is a '...' next to the 'Download' action.

Verified this issue on the build 20221027.6. Fixed. So, we are going to close it.


Update services and corresponding tests

Task: AB#11053

There's still a bit of updated test coverage required here. I've done some manual tests but planning to work on filling in the unit test gaps today, all being well.


Judge Policy Plugin design decision: by file name or stream?

#49 mentions that the policy plugin should support multiple outputs comparisons
This issue discusses which policy plugin design should be better in generic cases?
Suppose there are two files to be compared, actual.txt and expect.txt.
Which of the followings is the best regarding generic cases:

Pass both file names to the plugin
Pass input streams to the plugin

As the design goes,
it's refactored to use file names, with a signature like
JudgeResult judge(TestCase testCase, JudgeProfile judgeProfile, String actualStandardOutputFileName,
                             String expectStandardOutputFileName, Map<String, String> actualToExpectOutputFileNameMap)


"retweet" feature (to be renamed)

@fixedelineation suggests we could have an MVP where it simply creates a blog with the link to the Discussions post.
This is a good start, and we can tag the 'retweeted' user to the post as well.

@fixedelineation suggests we could have an MVP where it simply creates a blog with the link to the Discussions post.
This is a good start, and we can tag the 'retweeted' user to the post as well.

@fixedelineation suggested this be renamed to "Rehash" so I've updated the  title

@fixedelineation suggested this be renamed to "Rehash" so I've updated the  title


Not working anymore

Hi Rogier,
My scale is not reporting in HA anymore.  Any idea what is going on?
Are/should there be any logfiles?
Core 2024.9.3
Supervisor 2024.9.1
o.s. 13.1
Okokscale 0.03
Many thanks for your time!
Frank

here neither
Core 2024.10.1
Supervisor 2024.09.1
Operating System 13.1
Okokscale 0.03

Sorry, but for me it's working with the lates HA. Maybe you could enable debug logging and see what's being logged

With the latest software its working again indeed.
Many thanks for your support!


values.yaml from Helm dependencies are not configurable

Expected behavior
The contents of values.yaml files from helm dependencies (specified in requirements.yaml) should be configurable in Step 2.
Actual behavior
Only the values.yaml from the chart's root directory is configurable in Step 2.
Information

Ship version: 0.44.0
Docker version: n/a
Command line run: n/a

Example
Istio Helm chart depends on various charts which can be found in the charts/ directory. Each chart in the charts/ has its own values.yaml file which is not configurable via ship init or ship edit.

As workaround I do:
kiali:
enabled: true
hub: foo.bar
tag: 123
It would be nice though if dependent chart values are merged with parent one


Fix packaging of maven artifact

If you take a look at the .pom file in maven central, you'll notice that the packaging is 'aar.asc'.
http://search.maven.org/#artifactdetails|com.facebook.conceal|conceal|1.0.0|aar.asc
But it should be only 'aar'. I'm not sure what's causing this issue in your build.gradle file. You are using old build tools, that might be the reason.
To fix the dependency in gradle I used: compile 'com.facebook.conceal:conceal:1.0.0@aar'
It works, but isn't that pretty. Otherwise the maven artifact works great!

Please fix

Thanks for the report @JosephEarl . Didn't know that this was happening. I will fix this.

You can see more info on the accepted answer here http://stackoverflow.com/questions/22795455/transitive-dependencies-not-resolved-for-aar-library-using-gradle

Thank you for reporting this issue and appreciate your patience. We've  notified the core team for an update on this issue. We're looking for a response within the next 30 days or the issue may be closed.

Has the fix been submitted for this issue? The latest Maven central pom.xml still shows packaging as "aar.asc".
[pom.xml] (https://repo1.maven.org/maven2/com/facebook/conceal/conceal/1.0.1/conceal-1.0.1.pom) from Maven Central for v1.0.1
<modelVersion>4.0.0</modelVersion>
<groupId>com.facebook.conceal</groupId>
<artifactId>conceal</artifactId>
<version>1.0.1</version>
<packaging>aar.asc</packaging>
<name>Conceal</name>

Default configuration in the build.gradle is causing packaging to be used as "aar.asc" instead of "aar".
configurations {
    archives {
        extendsFrom configurations.default
    }
}

Thanks

This keeps being an issue... and a really nasty one.
Can this be fixed?
Thanks

Bump, this is still an issue in 1.1.3. Can this please be fixed?

Bump again.
This is still an issue in 2.0.1.    It is definitely causing a bit of grief with dependency handling if Conceal is being used in another AAR project.
Apparently, simply removing the default configuration (as mentioned in Anand's comment, above) may resolve it.

Hi. We just uploaded version 2.0.2 with the fix to produce a AAR packaging type.
I could not yet try it on the same configuration app -> LibraryAAR -> ConcealAAR but you could give it a try.


Fix the function to extract last 32bits from number

In previous version, the way to obtain the last 32 bits of the event_table_pos is in fact wrong, which is fix in this PR.
As for the file mentioned in the Issue #6550, named 914flankers.cnt, is in fact a special file which has no overflow problem, but the calculated event table pos and the readed version from SETUP section just unmatch. That means reading this file will still throw an exception. One possible reason is there is the number of samples is not correct, and I will give another PR to fix this problem (one simple solution is not to throw an exception, but return the readed value)
Before that, I have test all the cnt file in my hand, if possible, could try the new version on your dataset agian?

As for the file mentioned in the Issue #6550, named 914flankers.cnt, is in fact a special file which has no overflow problem, but the calculated event table pos and the readed version from SETUP section just unmatch. That means reading this file will still throw an exception. One possible reason is there is the number of samples is not correct, and I will give another PR to fix this problem (one simple solution is not to throw an exception, but return the readed value)

That was the problem all along, that the solution was breaking the current test suite  'cos the computation was not identifying those cases. i.e: https://github.com/mne-tools/mne-python/pull/6550/checks?check_run_id=165686581

Then we need to find a way around it. But we need to make both scenarios work.
Any ideas on what to do if we can't trust n_samples?

I guess the question is:
If n_samples is right do we always find a match between 	even_table_pos and the computation?
If so, it means that overflow or not we would match. If we don't match, we have to assume that either even_table_pos or n_samples are wrong. In which chase we can assume that the wrong one is n_samples.

WDYT?

I guess the question is:
If n_samples is right do we always find a match between even_table_pos and the computation?

Definetly, I think.  At first, there is little possible that the calc_event_table_pos could be wrong if n_samples is correct, n_channels has little possible to be wrong and overflow. And we also don't need to consider the situation n_samples overflows ( this takes a long experiment time, approximatly hundreds of hours).

If so, it means that overflow or not we would match. If we don't match, we have to assume that either even_table_pos or n_samples are wrong. In which chase we can assume that the wrong one is n_samples.
Yes, In my code, the n_samples will be considered as wrong instead of the event_table_pos. And the n_samples will returned directly without correction. Because I am not sure whether the data in the gap between n_samples and TEEG has any meaning, and should considered as sample point.
Meanless data?
|                               |xxxxxxxxxxxxxxxxxx|                     |
n_samples              Event_table_pos

did we finish all this? now I'm confused. can you refresh me @AimerLee


Removed console.log

I think console.log should not be used like this without checking because it will throw error on IE. I have commented it out.

I'll closed this since there are already at least two others pull request on this!


pipenv install times out while creating virtualenv

Running pipenv --three install fails at Creating a virtualenv for this project with the output:
Creating a virtualenv for this project...
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/pexpect/expect.py", line 97, in expect_loop
    return self.timeout()
  File "/usr/lib/python3.4/site-packages/pexpect/expect.py", line 70, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x7fe6f4e65f28>
searcher: searcher_re:
    0: EOF

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/pipenv", line 9, in <module>
    load_entry_point('pipenv==0.2.8', 'console_scripts', 'pipenv')()
  File "/usr/lib/python3.4/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.4/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.4/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.4/site-packages/pipenv/cli.py", line 393, in install
    ensure_project(dev=dev, three=three)
  File "/usr/lib/python3.4/site-packages/pipenv/cli.py", line 76, in ensure_project
    ensure_virtualenv(three=three)
  File "/usr/lib/python3.4/site-packages/pipenv/cli.py", line 57, in ensure_virtualenv
    do_create_virtualenv(three=three)
  File "/usr/lib/python3.4/site-packages/pipenv/cli.py", line 166, in do_create_virtualenv
    click.echo(crayons.blue(c.out))
  File "/usr/lib/python3.4/site-packages/delegator.py", line 78, in out
    self.__out = self._pexpect_out
  File "/usr/lib/python3.4/site-packages/delegator.py", line 66, in _pexpect_out
    result += self.subprocess.read().decode('utf-8')
  File "/usr/lib/python3.4/site-packages/pexpect/spawnbase.py", line 413, in read
    self.expect(self.delimiter)
  File "/usr/lib/python3.4/site-packages/pexpect/spawnbase.py", line 321, in expect
    timeout, searchwindowsize, async)
  File "/usr/lib/python3.4/site-packages/pexpect/spawnbase.py", line 345, in expect_list
    return exp.expect_loop(timeout)
  File "/usr/lib/python3.4/site-packages/pexpect/expect.py", line 107, in expect_loop
    return self.timeout(e)
  File "/usr/lib/python3.4/site-packages/pexpect/expect.py", line 70, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x7fe6f4e65f28>
searcher: searcher_re:
    0: EOF
<pexpect.popen_spawn.PopenSpawn object at 0x7fe6f4e65f28>
searcher: searcher_re:
    0: EOF

The .venv directory is created and subsequent use of pipenv install [package] work correctly.
Relevant package versions:
delegator.py==0.0.7
pexpect==4.2.1
pipenv==0.2.8
virtualenv==15.1.0

interesting, cannot reproduce locally

@jstutters did you by any chance have any networking problems at the moment that could potentially lead to packages needed for the initial virtualenv to take too long to download.
How to reproduce:
Modify virtualenv to add a more than 30 second sleep without outputing anything.
pipenv --three install

@kennethreitz Are you using the default timeout of 30 seconds for pexpect in delegator? Does that make sense in this case? Shouldn't it probably have a very larger timeout to accommodate slow connections or even not have a timeout at all and let the user stop it if he thinks its stuck somewhere?

I am, but there's no reason creating a virtualenv should take more than 30s. I suspect a setup issue is at hand.

@jstutters can you time the creation of a new virtualenv by hand?

@kennethreitz It's on a networked home directory which is certainly a potential source of the slowness (sorry- should have said this in the original report).  Execution time for making the virtualenv manually is 49 seconds so I guess a timeout increase would help.  I had a look at doing this myself but it didn't seem from the docs that delegator exposes the pexpect timeout argument.
jstutters@host ~/test % /usr/bin/time -f %E virtualenv -p /usr/bin/python3 foo
Running virtualenv with interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/jstutters/test/foo/bin/python3
Also creating executable in /home/jstutters/test/foo/bin/python
Please make sure you remove any previous custom paths from your /home/jstutters/.pydistutils.cfg file.
Installing setuptools, pip, wheel...done.
0:48.59

Okay, so right off the bat I do think we should leave the timeout by default at 30 seconds, because as @kennethreitz  mentioned there's not many cases in which this is not enough time. However, there are use cases when it may be useful (or even necessary).

In the do_create_virtualenv function, we make the call delegator.run(cmd, block=False).
If we go take a look at delegator/delegator.py, it makes the call PopenSpawn(self._popen_args, **self._default_pexpect_kwargs). Viewing the documentation for PopenSpawn, we see the constructor takes an optional timeout parameter
def __init__(self, cmd, timeout=30, maxread=2000, searchwindowsize=None, logfile=None,
             cwd=None, codec_errors='strict')

So if we're wanting to set a custom timeout for PopenSpawn, then we would need to modify the delegator constructor or delegator.run to take an optional timeout=30 parameter, or something to that effect. Then we would be able to pass in the value through Pipenv in whatever fashion is deemed suitable.

I think 30s is reasonable too.

It would be awesome if there was a fix for this issue. For me creating a virtualenv takes 40 seconds when I have my antivirus running. Disabling it drops the time to 2 seconds and then pipenv works fine, but that's not really an acceptable solution as disabling antivirus company-wide is not an option.

@radati, can you confirm for me if creating a new environment with the virtualenv command or pew new command takes the same amount of time as pipenv --three?

Yes, they all take about the same time: pipenv --three 50 s, virtualenv 40 s, pew new 40 s. For pipenv I increased the timeout manually to be able to measure it. All of them are a couple seconds if antivirus is turned off.

So I'm a bit confused what it is about the virtualenv creation that's being delayed so drastically by your AV. This feels like more of an issue with virtualenv, but we could implement a "fix" in Delegator.py. That's a separate project of @kennethreitz, so I'm going to tag him in to see if he has thoughts here.
Bumping the timeout makes for a worse experience for people with a legitimate failure, but will increase tolerance for cases like this. Given that we've only had two users encounter this though, I'm not sure how common it really is.

I am having this problem as well, also with a slow network home directory. A way of overriding the default timeout or specifying where the venvs are created would be very useful. It takes around 30s to create the venv normally so it only fails sometimes.
A hugely useful script otherwise. Thanks.

I ran into this problem too. pipenv times out if I have an intensive process running in the backround, but works fine if I kill the background process. Would prefer an option to override the default timeout.

This happens for me on AWS t2.x instances types, and likely will continue to happen inside containers as well where limitations are exaggerated. While 30s is reasonable, not allowing configuration through something as easy as an env variable is pretty unflexible

➜  pipenv --two
Creating a virtualenv for this project...
Traceback (most recent call last):
File "/usr/local/bin/pipenv", line 11, in
sys.exit(cli())
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1043, in invoke
return Command.invoke(self, ctx)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 756, in cli
ensure_project(three=three, python=python)
File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 135, in ensure_project
ensure_virtualenv(three=three, python=python)
File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 115, in ensure_virtualenv
do_create_virtualenv(three=three, python=python)
File "/usr/local/lib/python2.7/site-packages/pipenv/cli.py", line 377, in do_create_virtualenv
click.echo(crayons.blue(c.out), err=True)
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/delegator.py", line 78, in out
self.__out = self._pexpect_out
File "/usr/local/lib/python2.7/site-packages/pipenv/vendor/delegator.py", line 66, in _pexpect_out
result += self.subprocess.read().decode('utf-8')
File "/usr/local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 413, in read
self.expect(self.delimiter)
File "/usr/local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 321, in expect
timeout, searchwindowsize, async)
File "/usr/local/lib/python2.7/site-packages/pexpect/spawnbase.py", line 345, in expect_list
return exp.expect_loop(timeout)
File "/usr/local/lib/python2.7/site-packages/pexpect/expect.py", line 107, in expect_loop
return self.timeout(e)
File "/usr/local/lib/python2.7/site-packages/pexpect/expect.py", line 70, in timeout
raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x1052c2a10>
searcher: searcher_re:
0: EOF
<pexpect.popen_spawn.PopenSpawn object at 0x1052c2a10>
searcher: searcher_re:
0: EOF

Don't know why it takes so long normally:

➜  time virtualenv whatever
New python executable in /.../whatever/bin/python2.7
Also creating executable in /.../whatever/bin/python
Installing setuptools, pip, wheel...done.
virtualenv whatever  2.58s user 0.71s system 7% cpu 42.808 total

@kennethreitz and I had previously discussed adding a timeout option, or bumping the timeout on delegator.py. If he still feels that's a good way to go, we can look into a patch for this.

I'm experiencing the same issue as well but I solved it. I tried switching wifi networks because the problem is somehow related to the firewall setup. Didn't bother to ask IT as it will take time.

It was a networking problem for me. VPN was blocking it somehow. Better error message would be handy

Contributions on better error messages would be welcomed!

It would be good to make a reference to PIPENV_TIMEOUT environment variable in that exception.

I was on my phone's internet, and when I got back on the good wifi, this error went away.

This solution solved my problem: https://github.com/pypa/pipenv/issues/2681#issuecomment-504469464

Got similar problem with timeout. It waited for around 20 minutes, then failed:
positron@ubox:~/Documents$ pipenv install mesa
Creating a virtualenv for this project…
Pipfile: /home/positron/Documents/Pipfile
Using /usr/bin/python3 (3.5.2) to create virtualenv…
⠏ Creating virtual environment...Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/positron/.local/share/virtualenvs/Documents-rjeAWyrt/bin/python3
Also creating executable in /home/positron/.local/share/virtualenvs/Documents-rjeAWyrt/bin/python
Installing setuptools, pip, wheel...
done.

✔ Successfully created virtual environment!
Virtualenv location: /home/positron/.local/share/virtualenvs/Documents-rjeAWyrt
Creating a Pipfile for this project…
Installing mesa…
Adding mesa to Pipfile's [packages]…
✔ Installation Succeeded
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
⠇ Locking...
⠙ Locking...
⠸ Locking...
Traceback (most recent call last):
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/expect.py", line 109, in expect_loop
    return self.timeout()
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/expect.py", line 82, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x7f870d98f1d0>
searcher: searcher_re:
    0: re.compile('\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/positron/.local/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/cli/command.py", line 254, in install
    editable_packages=state.installstate.editables,
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/core.py", line 1992, in do_install
    skip_lock=skip_lock,
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/core.py", line 1244, in do_init
    pypi_mirror=pypi_mirror,
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/core.py", line 1068, in do_lock
    lockfile=lockfile
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/utils.py", line 649, in venv_resolve_deps
    c = resolve(cmd, sp)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/utils.py", line 517, in resolve
    result = c.expect(u"\n", timeout=environments.PIPENV_INSTALL_TIMEOUT)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/delegator.py", line 215, in expect
    self.subprocess.expect(pattern=pattern, timeout=timeout)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/spawnbase.py", line 341, in expect
    timeout, searchwindowsize, async_)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/spawnbase.py", line 369, in expect_list
    return exp.expect_loop(timeout)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/expect.py", line 119, in expect_loop
    return self.timeout(e)
  File "/home/positron/.local/lib/python3.5/site-packages/pipenv/vendor/pexpect/expect.py", line 82, in timeout
    raise TIMEOUT(msg)
pexpect.exceptions.TIMEOUT: <pexpect.popen_spawn.PopenSpawn object at 0x7f870d98f1d0>
searcher: searcher_re:
    0: re.compile('\n')
<pexpect.popen_spawn.PopenSpawn object at 0x7f870d98f1d0>
searcher: searcher_re:
    0: re.compile('\n')

Pipenv is 2018.11.26
OS is xubuntu 16.04.3 x64

@positron96 I had the same problem with u. Could u tell me how to solve the problem?


[api] Make sure collection list in entrypoint is returned sorted.

With the ever growing api.yml file, it's getting harder to keep
manually sorted. This PR makes sure the collection list being
returned is sorted.

@gtanzillo @imtayadeway minor enhancement.

👍 LGTM


svix-bridge: Invalid Svix secret: InvalidSecret(InvalidByte(5, 61))

Bug Report
cargo run --no-default-features --features gcp-pubsub,rabbitmq,redis,sqs -- --cfg-file svix-bridge.example.receivers.yaml
    Finished dev [unoptimized + debuginfo] target(s) in 0.81s
     Running `target\debug\svix-bridge.exe --cfg-file svix-bridge.example.receivers.yaml`
2023-11-21T17:18:20.519305Z  INFO svix_bridge: starting
2023-11-21T17:18:20.519894Z  WARN svix_bridge: No senders configured.
2023-11-21T17:18:20.519926Z  INFO svix_bridge: Starting JS Transformation Workers: 4
thread '2023-11-21T17:18:20.520034Z ERROR common_metrics_collector: svix_bridge: Unable to get allocator stats mibs: metric collection is not supported
main' panicked at svix-bridge\src\webhook_receiver\types.rs:65:67:
Invalid Svix secret: InvalidSecret(InvalidByte(5, 61))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\debug\svix-bridge.exe --cfg-file svix-bridge.example.receivers.yaml` (exit code: 101)

Version
cargo tree | rg svix
svix-bridge v1.14.0 (C:\w\rust\svix-webhooks\bridge\svix-bridge)
├── svix-bridge-plugin-queue v0.1.0 (C:\w\rust\svix-webhooks\bridge\svix-bridge-plugin-queue)
│   ├── omniqueue v0.1.0 (https://github.com/svix/omniqueue-rs?rev=247904053bcf90cf693df4429092923bf97770eb#24790405)
│   ├── svix-bridge-types v0.1.0 (C:\w\rust\svix-webhooks\bridge\svix-bridge-types)
│   │   ├── svix v0.85.1
├── svix-bridge-types v0.1.0 (C:\w\rust\svix-webhooks\bridge\svix-bridge-types) (*)
├── svix-ksuid v0.7.0
svix-bridge-plugin-queue v0.1.0 (C:\w\rust\svix-webhooks\bridge\svix-bridge-plugin-queue) (*)
svix-bridge-types v0.1.0 (C:\w\rust\svix-webhooks\bridge\svix-bridge-types) (*)

Platform
win11x64
Description
Just running the examples;  the senders.xml looks to be working (continues to run and events stream to console).  receivers panics.

The endpoint secrets are fake in the example file, e.g. whsec_XXXXX=.
You'd need to register an endpoint in svix, pointing to your running bridge, then put the secret for that endpoint in your bridge config.
You'd also need to be running a queue matching the output.
I think more than anything we need to do a pass on the config parser so it can make suggestions when it reports failures rather than just bubbling up a panic.

bubbling up a panic seems like an ok approach but we lose context of where it happened in the config which is unfortunate.
I didn't run the containers and I expected issues.
main' panicked at svix-bridge\src\webhook_receiver\types.rs:65:67:
Invalid Svix secret: InvalidSecret(InvalidByte(5, 61))

I didn't expect a panic on a invalid secret.
Running the bridge with no arguments panics with "config file path".
panics are fine; just need to spend some more time with the code.