[SPARK-29544] [SQL] optimize skewed partition based on data size

What changes were proposed in this pull request?
This PR implement a rule OptimizeSkewedPartitions to optimize the skewed partition based on the partition data size.

Why are the changes needed?
To optimize the skewed partition in runtime based on AQE

Does this PR introduce any user-facing change?
No

How was this patch tested?
UT

@cloud-fan Please help me review. Thanks for your help.

Test build #113434 has started for PR 26434 at commit c64df90.

Test build #113434 has finished for PR 26434 at commit c64df90.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class OptimizeSkewedPartitions(conf: SQLConf) extends Rule[SparkPlan]
case class SkewedShuffleReaderExec(
class SkewedShuffledRowRDD(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113434/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18328/
Test PASSed.

Test build #113435 has started for PR 26434 at commit f0f03c5.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18329/
Test PASSed.

Test build #113435 has finished for PR 26434 at commit f0f03c5.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113435/
Test FAILed.

Test build #113444 has started for PR 26434 at commit 5f56f89.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18338/
Test PASSed.

Test build #113444 has finished for PR 26434 at commit 5f56f89.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113444/
Test FAILed.

Please help to retest. Thanks.

retest this please

Test build #113484 has started for PR 26434 at commit 5f56f89.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18376/
Test PASSed.

Test build #113484 has finished for PR 26434 at commit 5f56f89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113484/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18505/
Test PASSed.

Test build #113622 has started for PR 26434 at commit bbf585d.

Test build #113622 has finished for PR 26434 at commit bbf585d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PostShufflePartitionReader(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/113622/
Test PASSed.

@cloud-fan fix the conflicts and resolve the comments. Please help review again. Thanks for your help.

Test build #114751 has started for PR 26434 at commit 18cdcd9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19573/
Test PASSed.

Test build #114751 has finished for PR 26434 at commit 18cdcd9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114751/
Test PASSed.

Test build #115014 has started for PR 26434 at commit d339287.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19834/
Test PASSed.

@cloud-fan updated the comments online and offline. Please help me review again. Thanks.

Test build #115014 has finished for PR 26434 at commit d339287.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartialShufflePartitionReader(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115014/
Test FAILed.

@cloud-fan @manuzhang Thanks for your reviews. Update the comments. Please help review again. Thanks.

Test build #115085 has started for PR 26434 at commit 3c0296d.

Test build #115085 has finished for PR 26434 at commit 3c0296d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class PartialShuffleReader(
case class SkewedShufflePartitionReader(

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115085/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19900/
Test PASSed.

Test build #115088 has started for PR 26434 at commit 2ba27ca.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19903/
Test PASSed.

Test build #115088 has finished for PR 26434 at commit 2ba27ca.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115088/
Test FAILed.

Test build #115220 has started for PR 26434 at commit a15f471.

Test build #115220 has finished for PR 26434 at commit a15f471.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115220/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20036/
Test PASSed.

Test build #115233 has started for PR 26434 at commit 98a064b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20042/
Test PASSed.

Test build #115233 has finished for PR 26434 at commit 98a064b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115233/
Test FAILed.

The failed test seems not related. Please help retest. Thanks

retest this please

Test build #115375 has started for PR 26434 at commit 98a064b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20181/
Test PASSed.

Test build #115375 has finished for PR 26434 at commit 98a064b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115375/
Test PASSed.

Test build #115444 has started for PR 26434 at commit e3eb3cf.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20247/
Test PASSed.

Test build #115444 has finished for PR 26434 at commit e3eb3cf.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class SkewedShuffledRowRDDPartition(override val index: Int) extends Partition

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115444/
Test FAILed.

Test build #115454 has started for PR 26434 at commit e412593.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20256/
Test PASSed.

Test build #115454 has finished for PR 26434 at commit e412593.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115454/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20790/
Test PASSed.

Test build #115998 has started for PR 26434 at commit f09a710.

Test build #115998 has finished for PR 26434 at commit f09a710.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115998/
Test PASSed.

looks good, can we have a simple benchmark?

Test build #116228 has started for PR 26434 at commit 9aa3e31.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21020/
Test PASSed.

Test build #116228 has finished for PR 26434 at commit 9aa3e31.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116228/
Test FAILed.

Test build #116295 has started for PR 26434 at commit 38ca8cf.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21086/
Test PASSed.

also cc @hvanhovell @juliuszsompolski

Test build #116295 has finished for PR 26434 at commit 38ca8cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116295/
Test PASSed.

Test build #116312 has started for PR 26434 at commit 8e6a6ea.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21104/
Test PASSed.

LGTM except a few minor comments. Can we create an example skew join query and show the perf benefit?

Test build #116312 has finished for PR 26434 at commit 8e6a6ea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116312/
Test PASSed.

Test build #116353 has started for PR 26434 at commit a0e03b6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21144/
Test PASSed.

Test build #116353 has finished for PR 26434 at commit a0e03b6.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116353/
Test FAILed.

retest this please

Test build #116374 has started for PR 26434 at commit a0e03b6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21165/
Test PASSed.

Test build #116374 has finished for PR 26434 at commit a0e03b6.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116374/
Test FAILed.

retest this please

Test build #116389 has started for PR 26434 at commit a0e03b6.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21180/
Test PASSed.

Test build #116389 has finished for PR 26434 at commit a0e03b6.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116389/
Test FAILed.

Test build #116446 has started for PR 26434 at commit a22b804.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21234/
Test PASSed.

Test build #116446 has finished for PR 26434 at commit a22b804.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116446/
Test FAILed.

Test build #116488 has started for PR 26434 at commit cee1c8c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21275/
Test PASSed.

@cloud-fan
We have test this PR with the following sql in 5 node clusters.
spark.range(0, 100000, 1, 6).selectExpr("id % 2 as key1").createOrReplaceTempView("test1");
spark.range(0, 100000, 1, 6).selectExpr("id % 2 +1 as key2").createOrReplaceTempView("test2");
import org.apache.spark.sql.SaveMode.Overwrite
spark.sql("select * from test1, test2 where key1 = key2").write.format("noop").mode(Overwrite).save()

The main spark configuration is:
spark.sql.shuffle.partitions 500
spark.sql.autoBroadcastJoinThreshold -1
spark.sql.adaptive.enabled true
spark.sql.adaptive.shuffle.localShuffleReader.enabled false
spark.sql.adaptive.shuffle.reducePostShufflePartitions.enabled false
spark.sql.adaptive.optimizeSkewedJoin.skewedPartitionSizeThreshold 500

This PR can gain about 6x performance improvement(27s vs 162s). And the following is the UI of with and without this PR.
With this PR:

Without this PR:

Test build #116488 has finished for PR 26434 at commit cee1c8c.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116488/
Test FAILed.

@cloud-fan Can you help to retest and change the test machine? Thanks.

@JkSelf Could you update the PR description and explain what kind of skew join this PR is trying to resolve.

retest this please

Test build #116546 has started for PR 26434 at commit cee1c8c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21332/
Test PASSed.

Test build #116546 has finished for PR 26434 at commit cee1c8c.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116546/
Test FAILed.

retest this please

Test build #116593 has started for PR 26434 at commit cee1c8c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21375/
Test PASSed.

Test build #116593 has finished for PR 26434 at commit cee1c8c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116593/
Test FAILed.

retest this please

Test build #116610 has started for PR 26434 at commit cee1c8c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21392/
Test PASSed.

Test build #116610 has finished for PR 26434 at commit cee1c8c.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116610/
Test FAILed.

retest this please

Test build #116616 has started for PR 26434 at commit cee1c8c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21397/
Test PASSed.

Test build #116616 has finished for PR 26434 at commit cee1c8c.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116616/
Test FAILed.

Test build #4991 has started for PR 26434 at commit cee1c8c.

Test build #4991 has finished for PR 26434 at commit cee1c8c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #4992 has started for PR 26434 at commit cee1c8c.

Test build #4992 has finished for PR 26434 at commit cee1c8c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

retest this please

Test build #116654 has started for PR 26434 at commit cee1c8c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21433/
Test PASSed.

Test build #116654 has finished for PR 26434 at commit cee1c8c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116654/
Test PASSed.

Test build #116695 has started for PR 26434 at commit ac17a7c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21474/
Test PASSed.

Test build #116695 has finished for PR 26434 at commit ac17a7c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116695/
Test PASSed.

thanks, merging to master!

@JkSelf I am very excited about this work, this will improve the end-user experience for a lot of users. I have left some additional comments; I hope you don't mind that I am late to the party.


Rpc error in validate packages step

See https://github.com/dart-lang/args/actions/runs/4115484830/jobs/7104258106#step:5:113, not sure if we need to do something to give it better credentials or if generally this can be handled better.

Thanks for the report! Things look like they're configured correctly. I think I'll wait until we've seen this a few more times before putting in guard code (perhaps this was an isolated incident w/ github?).
For posterity, the exception was when we tried to write back a 'package validated; tag w/ xxx to publish' comment to the PR.
RpcException: Forbidden
#0      Github.callRestApiPost.<anonymous closure> (package:firehose/src/github.dart:66:13)
<asynchronous suspension>
#1      Github.createComment (package:firehose/src/github.dart:104:18)
<asynchronous suspension>
#2      Firehose.validate (package:firehose/firehose.dart:62:9)
...

Could we be getting rate limited? It looks like you get a 403 forbidden response when that happens.

Possibly? I wouldn't think so, as we're authenticated at the point of the call.

It also looks like we aren't providing a user agent, which the docs say is required, but it clearly works on other repos...

But in any case the increased logging should help figure out what is up :)

I believe this is because the pull request is coming from a "public forked repo", and according to this that only assigns read permission?
It seems wrong that it can't even comment on the PR, but based on the message here "Resource not accessible by integration" that seems to be a likely cause.

There is this note though(from here) :
You can use the permissions key to add and remove read permissions for forked repositories, but typically you can't grant write access. The exception to this behavior is where an admin user has selected the Send write tokens to workflows from pull requests option in the GitHub Actions settings. For more information, see "Managing GitHub Actions settings for a repository."

Yeah, the above makes sense. I don't think I want to elevate permissions for incoming PRs. If we can't rely on being able to create comments on the PR, that may mean pivoting back to just using github action summaries.


Google colab breaks the connection after 5-6 minutes of work.

Google colab breaks the connection after 5-6 minutes of work. The connection may be established for longer, but after I generate about 6 pictures, the connection to Google Collab is broken. After that, it is proposed to reconnect and reload the neural network. I'm using the latest version of Stable Diffusion. Previously, Google did not break the connection. How can I solve this problem?

SD doesn't work on free colab

Google colab breaks the connection after 5-6 minutes of work. The connection may be established for longer, but after I generate about 6 pictures, the connection to Google Collab is broken. After that, it is proposed to reconnect and reload the neural network. I'm using the latest version of Stable Diffusion. Previously, Google did not break the connection. How can I solve this problem?
Here is a working fork of the free colab .
Fork.zip

@Toshik-One if it gets out there, it won't be long before google bans the new "blasphemy" encoding

@Toshik-One if it gets out there, it won't be long before google bans the new "blasphemy" encoding

Okay then, I'd better delete the post.
@TheLastBen   Do you know why there are no problems with RAM in this notebook?
notebook.zip

@Toshik-One this https://github.com/TheLastBen/fast-stable-diffusion/commit/d7bd514dfb11cde4e479c6b9cafd6dfbb67c65a7 fixes the issue

Hi there. I have the same issue.
I am using SD 1.5 With last ben with google colab with pay as you go (100 units). Free colab doesn t work anymore....
I can get process 4 images. and after my ram (12go) is full up and I get disconnected.
It is been a 2 month I haven t use it. It used work ok. getting disconnected after a while but not after 5 min of work.
I guess it is google coba issue?
Why the ram goes full so quickly?
I used SD1.5 With control net (canny depth)
Is there anything we can do before getting more ram with google colab. I guess the montly membersip you get more ram....

in google colab, set the runtime shape to high-RAM to get more ram

thanks. but where

what about https://github.com/TheLastBen/fast-stable-diffusion/commit/d7bd514dfb11cde4e479c6b9cafd6dfbb67c65a7
And I just copied one bit to my already existed google colab and not working. any idea?

try again A1111 and check how much RAM is used

running the cell with https://github.com/TheLastBen/fast-stable-diffusion/commit/d7bd514dfb11cde4e479c6b9cafd6dfbb67c65a7
give me this error

I have no google colab pro+, just pay as you go units
I have set my GPU to A100 (I have no high ram option, I have a100, v100, t4 option)
I have add the https://github.com/TheLastBen/fast-stable-diffusion/commit/d7bd514dfb11cde4e479c6b9cafd6dfbb67c65a7 on the requirement cell and I end up with the same RAM (13 go)

Vous avez besoin de colab pro pour profiter d'une RAM supérieur, ou mieux encore, utilisez ma template sur Runpod, c'est moins cher et plus rapide.

your french is perfect ! thanks


Docs: Adds IPv6 Tunneling Caveat to Networking Concepts

Previously, the network concepts doc did not state an IPv4 networking requirement. Therefore, users are safe to assume that tunnel mode is supported for IPv6-only networks.
This PR adds the IPv4 network support to the list of tunnel mode requirements.
Fixes #30360

cc @squeed

Thanks! Any chance you can squash the two commits, please?

/test


Plans for Python virtualenv support

Are there docs anywhere for how Python virtualenv support will be expanded?  I really like the idea of managing virtualenvs with rtx but there are rough edges right now.  Thought it might be helpful to identify the features that rtx will (and will not) support.

Set a base directory where all virtualenvs will be created.  Ala WORKON_HOME from virtualenvwrapper.
Add cli command(s) or extend use to manage virtualenvs: e.g. rtx use --venv<EMAIL_ADDRESS>How can multiple Python versions but also venvs be automatically supported?  E.g. rtx use --venv-prefix myproj<EMAIL_ADDRESS><EMAIL_ADDRESS>creates myproj-3.10 and myproj-3.11 virtualenvs.
Fix: rtx use<EMAIL_ADDRESS>will cause virtualenv settings to be removed in .rtx.toml
rtx should prompt before removing everything at the path of a virtualenv.  Currently, if you mistype the location, you could end up with big problems since that path is mercilessly replaced with a virtualenv.
Support for passing commands through to the venv creation?
Support creating venvs with virtualenv?

can't you just use an env var?
at least for right now I don't see myself modifying the rtx CLI for python-specific commands like this
you might be able to do this right now with templates
this is a bug that should be fixed, though it's a little tricky. What needs to happen is if a plugin exists it should retain the options I think.
this is harder to fix than it sounds. rtx isn't doing any deleting I don't think, that's all python.
like?
isn't that just for python2?

In general I don't see expanding venv support too much, except for better support for poetry and other build tools. The rough edges should be resolved, but this is a complex feature to support and I'm not a python developer so I want to keep the feature creep to a minimum.
The main purpose for including it at all was so python users wouldn't need to get direnv to work with rtx which was kind of a pain.

I agree that it would be nice to be able to add the venv part of python = {version='3.11', virtualenv='.venv'} from the CLI (some variation of option 2). Perhaps support for this could be done via a plugin?

@rsyring you can try pdm plugin
when you run pdm install, pdm create .venv in your project root.
with this config, when you cd into your project root, rtx will run source .venv/bin/activate for you automatically
$ cat .rtx.toml
[tools]
python = {version='3.10.12', virtualenv='.venv'}
pdm = {version='2.7.4'}

Per #690 - the time it takes to create the virtualenv is significant from a UX perspective.  Should at least let the user know what is happening: "Creating Python virtualenv at: /some/file/path..."

venv creation is now optional and behind a setting, defaulting to disabled: https://github.com/jdx/mise/discussions/1357
I also plan to move this configuration from the [tools] section to the [env] section #1303
I'm going to close this one because those tickets are more actionable.


96 attachment no

Regarding issue #96. Fixed a few errors and now it seems to work -- you can link to attachments without using the attachment id.

Merged with modifications in 1adf410. Thanks @ostrokach


Feature Request : minify generated Javascript

Describe the problem
I deployed a Sveltekit website on Vercel
I tried a lighthouse test, it recommends that the JS should be minified, that it could help save close to 1 second for the user to load the entire page
Edit : I confirm the scripts loaded & not minified are from Sveltekit & not static or 3rd party loaded scripts
I also opend manually the files & saw they could indeed be minified because there were quite a bunch of whitespace
I think this should be a no brainer if the functionnality is the same and the user experience is better
Describe the proposed solution
Probably an option to activate / deactivate this for builds
I would argue minified should be the default
Alternatives considered
Maybe compress with a script after builds
Importance
nice to have
Additional Information
No response

It is the default. We need a minimal reproducible as the issue template asked you for if it's not.
(Hint: Look at the files in your network tab instead of the server files)


Router

UI libraries often offer routing – an idiomatic mapping from URLs to application state and back.
Laminar does not currently have this built in. Unlike other browser integrations such as AJAX / networking, routing might be quite daunting to implement yourself, so I think Laminar should have it.
I do have a working Laminar Router implementation that I am quite happy with, however at the moment I do not have the time to bring it to the level of quality that I am comfortable subjecting other people to, and thus it remains unpublished.
I did share most of my router implementation here: https://gitter.im/Laminar_/Lobby?at=5c57f1338aa5ca5abf7fd85d The missing part is my mechanism of extracting data from URLs as well as populating URL templates with said data. That's the part I'm not quite happy with. I wanted to avoid bringing in a heavyweight third party parser, and my own implementation was optimized for development speed / PoC rather than a public release.

Evaluate https://github.com/sparsetech/trail for this. Not having a decent parser is my main gripe with my own implementation, and this looks pretty good.

This is definitely one of the big pain points for us to adopt Laminar.

Understandable. What library / router are you using now, and how do you like it?

Also, I am currently working on solving issues tagged as hard problem, and that will change Laminar's API somewhat. I'd rather do those changes first before implementing a respectable router solution.
Meanwhile, if anyone has suggestions for URL matching / building libraries like https://github.com/sparsetech/trail please post them here.

If I may, I can (shamelessly) suggest my own library : https://github.com/sherpal/url-dsl.
I've actually made it especially to use it with Laminar, in order to cope for the lack of router.
The way I use it for a Laminar router is quite straightforward. In the library, path and query parameters can output a type T from a url. A Route can then simply take an instance with type T, together with a function from T to a reactive element. And the router is a reactive list of routes that listen to changes in the document url.
I've used it for one of my project, it's quite easy to use.

If I may, I can (shamelessly) suggest my own library : https://github.com/sherpal/url-dsl.

@sherpal  The link is wrong because the label and url in the link markup are swapped, it should be: url-dsl

Finally got around to this. Here's my new Laminar router: https://github.com/raquo/Waypoint

awesome! Thanks


fix: drop deprecated apiextensions.k8s.io/v1beta1 CustomResourceDefinition

It was used in Tekton and kubernetes/cluster.go
fixes #204

I think you should run make embed_files since files in the embedded-files folder have changed.

Right, thanks. However why do we track generated files (statik.go) in git? Those belong into a release tarball, but not git :thinking:

Quicker ramp up for developers without the tooling to generate it ?
(This is also a point for our docs-to-be, list the tooling we use, and how to get/install it)

Quicker ramp up for developers without the tooling to generate it ?
(This is also a point for our docs-to-be, list the tooling we use, and how to get/install it)

statik is used in the Makefile, so without it, you can't even run make build.  That target even tries to install it (every time).

gitlint spotting the important things 😈 : 1: T1 Title exceeds max length (74>72): "fix: drop deprecated apiextensions.k8s.io/v1beta1 CustomResourceDefinition"

Superseded by #318


Use volatile to fix serialize issue

-Ref: https://github.com/appscode-ci/appscode-login-plugin/issues/12

volatile is not applicable for final field.
So Cache are remain unchanged.


Increment SDK version to 0.4.8

Updated GenerativeAISwift.version to 0.4.8 in preparation for the next release.

Note: I'm not intending to tag 0.4.8 right away, just don't want to forget to increment this before our next release.


KubernetesDiscoveryClientAutoConfiguration issue

After upgrading from Spring Cloud 2020.0.3 to 2020.0.5, we are encountering the following exception:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kubernetesInformerDiscoveryClient' defined in class path resource [org/springframework/cloud/kubernetes/client/discovery/KubernetesDiscoveryClientAutoConfiguration$KubernetesInformerDiscoveryConfiguration.class]: Invocation of init method failed; nested exception is java.lang.IllegalStateException: Timeout waiting for informers cache to be ready, is the kubernetes service up?

We are using Spring Boot 2.5.8, and after reverting back to Spring Cloud 2020.0.3, this exception goes away and our k8s microservices start successfully. We previously tried upgrading to 2020.0.4, but there was an issue  running our microservices in Minikube that prevented us from upgrading.

Can you provide a complete, minimal, verifiable sample that reproduces the problem? It should be available as a GitHub (or similar) project or attached to this issue as a zip file.

Unfortunately, the organization I work for strictly prohibits sharing any code, so I am not able to provide you an example given the nature of the project we are working on.

Not even if it's not the code from the project?
We just need a way to reproduce it.

I will try to put something together that reproduces what we are encountering.

Looking into our code a bit more, at this point, we really aren't leveraging the Discovery Client feature yet, and I'm not sure if we will. We do have @EnableDiscoveryClient specified at the moment, but we aren't leveraging this feature as we connect microservices to each other via their k8s service names. So, would you recommend that I disable this feature in our microservices for the time being at least?

Sure you can disable it, but I would still be curious how to reproduce the problem.

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.