[SPARK-29375][SPARK-28940][SPARK-32041][SQL] Whole plan exchange and subquery reuse

What changes were proposed in this pull request?
This PR:

Fixes an issue in ReuseExchange rule that can result a ReusedExchange node pointing to an invalid exchange. This can happen due to the 2 separate traversals in ReuseExchange when the 2nd traversal modifies an exchange that has already been referenced (reused) in the 1st traversal.
Consider the following query:
WITH t AS (
  SELECT df1.id, df2.k
  FROM df1 JOIN df2 ON df1.k = df2.k
  WHERE df2.id < 2
)
SELECT * FROM t AS a JOIN t AS b ON a.id = b.id

Before this PR the plan of the query was (note the <== this reuse node points to a non-existing node marker):
== Physical Plan ==
*(7) SortMergeJoin [id#14L], [id#18L], Inner
:- *(3) Sort [id#14L ASC NULLS FIRST], false, 0
:  +- Exchange hashpartitioning(id#14L, 5), true, [id=#298]
:     +- *(2) Project [id#14L, k#17L]
:        +- *(2) BroadcastHashJoin [k#15L], [k#17L], Inner, BuildRight
:           :- *(2) Project [id#14L, k#15L]
:           :  +- *(2) Filter isnotnull(id#14L)
:           :     +- *(2) ColumnarToRow
:           :        +- FileScan parquet default.df1[id#14L,k#15L] Batched: true, DataFilters: [isnotnull(id#14L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/petertoth/git/apache/spark/sql/core/spark-warehouse/org.apache.spar..., PartitionFilters: [isnotnull(k#15L), dynamicpruningexpression(k#15L IN dynamicpruning#26)], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>
:           :              +- SubqueryBroadcast dynamicpruning#26, 0, [k#17L], [id=#289]
:           :                 +- ReusedExchange [k#17L], BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true])), [id=#179]
:           +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true])), [id=#179]
:              +- *(1) Project [k#17L]
:                 +- *(1) Filter ((isnotnull(id#16L) AND (id#16L < 2)) AND isnotnull(k#17L))
:                    +- *(1) ColumnarToRow
:                       +- FileScan parquet default.df2[id#16L,k#17L] Batched: true, DataFilters: [isnotnull(id#16L), (id#16L < 2), isnotnull(k#17L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/petertoth/git/apache/spark/sql/core/spark-warehouse/org.apache.spar..., PartitionFilters: [], PushedFilters: [IsNotNull(id), LessThan(id,2), IsNotNull(k)], ReadSchema: struct<id:bigint,k:bigint>
+- *(6) Sort [id#18L ASC NULLS FIRST], false, 0
   +- ReusedExchange [id#18L, k#21L], Exchange hashpartitioning(id#14L, 5), true, [id=#184] <== this reuse node points to a non-existing node

After this PR:
== Physical Plan ==
*(7) SortMergeJoin [id#14L], [id#18L], Inner
:- *(3) Sort [id#14L ASC NULLS FIRST], false, 0
:  +- Exchange hashpartitioning(id#14L, 5), true, [id=#231]
:     +- *(2) Project [id#14L, k#17L]
:        +- *(2) BroadcastHashJoin [k#15L], [k#17L], Inner, BuildRight
:           :- *(2) Project [id#14L, k#15L]
:           :  +- *(2) Filter isnotnull(id#14L)
:           :     +- *(2) ColumnarToRow
:           :        +- FileScan parquet default.df1[id#14L,k#15L] Batched: true, DataFilters: [isnotnull(id#14L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/petertoth/git/apache/spark/sql/core/spark-warehouse/org.apache.spar..., PartitionFilters: [isnotnull(k#15L), dynamicpruningexpression(k#15L IN dynamicpruning#26)], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>
:           :              +- SubqueryBroadcast dynamicpruning#26, 0, [k#17L], [id=#103]
:           :                 +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true])), [id=#102]
:           :                    +- *(1) Project [k#17L]
:           :                       +- *(1) Filter ((isnotnull(id#16L) AND (id#16L < 2)) AND isnotnull(k#17L))
:           :                          +- *(1) ColumnarToRow
:           :                             +- FileScan parquet default.df2[id#16L,k#17L] Batched: true, DataFilters: [isnotnull(id#16L), (id#16L < 2), isnotnull(k#17L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/petertoth/git/apache/spark/sql/core/spark-warehouse/org.apache.spar..., PartitionFilters: [], PushedFilters: [IsNotNull(id), LessThan(id,2), IsNotNull(k)], ReadSchema: struct<id:bigint,k:bigint>
:           +- ReusedExchange [k#17L], BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true])), [id=#102]
+- *(6) Sort [id#18L ASC NULLS FIRST], false, 0
   +- ReusedExchange [id#18L, k#21L], Exchange hashpartitioning(id#14L, 5), true, [id=#231]

Fixes an issue with separate consecutive ReuseExchange and ReuseSubquery rules that can result a ReusedExchange node pointing to an invalid exchange. This can happen due to the 2 separate rules when ReuseSubquery rule modifies an exchange that has already been referenced (reused) in ReuseExchange rule.
Consider the following query:
WITH t AS (
  SELECT df1.id, df2.k
  FROM df1 JOIN df2 ON df1.k = df2.k
  WHERE df2.id < 2
),
t2 AS (
  SELECT * FROM t
  UNION
  SELECT * FROM t
)
SELECT * FROM t2 AS a JOIN t2 AS b ON a.id = b.id

Before this PR the plan of the query was (note the <== this reuse node points to a non-existing node marker):
== Physical Plan ==
*(15) SortMergeJoin [id#46L], [id#58L], Inner
:- *(7) Sort [id#46L ASC NULLS FIRST], false, 0
:  +- Exchange hashpartitioning(id#46L, 5), true, [id=#979]
:     +- *(6) HashAggregate(keys=[id#46L, k#49L], functions=[])
:        +- Exchange hashpartitioning(id#46L, k#49L, 5), true, [id=#975]
:           +- *(5) HashAggregate(keys=[id#46L, k#49L], functions=[])
:              +- Union
:                 :- *(2) Project [id#46L, k#49L]
:                 :  +- *(2) BroadcastHashJoin [k#47L], [k#49L], Inner, BuildRight
:                 :     :- *(2) Project [id#46L, k#47L]
:                 :     :  +- *(2) Filter isnotnull(id#46L)
:                 :     :     +- *(2) ColumnarToRow
:                 :     :        +- FileScan parquet default.df1[id#46L,k#47L] Batched: true, DataFilters: [isnotnull(id#46L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/petertoth/git/apache/spark/sql/core/spark-warehouse/org.apache.spar..., PartitionFilters: [isnotnull(k#47L), dynamicpruningexpression(k#47L IN dynamicpruning#66)], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>
:                 :     :              +- SubqueryBroadcast dynamicpruning#66, 0, [k#49L], [id=#926]
:                 :     :                 +- ReusedExchange [k#49L], BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true])), [id=#656]
:                 :     +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true])), [id=#656]
:                 :        +- *(1) Project [k#49L]
:                 :           +- *(1) Filter ((isnotnull(id#48L) AND (id#48L < 2)) AND isnotnull(k#49L))
:                 :              +- *(1) ColumnarToRow
:                 :                 +- FileScan parquet default.df2[id#48L,k#49L] Batched: true, DataFilters: [isnotnull(id#48L), (id#48L < 2), isnotnull(k#49L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/petertoth/git/apache/spark/sql/core/spark-warehouse/org.apache.spar..., PartitionFilters: [], PushedFilters: [IsNotNull(id), LessThan(id,2), IsNotNull(k)], ReadSchema: struct<id:bigint,k:bigint>
:                 +- *(4) Project [id#46L, k#49L]
:                    +- *(4) BroadcastHashJoin [k#47L], [k#49L], Inner, BuildRight
:                       :- *(4) Project [id#46L, k#47L]
:                       :  +- *(4) Filter isnotnull(id#46L)
:                       :     +- *(4) ColumnarToRow
:                       :        +- FileScan parquet default.df1[id#46L,k#47L] Batched: true, DataFilters: [isnotnull(id#46L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/petertoth/git/apache/spark/sql/core/spark-warehouse/org.apache.spar..., PartitionFilters: [isnotnull(k#47L), dynamicpruningexpression(k#47L IN dynamicpruning#66)], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>
:                       :              +- ReusedSubquery SubqueryBroadcast dynamicpruning#66, 0, [k#49L], [id=#926]
:                       +- ReusedExchange [k#49L], BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true])), [id=#656]
+- *(14) Sort [id#58L ASC NULLS FIRST], false, 0
   +- ReusedExchange [id#58L, k#61L], Exchange hashpartitioning(id#46L, 5), true, [id=#761] <== this reuse node points to a non-existing node

After this PR:
== Physical Plan ==
*(15) SortMergeJoin [id#46L], [id#58L], Inner
:- *(7) Sort [id#46L ASC NULLS FIRST], false, 0
:  +- Exchange hashpartitioning(id#46L, 5), true, [id=#793]
:     +- *(6) HashAggregate(keys=[id#46L, k#49L], functions=[])
:        +- Exchange hashpartitioning(id#46L, k#49L, 5), true, [id=#789]
:           +- *(5) HashAggregate(keys=[id#46L, k#49L], functions=[])
:              +- Union
:                 :- *(2) Project [id#46L, k#49L]
:                 :  +- *(2) BroadcastHashJoin [k#47L], [k#49L], Inner, BuildRight
:                 :     :- *(2) Project [id#46L, k#47L]
:                 :     :  +- *(2) Filter isnotnull(id#46L)
:                 :     :     +- *(2) ColumnarToRow
:                 :     :        +- FileScan parquet default.df1[id#46L,k#47L] Batched: true, DataFilters: [isnotnull(id#46L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/petertoth/git/apache/spark/sql/core/spark-warehouse/org.apache.spar..., PartitionFilters: [isnotnull(k#47L), dynamicpruningexpression(k#47L IN dynamicpruning#66)], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>
:                 :     :              +- SubqueryBroadcast dynamicpruning#66, 0, [k#49L], [id=#485]
:                 :     :                 +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true])), [id=#484]
:                 :     :                    +- *(1) Project [k#49L]
:                 :     :                       +- *(1) Filter ((isnotnull(id#48L) AND (id#48L < 2)) AND isnotnull(k#49L))
:                 :     :                          +- *(1) ColumnarToRow
:                 :     :                             +- FileScan parquet default.df2[id#48L,k#49L] Batched: true, DataFilters: [isnotnull(id#48L), (id#48L < 2), isnotnull(k#49L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/petertoth/git/apache/spark/sql/core/spark-warehouse/org.apache.spar..., PartitionFilters: [], PushedFilters: [IsNotNull(id), LessThan(id,2), IsNotNull(k)], ReadSchema: struct<id:bigint,k:bigint>
:                 :     +- ReusedExchange [k#49L], BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true])), [id=#484]
:                 +- *(4) Project [id#46L, k#49L]
:                    +- *(4) BroadcastHashJoin [k#47L], [k#49L], Inner, BuildRight
:                       :- *(4) Project [id#46L, k#47L]
:                       :  +- *(4) Filter isnotnull(id#46L)
:                       :     +- *(4) ColumnarToRow
:                       :        +- FileScan parquet default.df1[id#46L,k#47L] Batched: true, DataFilters: [isnotnull(id#46L)], Format: Parquet, Location: InMemoryFileIndex[file:/Users/petertoth/git/apache/spark/sql/core/spark-warehouse/org.apache.spar..., PartitionFilters: [isnotnull(k#47L), dynamicpruningexpression(k#47L IN dynamicpruning#66)], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>
:                       :              +- ReusedSubquery SubqueryBroadcast dynamicpruning#66, 0, [k#49L], [id=#485]
:                       +- ReusedExchange [k#49L], BroadcastExchange HashedRelationBroadcastMode(List(input[0, bigint, true])), [id=#484]
+- *(14) Sort [id#58L ASC NULLS FIRST], false, 0
   +- ReusedExchange [id#58L, k#61L], Exchange hashpartitioning(id#46L, 5), true, [id=#793]

(This example contains issue 1 as well.)

Improves the reuse of exchanges and subqueries by enabling reuse across the whole plan. This means that the new combined rule utilizes the reuse opportunities between parent and subqueries by traversing the whole plan. The traversal is started on the top level query only.

This PR fixes the above 3 issues by unifying the separate rules into a ReuseExchangeAndSubquery rule that does a 1 pass, whole-plan, bottom-up traversal.
Why are the changes needed?
Performance improvement.
How was this patch tested?

New UTs in ReuseExchangeAndSubquerySuite to cover 1. and 2.
New UTs in DynamicPartitionPruningSuite, SubquerySuite and ExchangeSuite to cover 3.
TPCDS benchmarks.

Test build #124334 has started for PR 28885 at commit dbd8606.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/28953/
Test PASSed.

I measured considerable improvement (~30%) on TPCDS Q14a, Q14b, Q23a, Q23b, Q47 and Q57 queries.

cc @cloud-fan, @dongjoon-hyun

I've just realized that this PR partly overlaps with https://github.com/apache/spark/pull/28881. @prakharjain09 opened it 3 hours before, but my PR does a bit more than that and actually does the combined reuse in a bit different way so I wouldn't close mine yet.

Test build #124334 has finished for PR 28885 at commit dbd8606.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class WholePlanReuse(conf: SQLConf) extends Rule[SparkPlan]

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124334/
Test PASSed.

Thank you for pinging me, @peter-toth .

cc @maryannxue too FYI

Test build #124538 has started for PR 28885 at commit 8eca64a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29159/
Test PASSed.

Test build #124538 has finished for PR 28885 at commit 8eca64a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124538/
Test PASSed.

Test build #124572 has started for PR 28885 at commit c49a0f9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29193/
Test PASSed.

Test build #124572 has finished for PR 28885 at commit c49a0f9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124572/
Test PASSed.

Test build #124602 has started for PR 28885 at commit 2b3cde2.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29223/
Test PASSed.

Thanks for doing this! I think the idea of whole plan reuse is good and your approach is correct, but I think some parts can be done differently IMO, I left some comments.

Thanks for the review and comments @dbaliafroozeh!

Test build #124602 has finished for PR 28885 at commit 2b3cde2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124602/
Test PASSed.

Changes look good to me, added 3 minor comments.

Thanks @dbaliafroozeh.
@cloud-fan, @maryannxue could you review this PR?

Test build #124663 has started for PR 28885 at commit fe78a31.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29279/
Test PASSed.

Test build #124663 has finished for PR 28885 at commit fe78a31.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124663/
Test FAILed.

Test build #124679 has started for PR 28885 at commit 8b50e81.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29294/
Test PASSed.

Test build #124679 has finished for PR 28885 at commit 8b50e81.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124679/
Test FAILed.

Test build #124753 has started for PR 28885 at commit fd522ea.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29365/
Test PASSed.

Test build #124753 has finished for PR 28885 at commit fd522ea.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124753/
Test FAILed.

retest this please

Test build #124810 has started for PR 28885 at commit fd522ea.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/29420/
Test PASSed.

Test build #124810 has finished for PR 28885 at commit fd522ea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/124810/
Test PASSed.

@cloud-fan, @maryannxue, @maropu, @viirya could you please review this PR?

I've added a few more comments to explain how the PR works.
@cloud-fan, @maryannxue, @maropu, @viirya please let me know if you have any concerns, suggestions or you are missing something from this PR.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/30664/
Test PASSed.

Test build #126051 has started for PR 28885 at commit 100e5c2.

I've updated the description and added some simpler test cases than TPCDS Q14a to repro the issue and test the fix.

Test build #126051 has finished for PR 28885 at commit 100e5c2.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ReuseExchangeAndSubquerySuite extends SparkPlanTest with SharedSparkSession

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126051/
Test FAILed.

Merged build finished. Test PASSed.

Test build #126174 has started for PR 28885 at commit 0d18e28.

Test build #126174 has finished for PR 28885 at commit 0d18e28.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126174/
Test PASSed.

@cloud-fan, @maryannxue, @maropu, @viirya please let me know if you have any concerns, suggestions or you are missing something from this PR.

Thanks @squito for the review, very appreciated.

I discussed this offline with Peter a little bit. This is not my area of expertise, so take my comments with a grain of salt.

I think the internal details of ReuseMap are a little more complicated than they need to be, given the main purpose of this PR.  I'd prefer to defer the optimizations to a separate change, and keep the initial implementation simple.

I'm fine with this. I can revert it and some UTs to it in the next commit. If anyone has any comments on this, please let us know.

I'd like to see some unit tests on ReuseMap.  Even just really simple stuff, but at least making sure that if there are two things with a matching schema, you get the right one.

Ok, I will add it soon.

You mention a few cases when a ReusedExchange refers to something which no longer exists.  Does spark outright fail in those cases?  if so, I'd highlight that in the description, that these are bugs and not just performance improvements (maybe this obvious to everyone other than me)

As I mentioned I don't know how to categorize this kind of issue, it doesn't affect the correctness of the result, but can cause serious performance issues: https://github.com/apache/spark/pull/28885#discussion_r459273107

I'm not totally clear which test covers part 3 of your description -- I guess its ExchangeSuite."Exchange reuse across the whole plan"? or maybe just part of it?  A comment in the PR description at least, and perhaps even one on the test itself would be helpful.

is about whole plan reuse, this naturally comes from the 1 pass, whole plan, bottom-up traversal I suggest in this PR to fix 1. and 2. Please note that currently ExchangeReuse already does a limited version of multi-level exchange reuse to support DPP :

    } transformAllExpressions {
      // Lookup inside subqueries for duplicate exchanges
      case in: InSubqueryExec =>
        val newIn = in.plan.transformUp {
          case exchange: Exchange => reuse(exchange)
        }
        in.copy(plan = newIn.asInstanceOf[BaseSubqueryExec])
    }

I don't see a reason why we couldn't extend this to other type of subqueries (ScalarSubquery) and to all levels of the plan which results the very clean new code in ReuseExchangeAndSubquery.
I added test cases  to DynamicPartitionPruningSuite, SubquerySuite and ExchangeSuite cover it.

Test build #126408 has started for PR 28885 at commit 5df4c53.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31022/
Test PASSed.

Commit https://github.com/apache/spark/pull/28885/commits/eaf4ad046ce4de55f71fc195daa18d6b9a8af520 adds UT coverage to ReuseMap.

Test build #126408 has finished for PR 28885 at commit 5df4c53.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ReuseMap[T <: T2, T2 <: QueryPlan[T2]]

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126408/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31545/
Test PASSed.

Test build #126921 has started for PR 28885 at commit 5d5fd04.

Test build #126921 has finished for PR 28885 at commit 5d5fd04.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/126921/
Test PASSed.

@cloud-fan, @maryannxue, @maropu, @viirya could you please help and review this PR?

Test build #128013 has started for PR 28885 at commit cd064e4.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32639/
Test PASSed.

Test build #128013 has finished for PR 28885 at commit cd064e4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128013/
Test PASSed.

@cloud-fan, @maropu, @viirya, can you please help me how to move forward with this PR?
The latest commit updates expected plans of PlanStability suites where you can see the new reuse nodes this PR adds to TPCDS queries. My benchmarks showed that this PR brings ~30% improvement to some of the queries. Please let me know if you have any concerns with this PR.

Test build #128590 has started for PR 28885 at commit 050ab40.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33215/
Test PASSed.

Test build #128590 has finished for PR 28885 at commit 050ab40.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128590/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33217/
Test PASSed.

Test build #128592 has started for PR 28885 at commit e1db0d4.

Test build #128592 has finished for PR 28885 at commit e1db0d4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128592/
Test PASSed.

Test build #129500 has started for PR 28885 at commit e0f1ee1.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34106/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34106/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34106/
Test PASSed.

Test build #129500 has finished for PR 28885 at commit e0f1ee1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129500/
Test FAILed.

Test build #129528 has started for PR 28885 at commit 913483c.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34133/
Test FAILed.

Test build #129528 has finished for PR 28885 at commit 913483c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129528/
Test FAILed.

Test build #129542 has started for PR 28885 at commit abf0a4a.

Test build #129546 has started for PR 28885 at commit 012ef8f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34148/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34148/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34148/
Test PASSed.

Test build #129546 has finished for PR 28885 at commit 012ef8f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #129542 has finished for PR 28885 at commit abf0a4a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129546/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129542/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34152/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34152/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34152/
Test PASSed.

retest this please

Test build #129552 has started for PR 28885 at commit 012ef8f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34158/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34158/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34158/
Test PASSed.

Test build #129552 has finished for PR 28885 at commit 012ef8f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129552/
Test PASSed.

Test build #129637 has started for PR 28885 at commit ab2f369.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34241/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34241/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34241/
Test PASSed.

Test build #129637 has finished for PR 28885 at commit ab2f369.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129637/
Test PASSed.

Test build #129680 has started for PR 28885 at commit b4bcbce.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34285/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34285/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34285/
Test FAILed.

Test build #129680 has finished for PR 28885 at commit b4bcbce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129680/
Test PASSed.

@cloud-fan, @maropu, @viirya any thoughts on this PR?

Test build #134232 has started for PR 28885 at commit 7e1e5bf.

Test build #134232 has started for PR 28885 at commit 7e1e5bf.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38817/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38817/

Test build #134232 has finished for PR 28885 at commit 7e1e5bf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134232 has finished for PR 28885 at commit 7e1e5bf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134232/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134232/

I've updated this PR with minor changes due to AQE is now enabled by default.
Without this PR invalid reuse nodes are in many TPCDS queries causing performance degradation: q14a, q14b, q23a, q23b, q47, q57, q5a-v2.7, q14-v2.7, q14a-v2.7, q22a-v2.7, q36a-v2.7, q47-v2.7, q51a-v2.7, q57-v2.7, q67a-v2.7, q70a-v2.7, q77a-v2.7, q80a-v2.7, q86a-v2.7
https://github.com/apache/spark/pull/31243 updates PlanStabilitySuite tests to cover partitioned tables (DPP kicks in) and the issue appears as ReusedExchange [Reuses operator id: unknown] nodes in explain.txts of that PR.
@cloud-fan @viirya @wangyum could you please review this PR?

I've updated this PR with minor changes due to AQE is now enabled by default.
Without this PR invalid reuse nodes are in many TPCDS queries causing performance degradation: q14a, q14b, q23a, q23b, q47, q57, q5a-v2.7, q14-v2.7, q14a-v2.7, q22a-v2.7, q36a-v2.7, q47-v2.7, q51a-v2.7, q57-v2.7, q67a-v2.7, q70a-v2.7, q77a-v2.7, q80a-v2.7, q86a-v2.7
https://github.com/apache/spark/pull/31243 updates PlanStabilitySuite tests to cover partitioned tables (DPP kicks in) and the issue appears as ReusedExchange [Reuses operator id: unknown] nodes in explain.txts of that PR.
@cloud-fan @viirya @wangyum could you please review this PR?

Test build #134415 has started for PR 28885 at commit 8f24add.

Test build #134415 has started for PR 28885 at commit 8f24add.

Test build #134415 has finished for PR 28885 at commit 8f24add.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134415 has finished for PR 28885 at commit 8f24add.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39001/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39001/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134415/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134415/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39001/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39001/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39001/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39001/

Test build #134453 has started for PR 28885 at commit 4db5801.

Test build #134453 has started for PR 28885 at commit 4db5801.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39039/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39039/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39039/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39039/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39039/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39039/

Test build #134453 has finished for PR 28885 at commit 4db5801.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134453 has finished for PR 28885 at commit 4db5801.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134453/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134453/

Now that https://github.com/apache/spark/pull/31243 got merged the invalid reuse references (Reuses operator id: unknown) show up in many golden files. Each of them mean a missed reuse opportunity and a wrong explain plan.
Before this PR:
tpcds-plan-stability petertoth$ grep -Ril "Reuses operator id: unknown" . | sort
./approved-plans-v1_4/q14a.sf100/explain.txt
./approved-plans-v1_4/q14a/explain.txt
./approved-plans-v1_4/q14b.sf100/explain.txt
./approved-plans-v1_4/q14b/explain.txt
./approved-plans-v1_4/q23a.sf100/explain.txt
./approved-plans-v1_4/q23a/explain.txt
./approved-plans-v1_4/q23b.sf100/explain.txt
./approved-plans-v1_4/q23b/explain.txt
./approved-plans-v1_4/q47.sf100/explain.txt
./approved-plans-v1_4/q47/explain.txt
./approved-plans-v1_4/q57.sf100/explain.txt
./approved-plans-v1_4/q57/explain.txt
./approved-plans-v2_7/q14.sf100/explain.txt
./approved-plans-v2_7/q14/explain.txt
./approved-plans-v2_7/q14a.sf100/explain.txt
./approved-plans-v2_7/q14a/explain.txt
./approved-plans-v2_7/q18a.sf100/explain.txt
./approved-plans-v2_7/q22a.sf100/explain.txt
./approved-plans-v2_7/q22a/explain.txt
./approved-plans-v2_7/q36a.sf100/explain.txt
./approved-plans-v2_7/q36a/explain.txt
./approved-plans-v2_7/q47.sf100/explain.txt
./approved-plans-v2_7/q47/explain.txt
./approved-plans-v2_7/q51a.sf100/explain.txt
./approved-plans-v2_7/q51a/explain.txt
./approved-plans-v2_7/q57.sf100/explain.txt
./approved-plans-v2_7/q57/explain.txt
./approved-plans-v2_7/q5a.sf100/explain.txt
./approved-plans-v2_7/q5a/explain.txt
./approved-plans-v2_7/q67a.sf100/explain.txt
./approved-plans-v2_7/q67a/explain.txt
./approved-plans-v2_7/q70a.sf100/explain.txt
./approved-plans-v2_7/q70a/explain.txt
./approved-plans-v2_7/q77a.sf100/explain.txt
./approved-plans-v2_7/q77a/explain.txt
./approved-plans-v2_7/q80a.sf100/explain.txt
./approved-plans-v2_7/q80a/explain.txt
./approved-plans-v2_7/q86a.sf100/explain.txt
./approved-plans-v2_7/q86a/explain.txt

After this PR:
tpcds-plan-stability petertoth$ grep -Ril "Reuses operator id: unknown" . | sort
tpcds-plan-stability petertoth$

Now that https://github.com/apache/spark/pull/31243 got merged the invalid reuse references (Reuses operator id: unknown) show up in many golden files. Each of them mean a missed reuse opportunity and a wrong explain plan.
Before this PR:
tpcds-plan-stability petertoth$ grep -Ril "Reuses operator id: unknown" . | sort
./approved-plans-v1_4/q14a.sf100/explain.txt
./approved-plans-v1_4/q14a/explain.txt
./approved-plans-v1_4/q14b.sf100/explain.txt
./approved-plans-v1_4/q14b/explain.txt
./approved-plans-v1_4/q23a.sf100/explain.txt
./approved-plans-v1_4/q23a/explain.txt
./approved-plans-v1_4/q23b.sf100/explain.txt
./approved-plans-v1_4/q23b/explain.txt
./approved-plans-v1_4/q47.sf100/explain.txt
./approved-plans-v1_4/q47/explain.txt
./approved-plans-v1_4/q57.sf100/explain.txt
./approved-plans-v1_4/q57/explain.txt
./approved-plans-v2_7/q14.sf100/explain.txt
./approved-plans-v2_7/q14/explain.txt
./approved-plans-v2_7/q14a.sf100/explain.txt
./approved-plans-v2_7/q14a/explain.txt
./approved-plans-v2_7/q18a.sf100/explain.txt
./approved-plans-v2_7/q22a.sf100/explain.txt
./approved-plans-v2_7/q22a/explain.txt
./approved-plans-v2_7/q36a.sf100/explain.txt
./approved-plans-v2_7/q36a/explain.txt
./approved-plans-v2_7/q47.sf100/explain.txt
./approved-plans-v2_7/q47/explain.txt
./approved-plans-v2_7/q51a.sf100/explain.txt
./approved-plans-v2_7/q51a/explain.txt
./approved-plans-v2_7/q57.sf100/explain.txt
./approved-plans-v2_7/q57/explain.txt
./approved-plans-v2_7/q5a.sf100/explain.txt
./approved-plans-v2_7/q5a/explain.txt
./approved-plans-v2_7/q67a.sf100/explain.txt
./approved-plans-v2_7/q67a/explain.txt
./approved-plans-v2_7/q70a.sf100/explain.txt
./approved-plans-v2_7/q70a/explain.txt
./approved-plans-v2_7/q77a.sf100/explain.txt
./approved-plans-v2_7/q77a/explain.txt
./approved-plans-v2_7/q80a.sf100/explain.txt
./approved-plans-v2_7/q80a/explain.txt
./approved-plans-v2_7/q86a.sf100/explain.txt
./approved-plans-v2_7/q86a/explain.txt

After this PR:
tpcds-plan-stability petertoth$ grep -Ril "Reuses operator id: unknown" . | sort
tpcds-plan-stability petertoth$

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39471/

Test build #134885 has started for PR 28885 at commit ec2324a.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39471/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39471/

Test build #134885 has finished for PR 28885 at commit ec2324a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134885/

Test build #135083 has started for PR 28885 at commit c3cbdb2.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39665/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39665/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39665/

Test build #135083 has finished for PR 28885 at commit c3cbdb2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
trait UserDefinedExpression
 implicit class MetadataColumnHelper(attr: Attribute)
case class SubqueryAdaptiveBroadcastExec(
case class PlanAdaptiveDynamicPruningFilters(
case class PlanAdaptiveSubqueries(

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135083/

@cloud-fan could you please help me with this PR?
The incorrect reuse nodes cause performance issues currently and in special cases they can even cause performance regression when someone upgrades from 2.x to 3.x. An example is TPCDS q23b in which the second traversal in ReuseExchange rule (added with DPP in Spark 3) ruins a better, bigger reuse node.
I see that AQE is the main way now, where these issues doesn't come up, but I think we are still far from full AQE support and we should fix the non-adaptive path as well.

This is till a serious issue in Spark 3. I've updated the PR with the latest commit from master.

Test build #136257 has started for PR 28885 at commit 8822acc.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40839/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40839/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40839/

Test build #136257 has finished for PR 28885 at commit 8822acc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136257/

Test build #139857 has started for PR 28885 at commit bf29f1a.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44386/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44386/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44386/

Test build #139857 has finished for PR 28885 at commit bf29f1a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ReuseMap[T <: T2, T2 <: QueryPlan[T2]]

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139857/

What's still WIP? This idea LGTM and I'll take a look when the code is ready for review.

What's still WIP? This idea LGTM and I'll take a look when the code is ready for review.

Thanks! Actually it's not WIP, I just rebased it on https://github.com/apache/spark/pull/32885 yesterday and waited for the tests. Results looked Ok, but it seems I need to merge master and regenerate golden files again.

Test build #139947 has started for PR 28885 at commit c346387.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44474/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44474/

I updated the new rule to use pruned transforms and fixed a test suite because AQE is enabled by default now.
This PR is ready for review.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44474/

Test build #139950 has started for PR 28885 at commit 5b4b719.

Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44477/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44477/

Test build #139947 has finished for PR 28885 at commit c346387.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class TempResolvedColumn(child: Expression, nameParts: Seq[String]) extends UnaryExpression
case class MakeDTInterval(
case class StateStoreCustomSumMetric(name: String, desc: String) extends StateStoreCustomMetric
case class StateStoreCustomSizeMetric(name: String, desc: String) extends StateStoreCustomMetric
case class StateStoreCustomTimingMetric(name: String, desc: String) extends StateStoreCustomMetric

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139947/

Test build #139950 has finished for PR 28885 at commit 5b4b719.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139950/

I measured considerable improvement (~30%) using TPCDSQueryBenchmark on scaleFactor=5 data with the TPCDS Q14a, Q14b, Q23a, Q23b, Q47 and Q57 queries.

Did you see any regressions?

I measured considerable improvement (~30%) using TPCDSQueryBenchmark on scaleFactor=5 data with the TPCDS Q14a, Q14b, Q23a, Q23b, Q47 and Q57 queries.

Did you see any regressions?

Let me rerun the benchmarks again and come back to you.

Test build #139987 has started for PR 28885 at commit 7187ebd.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44514/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44514/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44514/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139987/

I measured considerable improvement (~30%) using TPCDSQueryBenchmark on scaleFactor=5 data with the TPCDS Q14a, Q14b, Q23a, Q23b, Q47 and Q57 queries.

Did you see any regressions?

Let me rerun the benchmarks again and come back to you.

The previous benchmark run was quite some time ago. As AQE has been enabled by default since that and works with most of those affected TPCDS queries the improvement is only visible if I disable AQE. But in that case it is still considerable: https://github.com/peter-toth/spark/blob/ca70def0793d35681c1c308d7410baaa8af0a559/sql/core/benchmarks/TPCDSQueryBenchmark-results.txt
I don't see any regression.

thanks, merging to master!

thanks, merging to master!

Thanks for the review!


Falsely detecting empty spacer columns

Empty columns can be used for styling, like

 with col defining a border or background-color. In this case W009's are detected even though the "empty" column can't be replaced.

Seems like you should typically be able to accomplish the same thing without spacer columns by moving some styles from the column to the row. What's your real-world example where spacer columns seem necessary?
Also, W009 is a Warning rather than an Error precisely because it may have false-positives.

For example http://codepen.io/stephtr/full/grOmQe/
A real-world example would be if you include colored blocks into something like http://itsashapechristmas.co.uk/.
I reported it mainly because the twbs-lmvtfy bot treats these warnings as errors, but this is intentionally (https://github.com/cvrebert/lmvtfy/issues/91).

@stephtr The links in your comment are broken. Markdown links can be written as either just the raw URL, or as [link text](http://your.url.goes/here/)

X-Ref: https://github.com/twbs/bootstrap/issues/11130

I corrected the links (http://codepen.io/stephtr/full/grOmQe/ and http://itsashapechristmas.co.uk/ ).

http://itsashapechristmas.co.uk/

I'm not getting any W009 errors on that page:

That website does not even use Bootstrap (it was intended to show where empty columns could be useful). I meant if someone would build a website like that with Bootstrap and insert colored spaces then there would be a W009.
My suggestion would be to check not only if there are col-..-.. classes mentioned in the html code and if the element has no content but also if there are no custom classes (or classes other than col-..-..) mentioned.
For example

 would be a W009 rather sure, but something like

 is propably a false-positive.

probably a false-positive.

Eh, it's an imperfect heuristic either way. What if we just tweaked the warning message text to clarify that it's uncertain?
As for the bot, I reopened https://github.com/cvrebert/lmvtfy/issues/91

Another option would be to ignore that border case and leave the warning message as it is (there probably are much more true-positives than false-positives).


upgrade to newer helm

Fixes #59 (hopefully). This should always quote the value of the appVersion field when packaging a chart. I cannot guarantee that ACR will preserve those quotes when it adds the packaged chart to its index, but 🤞 .

Codecov Report

Merging #60 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #60   +/-   ##
=======================================
  Coverage   40.53%   40.53%
=======================================
  Files          14       14
  Lines         977      977
=======================================
  Hits          396      396
  Misses        552      552
  Partials       29       29

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a3376b...ce3963c. Read the comment docs.


French language and Upgrade for nodebb 3.x

Hello @Skramewell
it could be interesting to maintain this plugin for the last version of nodeBB
Here file for french language (fr-FR) 👍
{
  "hide-message": "S'enregistrer ou se connecter pour voir le contenu caché]"
}

bye

Hello, the fr-FR was added, can you validate it?
Any contribution is welcome 🙂

In the next few days, I will update the plugin to the latest version of nodebb

Hello, the fr-FR was added, can you validate it? Any contribution is welcome 🙂

The capital S is just missing at the beginning of the sentence.
Like this :
{
  "hide-message": "[S'enregistrer ou se connecter pour voir le contenu caché]"
}

In the next few days, I will update the plugin to the latest version of nodebb

good news for the update
👍

fixed, thanks 🙂


Multiple Contemporary Hosts

Add the opportunity to connect multiple hosts at the same time

I guess you would like to define multiple docker endpoints and then switch between them.
This could be implemented but there would be a limitation: once you've changed docker endpoint the modification also applies to every other user connected to the same docker-compose-ui webapp.
This happens because the current implementation of docker-compose-ui to set a remote docker host is based on the DOCKER_HOST env, just like the docker-compose cli.

If I understood, you mention a sort of docker host bookmarks that are also manageable from the UI.
That would be enough for me.
I have also a question,  how are you interacting with docker-compose. I imagine you import it as a library then you wrap the API to for a programmatic usage.
Would be possible to integrate this behavior with Docker-Py and instantiate client "manually":
docker.DockerClient(base_url='host:port')
IDK if in this way you can keep the 100% compatibility with compose API though.

Hi @riccardotommasini I think you're right. I am actively investigating on how to implement this feature.

it should be quite easy to instantiate different clients to multiple docker hosts, my only concern is where to store the hosts list (since docker-compose-ui does not depend on a database).

Sounds naive, but I think you can keep them in memory as it is right now.

For persistence, how about a json format file in a directory that can be mounted on the host OS ?


Update configure local git

Update configure local git to more closely match https://github.com/MicrosoftDocs/azure-docs/blob/master/includes/app-service-web-configure-local-git-no-h.md.

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

@burkeholland : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rloutlaw - Please review these proposed changes. Thanks

reviewed and #sign-off on the proposed changes


MAINT: interpolate: remove duplicated FITPACK interface _fitpack.spl.

Reference issue

A part of https://github.com/scipy/scipy/issues/16729
Ref: https://github.com/scipy/scipy/pull/17038, https://github.com/scipy/scipy/pull/17091, #17303
What does this implement/fix?

This PR removes handwritten spl wrapper, use f2py wrapper instead.
Additional information

LGTM, thank you @AtsushiSakai !


How to get a 2D map contains both roads and buildings in Carla?

CARLA version: 0.9.14 build from source, Platform/OS: Ubuntu 20.04.
Hi guys, is it possible that I can get a 2D map in Carla that contains both the roads and top-down shapes of buildings? Moreover, it would be awesome if the map is vectorized.
Python API map.save_to_disk() is good, but there is no shapes of the buildings. Looking forward to awesome guidance!

Currently we do not have that feature implemented we can create a request as a feature, but it won't coming back soon. What you can do is to create a camsera sensor and put it into the sky looking downwards. Other thing you can do is create a custom sensor which does it for you with ortographic camera. but yes, we do not have this feature. As this is not an issue I will be closing it. Are you happy with that?

This would be a useful feature for presentations / situational awareness of scenarios, planning to go the downward facing camera approach described by Blyron.
It would be easier to implement agnostic to map if we could programmatically get the usable (i.e. maximum bounding box where any vehicle could spawn on a road or walker spawn on a sidewalk) dimensions of the map to automatically place said camera for any given town.


Add APIs for microsoft.resources to list deployments, also can add other operations like create, cancel and delete

MSDN documentation on those APIs
https://msdn.microsoft.com/en-us/library/azure/dn848363.aspx
https://msdn.microsoft.com/en-us/library/azure/dn790518.aspx

@stuartleeks this is the issue that tracks the work you wanted to help with.
@shobak101 can you point Stuart at the relevant code to do that?

@stuartleeks the place to put it is in operationscontroller in Get method, there are already 3 sources for the APIs we show HyakSpecs, jsonSpecs and specless APIs (HyakUtils.GetSpeclessCsmOperationsAsync()) which is the location to add those couple of APIs


Navbar Improved

Fixes #378
Please check now
Deployment link: https://community-site-eight.vercel.app/

Thank you @sau-mili


Fix Popper preventOverflow boundary config

Currently, the boundary config is being assigned to the wrong var (rootBoundary) in the popper config.  It should be assigned to the boundary var in popper's config.
Ref: https://popper.js.org/docs/v2/utils/detect-overflow/#boundary

@kyletsang do we have any issue about this? I don't suppose this issue could be related to any of the other Popper issues we have in v5?

@kyletsang do we have any issue about this? I don't suppose this issue could be related to any of the other Popper issues we have in v5?

@XhmikosR, currently no issue reported.  I was reviewing the code to see if there's any new modifiers I needed to port over to react-bootstrap when I stumbled on it.
This could potentially cause issues due to incorrect usage of rootBoundary.  Unfortunately, I haven't tested in-depth though.

@XhmikosR, currently no issue reported.  I was reviewing the code to see if there's any new modifiers I needed to port over to react-bootstrap when I stumbled on it.
This could potentially cause issues due to incorrect usage of rootBoundary.  Unfortunately, I haven't tested in-depth though.

@kyletsang do you think you could add some tests about this?

@kyletsang do you think you could add some tests about this?

Ugh, I think I'm going to need some assistance on this.  I can't seem to come up with a demo that shows any difference when setting an incorrect rootBoundary vs a correct one.

Ugh, I think I'm going to need some assistance on this.  I can't seem to come up with a demo that shows any difference when setting an incorrect rootBoundary vs a correct one.

Unfortunately, I have no idea how to test for this. We should have a test to make sure we don't hit this again, but if we can't have a test, I guess we could make an exception here.

@rohit2sharma95 LGTY?


feat: Improve Accessibility of the Searchbar Component

Feature Request
Ionic version:
[x] 4.x
Description
Improve the overall accessibility for the searchbar component. The searchbar currently isn't recognizable to screen readers and considered a 'generic component'. Also, buttons are lacking descriptive text and relying on icons to convey meaning. The search input is lacking the required label.
Preferred Solution
Add recommended role (ARIA search landmark) to the component, following WAI-ARIA authoring best practices. The search landmark identifies the search area within the page and facilitates keyboard navigation. Add aria-labels to buttons and search input. Hide icons from screen-readers if they are unnecessary and confusing.
NOTE: I intend to submit a PR for this. :)
Alternatives
I considered not suggesting that we hide the 'close' and 'arrow-back' icons when I saw they contained an aria-label element. But then recalled a situation where these were called into question. The 'close' icon used to reset the search form, in particular, left our accessibility testers confused. They even created an issue for our in-house developer team to try to change it. The presence of aria-label attributes on the ion-icons shows good intentions, but seem to more accurately describe the image itself rather than the functionality, in this case.
Repro Steps
I can reproduce the problem by forking the Ionic repo, running npm install, npm start, navigating to the searchbar component and opening the test/basic/index.html. From there, I can confirm these accessibility issues using some well-known developer tools and extensions. Here are the results, at a high-level:
Chrome Lighthouse  Accessibility Audit: 65%
WAVE web accessibility evaluation tool: 49 errors
axe for Chrome: 28 violations
Additional Context
Supporting Documentation & Guidelines:
https://www.w3.org/TR/wai-aria-practices-1.1/examples/landmarks/search.html
https://developers.google.com/web/tools/lighthouse/audits/button-name
https://dequeuniversity.com/rules/axe/3.3/button-name (note: user impact rating is labeled 'critical')
https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques/Using_the_aria-hidden_attribute
Related Issues
These show interest in the same feature, but appear to have had challenges during the Ionic Framework transition from v3 - v4 and never made it in:
https://github.com/ionic-team/ionic-v3/issues/393
https://github.com/ionic-team/ionic/issues/12048

Thank you for the issue and the PR! This will be added in the 4.7.0 release. Let me know if there is still something missing and I can reopen this. 🙂


Using aws-serverless-express

Hi,
I'm using aws-serverless-express for Lambda.
It up and runs node.js server (creating socket in the Docker container file system) proxying API Gateway request to it.
That works fine and I'm able to see logs out from this created server route.
But I cannot perform any async operations inside of this route, because aws-serverless-express  immediately gets Error: socket hang up
The Lambda timeout is set up to 20s
Inside of AWS environment everything works fine.

Won't work out of the box right now, AFAIK.
Following https://github.com/awslabs/aws-serverless-express/issues/137

This should be fixed as of v0.3.0. Feel free to re-open if it still doesn't work


Improvements to self-signed-ca.sh

/kind bug
What steps did you take and what happened:
Hi! I am trying to deploy kfserving 0.5 on minikube for testing (k8s 1.20, istio 1.6.14, knative 0.18.1 and go 1.15) and I am using the self-signed-ca.sh script instead of cert-manager. The script works fine, but I end up getting two separate errors when deploying a simple InferenceService resource:

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching - this one seems to be due to how the TLS certificate is created in https://github.com/kubeflow/kfserving/blob/master/hack/self-signed-ca.sh#L97, since from what I can see the .csr contains SANs but the .crt does not. I ended up fixing it adding -extensions v3_req to the cmd line, to force the .crt to pick up the SANs. After applying this fix the error went away, and I got the next one.
Error from server: error when creating "service.yaml": conversion webhook for serving.kubeflow.org/v1alpha2,  Kind=InferenceService failed: Post "https://kfserving-webhook-server-service.kfserving-system.svc:443/convert?timeout=30s":  x509: certificate signed by unknown authority - in this case it seems that the webhook responsible to convert APIs versions is not patched by the self-signed-ca.sh script, so its CABundle remains empty ending up in the above error. This happens if I use the v1alpha2 API, meanwhile if I use the v1beta1 everything works as expected.

Thanks in advance!
What did you expect to happen:
The script should run, patch the various webhooks with custom CA + TLS cert and it should be possible to deploy an InferenceService without errors.
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:

Istio Version: 1.6.14
Knative Version: 0.18.1
KFServing Version: 0.5
Kubeflow version: (kfserving standalone)
Minikube version: v1.19.0
Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

OS (e.g. from /etc/os-release): Debian 11 (Bullseye)

Maybe this PR can solve your issue

Maybe this PR can solve your issue， you can try the latest scripts in master branch

Hi @haijohn! The errors reported above are related to the master version of the script, I am aware of the patches to mutating and validating webhooks. What it is missing, in my opinion, is the patch to the conversion webhook that's missing (the caBundle doesn't get updated by the script). In my case, when a conversion between versions needed to happen, the unable to validate etc.. error popped up.

Maybe this PR can solve your issue， you can try the latest scripts in master branch

Hi @haijohn! The errors reported above are related to the master version of the script, I am aware of the patches to mutating and validating webhooks. What it is missing, in my opinion, is the patch to the conversion webhook that's missing (the caBundle doesn't get updated by the script). In my case, when a conversion between versions needed to happen, the unable to validate etc.. error popped up.

I see, so we also need to patch the conversion webhook caBundle in the inferenceservice CRD

Maybe this PR can solve your issue， you can try the latest scripts in master branch

Hi @haijohn! The errors reported above are related to the master version of the script, I am aware of the patches to mutating and validating webhooks. What it is missing, in my opinion, is the patch to the conversion webhook that's missing (the caBundle doesn't get updated by the script). In my case, when a conversion between versions needed to happen, the unable to validate etc.. error popped up.

I see, so we also need to patch the conversion webhook caBundle in the inferenceservice CRD

Yep this is my idea as well, but I didn't find a way to patch the CRD yet :D

the caBundle resides in this patch file: https://github.com/kubeflow/kfserving/blob/595186347aad89218c8e8866b77052f59de68417/config/default/inferenceservice_conversion_webhook.yaml#L3-L19
so, you can replace the caBundle with a patch as follows:
conversionPatchString='[{"op": "replace", "path": "/spec/conversion/webhook/clientConfig/caBundle", "value":"{{CA_BUNDLE}}"}]'
conversionPatchString=$(echo ${conversionPatchString} | sed "s|{{CA_BUNDLE}}|${caBundle}|g")
kubectl patch CustomResourceDefinition inferenceservices.serving.kubeflow.org \
    --type='json' -p="${conversionPatchString}"

@haijohn worked nicely, just tested it on minikube. I updated my pull request to include your fix (added you as co-author, lemme know if it is ok).

@haijohn worked nicely, just tested it on minikube. I updated my pull request to include your fix (added you as co-author, lemme know if it is ok).

Hi，@elukey, It's ok.  According to this https://github.com/kubeflow/kfserving/issues/1618 issue，v1alpha2 will be deprecated in the future release，so the conversion webhook may no longer needed or maybe there will be a v1/v1beta conversion.  To be future compatible,  I think some error check (caBundle not exists) may also needed.

@haijohn good point, but I think that we could simply report this note to the other issue and let future changes to lead the next step (should be easier to fix when it will be clear how the conversion webhook will be used in the future, maybe it will be needed for a lot of next releases). Let me know what you think, if you feel strongly about adding a check I'll work on it.

fixed in https://github.com/kubeflow/kfserving/pull/1661


Fixed "window -> swap" swapping panes instead of windows

Hi! I noticed that when selecting "window" and "swap" from the fzf menu something weird happens - it seemed like the pane got swapped randomly and the windows remained the same. Upon further investigation I realized it's a result of window -> swap triggering swap-pane instead of swap-window, so I fixed it.

That's my mistake, thanks!


dev(narugo): add upscale code and benchmark

Codecov Report

Merging #33 (5db1f0e) into main (e94f96e) will decrease coverage by 4.14%.
The diff coverage is 0.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff             @@
##             main      #33      +/-   ##
==========================================
- Coverage   98.72%   94.58%   -4.14%
==========================================
  Files          46       48       +2
  Lines        1257     1312      +55
==========================================
  Hits         1241     1241
- Misses         16       71      +55

Flag
Coverage Δ

unittests
94.58% <0.00%> (-4.14%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

imgutils/upscale/__init__.py
0.00% <0.00%> (ø)

imgutils/upscale/real_esrgan.py
0.00% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


PDA-325 Create common GHA for PDA services

Change List:

Breaking change on name of eks action to accommodate a second one
Creation of eks cache composite action; simply wraps github's action/cache@v3  - Why? We are using this in a few places so keeping track of the default path we're using everywhere is less than ideal

Looks good to me - are any other repos using these actions? I.e. is this breaking change only affecting the other actions in this repo or do other repos have to change?

There are other PDA repos using this for sure. It'll be a breaking change when the repos update to v0.11 or whatever we tag this one. I can go through these PDA repos and make the change as part of my current story.
I intend for there to be more changes to this branch before I merge, but wanted an early buy in that the strategy for our loaders (which is the easier set of actions) was looking okay. 😄

As long as it works and speeds things up I'm good with it :grin:

@dorner - I've gone through each of our go repos with these actions now. I had to make a couple more re-organization pieces based on the difference between our lambda and eks deploys. The go ones are in a good state - I think we can change the @PDA tag I'm using throughout to a v0 tag.
I suggest this instead of @main so that we have a clear way to introduce breaking changes (assuming other teams start using these actions). To stick with trunk based development with tags, we can use a strategy like Github is using in github-actions/checkout and just update the v0 tag each time we push a minor release. When we introduce a breaking change it can get tagged as a v1. Unfortunately it doesn't look like github-actions respects semantic versioning right now and just pulls the tag.

Thank you!
I temporarily tagged the latest commit on this branch as v0 so that I can go through and update all the PDA services to use the latest commit. I'll tag the version as v0.11 when we merge to main and then update the v0 tag to point main.


Create one headless service for one replica/podgroup

What would you like to be added:
One headless service should be created for each replica/podgroup
Why is this needed:
For large scale workload, there is a limit of the how many pods can be included in the single headless service. And actually we don't need communication for cross replicas
Completion requirements:
Potentially provide an API for configuring the headless service
This enhancement requires the following artifacts:

[x] Design doc
[x] API change
[x] Docs update

The artifacts should be linked in subsequent comments.

/assign
Would it be okay if I take this?
I'll write up a KEP first once I get a good understanding of the code.

sure, sgtm @kannon92

Does the leader get its own headless service also?

yes,  leaders will have headless service as well

So the goal of this proposal would be that each group of leader-worker would have their own headless service.
This is because we want leaders to community with workers but we don't need them to communicate with other replicas.
So we have as many headless services as we have replicas?
Sorry if this obvious. I'm trying to understand the difference between Replicas and size.


upgrade all stats of item

I understand that in your design, only one stat can be selected for each upgrade. If I want to upgrade all the stats by 5% of an item in one upgrade, how can I do it?

You mean as a shortcut so you don't have to do extra few clicks? I didn't really think about this to be honest, since I wanted to have more control over each rank upgrade (show percentage increases, show all requirements for the rank). Obviously bulk upgrade can be done, maybe could be a feature for future ;)

Added this functionality in [1ace74d2ed293ebafcf57e45a64cf3032445df22]https://github.com/silviu20092/azerothcore-item-upgrades/commit/1ace74d2ed293ebafcf57e45a64cf3032445df22


Support atrous conv in same padded convolution and add an error if use…

… transposed convolution with same or valid padding.
PiperOrigin-RevId: 283517237

Have changed the colab to essentially VALID padding (in the sense of if it was a normal convolution) so it illustrates the growth of the image. Have a few changes to go in after this but then will try and update the notebook to hopefully clarify exactly what the ops (including conv_transpose) implement.

The test failure seems unrelated and is likely fixed at head; can you rebase?

Have rebased and seems all fixed now
On Mon, Dec 9, 2019 at 10:00 AM James Bradbury<EMAIL_ADDRESS>wrote:

The test failure seems unrelated and is likely fixed at head; can you
rebase?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/google/jax/pull/1806?email_source=notifications&email_token=AD75SWN7CU4IX3X6YZ4FJCLQXYJLDA5CNFSM4JU2RTY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGIREBQ#issuecomment-563155462,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD75SWMOIKZSQEPT4S77FS3QXYJLDANCNFSM4JU2RTYQ
.


Initial ADC support for on STM32F1xx

Add an ADC implementation for F1 based chips. Primarily tested using ADC1, proper functionality for ADC2 probably needs some extra work as it's mainly a slave and can't e.g. measure vrefint by itself.
Needs https://github.com/embassy-rs/stm32-data/pull/115

bors r+

bors r+

bors cancel

bors cancel

bors r+

bors r+


Can i rename history_id field

I want use history model and change from default history_id field to id, like that
class ThreadHistory(models.Model):
    id = models.AutoField(primary_key=True)

    class Meta:
        abstract = True

class MyCustomThread(models.Model):
    ...fields...
    history = HistoricalRecords(
        excluded_fields=['id'],
        bases=[ThreadHistory, ],
    )

i get error after migrate: AssertionError: Model api.HistoricalThread can't have more than one AutoField. Can i rename from history_id field to id field

I just started using django-simple-history but looking at the documentation and the source code it's not possible to rename history_id field without bigger changes (like overriding get_extra_fields and get_meta_options). It's only possible to change the type of the field (to UUID for example). More details: custom history id and models.py.
Please correct me if I'm wrong.


Add addr cheatcode

Computes the address for a given private key.
address alice = cheats.addr(1);
emit log_address(alice); // 0x7e5f4552091a69125d5dfcb7b8c2659029395bdf

Is this relevant in starknet context?

It might be useful if we add support for testing signing.

Internal discussion: a user can import a function from the starknet


type of other class gives error :: Cannot read property 'name' of undefined

Hi,
I have issue in following model,

here , if I make language in schoolDetails of type LanguageLOV, I get error :
Cannot read property 'name' of undefined
TypeError: Cannot read property 'name' of undefined
at null.<anonymous> (C:\Users\arjun\AppData\Roaming\npm\node_modules\jhipster-uml\lib\editors\modelio_parser.js:258:96)
at Array.forEach (native)
at ModelioParser.fillConstraints (C:\Users\arjun\AppData\Roaming\npm\node_modules\jhipster-uml\lib\editors\modelio_parser.js:242:34)
at ModelioParser.parse (C:\Users\arjun\AppData\Roaming\npm\node_modules\jhipster-uml\lib\editors\modelio_parser.js:29:8)
at Object.<anonymous> (C:\Users\arjun\AppData\Roaming\npm\node_modules\jhipster-uml\jhipster-uml.js:71:27)
at Module._compile (module.js:435:26)
at Object.Module._extensions..js (module.js:442:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:311:12)
at Function.Module.runMain (module.js:467:10)

can you please help me debug this.
content of my XMI file is :
<?xml version="1.0" encoding="UTF-8"?><uml:Model     xmlns:uml="http://www.omg.org/spec/UML/20110701" xmlns:xmi="http://schema.omg.org/spec/XMI/2.1" xmi:version="2.1" xmi:id="_l00fQK9wEeWWpIRXoeM4Aw" name="iris2">

<Enter note text here>

First of, happy new year man!
Now for the ugly part of my answer: it may have worked with UML Designer, but it is still not correct.
If you're having a problem with Liquibase or the JHipster generator, I suggest you ask the fantastic JHipster team for help. I'll look into it later to see if there's an issue with JHipster-UML.

First of, happy new year man!
Now for the ugly part of my answer: it may have worked with UML Designer, but it is still not correct.
If you're having a problem with Liquibase or the JHipster generator, I suggest you ask the fantastic JHipster team for help. I'll look into it later to see if there's an issue with JHipster-UML.

thanks.! :)
and Happy new year to you too :) Its amazed to see that, U reply on the new
year's eve too :)
Regards,
Mayur
(c)323-873-2822
On Fri, Jan 1, 2016 at 1:00 AM, Mathieu ABOU-AICHI<EMAIL_ADDRESS>
wrote:

Reopened #100 https://github.com/jhipster/jhipster-uml/issues/100.
—
Reply to this email directly or view it on GitHub
https://github.com/jhipster/jhipster-uml/issues/100#event-503753093.

thanks.! :)
and Happy new year to you too :) Its amazed to see that, U reply on the new
year's eve too :)
Regards,
Mayur
(c)323-873-2822
On Fri, Jan 1, 2016 at 1:00 AM, Mathieu ABOU-AICHI<EMAIL_ADDRESS>
wrote:

Reopened #100 https://github.com/jhipster/jhipster-uml/issues/100.
—
Reply to this email directly or view it on GitHub
https://github.com/jhipster/jhipster-uml/issues/100#event-503753093.

Hi,
did exactly what you suggested, and it worked and generated all the code.
it created  following code  in SchoolDetails class. along with other member variables.
  @OneToOne
    private CountryLOV countryLOV;

but now, when I try to insert a a new school details.
It creates a insert statement, with column id as "countrylov_id"
where as the actual column that got created in DB is "country_lov_id", and hence the insert fails.
is it beacause of the nomenclature of the class , "CountryLOV," should it be "Countrylov" ?


spelling: favorites

This will require a release note and other bits, split off from https://dart-review.googlesource.com/c/sdk/+/276560/1..3/tools/dom/src/KeyName.dart#b44

CL: https://dart-review.googlesource.com/c/sdk/+/276773


The fn-ui won't work in the kubernetes deployment from the documentation

Enviroment Infos:
Kubernetes Version: v1.7.0
Minikube Version: v0.23.0
Issue Description:
I followed the instruction given in the kubernetes documentation and deployed the fnproject to my local minikube kubernetes cluster. I was able to use curl, from my host system, to generate an app and also set up the route. However, when I tried the same using the UI it failed [Error Message: Something went terribly wrong (Status Code: 0)]. I was able to visit the UI site, but at that point, my previously created app was not shown, further when I created a new app using the formula it failed. My investigation showed that, at least for my setup. The UI could not connect correctly to the fn-service.
Fix:
I went ahead and created and fork with the modification to fn-service.yaml.  Which can be viewed here. If it's fine by you, I will fill out the CLA and setup up an pull request after updating the documentation to make sure it reflects the namespace.
Best regards Simon
p.s. Thanks for the awesome work!

I have the same issue. Would be great if this fix finds his way into the master brunch.

Thanks for raising the issue. Apparently, the provided yaml needed some tweaking for the UI to work with minikube (at least with the DaemonSet service type). Probably something related to kube-dns, but didn't really dig into it. Submitting a quick fix (#499).
Note, we'll have a Helm chart coming soon, which should alleviate some of the madness around k8s configs.

@derekschultz thx :) I also could have submitted the pull request


ngAppRoot

A new directive to make it easier to run Angular in a subdirectory.
<html ng-app="app" ng-app-root="/myContext"></html>

@IgorMinar the components that should be affected are $location, $routeProvider, $resource, and $http. Have I missed any?

+1

+1

+1

+1

+1

+1

+1

+1

+1

I kind of agree with @taylorcode here: https://github.com/angular/angular.js/issues/3102#issuecomment-55693887.
@IgorMinar and @geddski can we reconsider splitting the configuration of "root" paths for the different components and configuring them in their respective providers instead of a directive?

I'm having issues with $http get/ post etc working, is there a fix for $http requests base href?
 doesn't work for this, nor does .htaccess (As I'd have to place the .htaccess in root folder)

+1


Bug 1918376: Allowing system:registry to list ICSP rules

Allows service account used by image-registry to access ICSP objects.

/retest

/retest

/retest

/retest

/assign @dmage

/lgtm

/retest

/override ci/prow/e2e-upgrade


Additional Keystone Strategies

When attempting to use this library with on a few Rackspace projects, I realized the provided strategy is really intended for a web app where the username and password are provided on a login screen and the result of authentication is stored in the session. While this works great for a web app with a UI, it doesn't translate to building an API where an auth token is provided and must be validated using the Keystone API. In order to allow this library to work well with APIs, I'm proposing the following changes to this library:

Rename Strategy to AuthenticationStrategy that corresponds to Token Authentication
Add support for API key-based authentication to AuthenticationStrategy that corresponds to Example 4.57.
Add TokenValidationStrategy that corresponds to the token validation call in the admin API.
Stop exporting a default strategy. Instead, use named exports to export each individual strategy.
Extract a common set of models for interacting with the result of authentication and validation.
Implement a Connect middleware for session expiration based on token expiration.

Additionally, we might want some additional changes in the future:

Support for Keystone v3.
Cache the results of token validation in a store like Redis or Memcached.

:: shakes dust off project ::
Thanks for the detailed issue! Originally just needed something to plug and play with an express web app. So many assumptions, glad you still checked this out and gave feedback.

took me awhile to realize what auth types were supported, then I clicked to expand the details for the anchor link. :+1:

2-4) :+1:

it's been awhile, but I'm really only familiar with service catalogs, anything else out there?

new project or include it here?

These items are a great proposal for a v1 release and in general a solid direction for this project. I added you as a collaborator if you've got some of these ideas to push to a branch. Also, wouldn't mind pushing this somewhere a bit more visible like github.com/rackerlabs. just give me a :+1: and I'll move it over.

it's been awhile, but I'm really only familiar with service catalogs, anything else out there?

It exposes the service catalog, the current user, and information about the token.

new project or include it here?

It probably won't be in my first pass of work anyway since we don't actually need it for my project, so we can defer the decision.

Hi
Are theses issues still open ?
I'm using a fork of this library, and bumped on the some of these problems, so I will develops the workaround soon.


Fix to find dynamic libraries

After #290, I looked again at CMake's code to find the location of dynamic libraries.
CMake finds libraries in a different way, but should result in the same final library to be dynamically loaded.
CMake does this in 2 steps:

Use find_package to find a dependency.
Use the library location from step 1 to create the dynamic library name:

I noticed that in install prefixes, dynamic links are used to point between files.
e.g. on my system, there are 2 symblinks to /usr/lib64/libjxl.so.0.6.1lrwxrwxrwx. 1 root root      13 20 jan 15:05 /usr/lib64/libjxl.so -> libjxl.so.0.6
lrwxrwxrwx. 1 root root      15 20 jan 15:05 /usr/lib64/libjxl.so.0.6 -> libjxl.so.0.6.1
-rwxr-xr-x. 1 root root 2916224 20 jan 15:06 /usr/lib64/libjxl.so.0.6.1

- CMake finds all links in the same folder that point to `libjxl.so.0.6.1`, sorts these by name and selects the first one.
  This excludes the dynamic libraries ending with `.so` and `.dylib`.
- This should always result in the one with the lowest string length to be chosen.

If the above fails, then simply use the library name from step 1.

This generally seems fine to me. Do you handle the case where these files all exist?
lrwxrwxrwx  1 Sam Lantinga None 14 Jul 13 07:23 libjxl.so -> libjxl.so.0.10
lrwxrwxrwx  1 Sam Lantinga None 16 Jul 12 11:50 libjxl.so.0.10 -> libjxl.so.0.10.1
-rw-rw-r--+ 1 Sam Lantinga None  0 Jul 12 11:50 libjxl.so.0.10.1
lrwxrwxrwx  1 Sam Lantinga None 15 Jul 12 08:40 libjxl.so.0.6 -> libjxl.so.0.6.1
-rw-rw-r--+ 1 Sam Lantinga None  0 Jul 12 08:39 libjxl.so.0.6.1
lrwxrwxrwx  1 Sam Lantinga None 15 Jul 12 10:18 libjxl.so.0.7 -> libjxl.so.0.7.1
-rw-rw-r--+ 1 Sam Lantinga None  0 Jul 12 10:18 libjxl.so.0.7.1

This generally seems fine to me. Do you handle the case where these files all exist?
lrwxrwxrwx  1 Sam Lantinga None 14 Jul 13 07:23 libjxl.so -> libjxl.so.0.10
lrwxrwxrwx  1 Sam Lantinga None 16 Jul 12 11:50 libjxl.so.0.10 -> libjxl.so.0.10.1
-rw-rw-r--+ 1 Sam Lantinga None  0 Jul 12 11:50 libjxl.so.0.10.1
lrwxrwxrwx  1 Sam Lantinga None 15 Jul 12 08:40 libjxl.so.0.6 -> libjxl.so.0.6.1
-rw-rw-r--+ 1 Sam Lantinga None  0 Jul 12 08:39 libjxl.so.0.6.1
lrwxrwxrwx  1 Sam Lantinga None 15 Jul 12 10:18 libjxl.so.0.7 -> libjxl.so.0.7.1
-rw-rw-r--+ 1 Sam Lantinga None  0 Jul 12 10:18 libjxl.so.0.7.1

In this case, our find_package will pick up libjxl.so.
The search algorithm will detect that libjxl.so.0.10 and libjxl.so.0.10.1 also point to the same file (or are the target file).
Since libjxl.so.0.10 is the shortest, that one will be picked.
The other libraries are not even considered.

Perfect. Can you apply the same fixes to SDL and SDL_mixer?


IssueFix22979

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Can you fill the PR template giving it a meaningful description?

Please just push more commits to one PR and leave it open, rather than closing PRs and opening new ones. It helps us stay on top of things :)


Incompatibility with graves mods - Back slot item drops on the ground instead of being stored in graves.

I've tested this with Universal Graves, Gravestones, and Coffins.
With all three mods, dying drops the back slot item onto the ground instead of storing it into the grave where it's safe.
Fabric Loader Version: 0.15.11
Trinkets Version: 3.7.2
BeansBackpacks Version: 2.0.2

Trinkets isn't needed for this same bug to happen, since elytras can be equipped to the backpack slot without trinkets.  Otherwise this still is broken in the same way as described above.


Update poutine aliases to be usable as decorators and context managers

This PR follows up on #1000 and replaces #1001 to make the poutine aliases more flexible and ergonomic.  Now all poutine aliases take only keyword arguments and return a Messenger if they aren't passed a callable as their first argument.  This enables use cases like
import pyro.poutine.decorators as poutine

@poutine.scale(scale=0.1)
def model(...)
    ...

with poutine.scale(scale=0.1):
   ...

x = poutine.scale(model, scale=0.1)(...)

This PR does not change any behavior outside of pyro.poutine, just changes call signatures to be compatible with the new interface.  In followup PR(s), I'll update all the documentation, rename files in poutine, and remove the old interface entirely.
Along with #1000, addresses #999.

Needs a few smoke tests

Needs a few smoke tests

I think it's fine to sprinkle some decorator syntax in a few tests, just to ensure we're using all the different syntax variations.

is something blocking this?

@eb8680 we're planning to release mid next week. let's try to merge this before release.

Should be good to merge after review


DEBUG investigate synthetic_tenant_size calculation

Don't merge!
This is just a brain dump of investigation process.
run test
cargo test tenant::size::investigation_test
debug output:
---- tenant::size::investigation_test stdout ----
insert_point: lastseg_id 0 , newseg_id 1
insert_point: lastseg Segment { parent: None, op: "", start_lsn: 0, end_lsn: 0, start_size: 0, end_size: Some(0), children_after: [1], needed: false } , newseg Segment { parent: Some(0), op: "", start_lsn: 0, end_lsn:<PHONE_NUMBER>76, start_size: 0, end_size: Some(220054675456), children_after: [], needed: false }
insert_point: lastseg_id 1 , newseg_id 2
insert_point: lastseg Segment { parent: Some(0), op: "", start_lsn: 0, end_lsn:<PHONE_NUMBER>76, start_size: 0, end_size: Some(220054675456), children_after: [2], needed: false } , newseg Segment { parent: Some(1), op: "", start_lsn:<PHONE_NUMBER>76, end_lsn:<PHONE_NUMBER>40, start_size:<PHONE_NUMBER>56, end_size: None, children_after: [], needed: false }
calculate loop
last_seg_id 0 last_seg: Segment { parent: None, op: "", start_lsn: 0, end_lsn: 0, start_size: 0, end_size: Some(0), children_after: [1], needed: false } retention_period 67108864
cutoff_lsn: 0
seg: Segment { parent: None, op: "", start_lsn: 0, end_lsn: 0, start_size: 0, end_size: Some(0), children_after: [1], needed: false }
seg.needed = true
breaking because no parent
calculate loop
last_seg_id 2 last_seg: Segment { parent: Some(1), op: "", start_lsn:<PHONE_NUMBER>76, end_lsn:<PHONE_NUMBER>40, start_size:<PHONE_NUMBER>56, end_size: None, children_after: [], needed: false } retention_period 67108864
cutoff_lsn:<PHONE_NUMBER>12
seg: Segment { parent: Some(1), op: "", start_lsn:<PHONE_NUMBER>76, end_lsn:<PHONE_NUMBER>40, start_size:<PHONE_NUMBER>56, end_size: None, children_after: [], needed: false }
seg.needed = true
prev_seg_id = 1
seg: Segment { parent: Some(0), op: "", start_lsn: 0, end_lsn:<PHONE_NUMBER>76, start_size: 0, end_size: Some(220054675456), children_after: [2], needed: false }
seg.needed = true
prev_seg_id = 0
seg: Segment { parent: None, op: "", start_lsn: 0, end_lsn: 0, start_size: 0, end_size: Some(0), children_after: [1], needed: true }
breaking because seg.end_lsn 0 < cutoff_lsn<PHONE_NUMBER>12
size_from_snapshot_later: seg0: 1 needed: true
size_from_wal: seg_id 1: this_size<PHONE_NUMBER>76 =<PHONE_NUMBER>76 - 0
size_from_wal: seg_id 2: this_size 67108864 =<PHONE_NUMBER>40 -<PHONE_NUMBER>76
size_from_snapshot_later: child.needed use p1 children loop child_id 1: size_from_wal(child_id = 1) =<PHONE_NUMBER>76
size_from_snapshot_later: WalNeeded seg_id 0: seg.start_size 0
method WalNeeded 0:t this_size=0 , ch=305614444640
calculated synthetic size:<PHONE_NUMBER>40
latest_gc_cutoff lsn 47/240A5860 is<PHONE_NUMBER>76, last_lsn lsn 47/280A5860 is<PHONE_NUMBER>40

human-readable interpretation:

We have some inputs.

calculate() function converts inputs to Storage->Segments and Storage->Branches.
Why do we even need it? We already have inputs in a nice sorted format...

we have 3 segments

seg 0 is trivial, Why do we even need it?
seg 1 represents history from 0 to latest_gc_cutoff. We don't care about this history, we only need to know
size at the end of the segment (this size is<PHONE_NUMBER>56) to use it as the start_size of the next segment.
seg 2 represents PITR window.

So far so good.

Let's compute the synthetic size.

What algorithm must do?
seg2.start_size + (seg2.end_lsn -seg2.start_lsn)
What it really does?
seg0.end_size + (seg1.end_lsn -seg1.start_lsn) + (seg2.end_lsn -seg2.start_lsn)

I agree it looks wrong

What algorithm must do?
seg2.start_size + (seg2.end_lsn -seg2.start_lsn)

Agreed.
Note: I think your assertion in the test is slightly wrong too. The result of the above is:
220054675456 +<PHONE_NUMBER>40 -<PHONE_NUMBER>40) =<PHONE_NUMBER>20

but the assertion checks against<PHONE_NUMBER>56
I'm pretty sure the code we have in 'main' has a bug, or multiple bugs. See my comment at https://github.com/neondatabase/neon/pull/2817#issuecomment-1411825811. I think we should move ahead with PR #2817, instead of spending much time debugging the current code in main. The code in PR #2817 is not great either, and might well contain bugs too, but it makes it much easier to verify what it's doing, and whether the calculation is correct or not.
I adapted your test case from here on top of PR #2817, it's now included there as a separate commit. I constructed the input document there by hand, copying the values from here, so I didn't check if the implementation would produce the same inputs. But given those inputs, the calculation there produces the correct result.

Note: I think your assertion in the test is slightly wrong too. The result of the above is:

Thanks for correction.
I'll take a look at #2817.