[SPARK-33308][SQL] Refactor current grouping analytics

What changes were proposed in this pull request?
As discussed in
https://github.com/apache/spark/pull/30145#discussion_r514728642
https://github.com/apache/spark/pull/30145#discussion_r514734648
We need to rewrite current Grouping Analytics grammar to support  as flexible as Postgres SQL to support subsequent development.
In  postgres sql, it support
select a, b, c, count(1) from t group by cube (a, b, c);
select a, b, c, count(1) from t group by cube(a, b, c);
select a, b, c, count(1) from t group by cube (a, b, c, (a, b), (a, b, c));
select a, b, c, count(1) from t group by rollup(a, b, c);
select a, b, c, count(1) from t group by rollup (a, b, c);
select a, b, c, count(1) from t group by rollup (a, b, c, (a, b), (a, b, c));

In this pr,  we have done three things as below, and we will split it to different pr:

Refactor CUBE/ROLLUP (regarding them as ANTLR tokens in a parser)
Refactor GROUPING SETS (the logical node -> a new expr)
Support new syntax for CUBE/ROLLUP (e.g., GROUP BY CUBE ((a, b), (a, c)))

Why are the changes needed?
Rewrite current Grouping Analytics grammar to support  as flexible as Postgres SQL to support subsequent development.
Does this PR introduce any user-facing change?
User can  write Grouping Analytics grammar as flexible as Postgres SQL to support subsequent development.
How was this patch tested?
Added UT

Test build #130484 has started for PR 30212 at commit 3779583.

FYI @maropu @cloud-fan

Test build #130486 has started for PR 30212 at commit 26832f3.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35088/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35090/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35088/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35088/
Test FAILed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35090/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35090/
Test FAILed.

Test build #130487 has started for PR 30212 at commit e5bfa3f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35091/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35091/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35091/
Test FAILed.

Test build #130484 has finished for PR 30212 at commit 3779583.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #130487 has finished for PR 30212 at commit e5bfa3f.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Test build #130486 has finished for PR 30212 at commit 26832f3.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130487/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130486/
Test FAILed.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130484/
Test FAILed.

retest. this please

Test build #130488 has started for PR 30212 at commit e5bfa3f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35092/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35092/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35092/
Test PASSed.

Test build #130488 has finished for PR 30212 at commit e5bfa3f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130488/
Test FAILed.

also ping @viirya

Test build #130494 has started for PR 30212 at commit 7bce0e1.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35098/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35098/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35098/
Test FAILed.

Test build #130494 has finished for PR 30212 at commit 7bce0e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130494/
Test PASSed.

Any suggestion?

gentle ping @maropu  @cloud-fan Do you have any suggestion for this one.  A series of PR changes need to be made to accommodate this support

Test build #130724 has started for PR 30212 at commit ee67839.

Test build #130725 has started for PR 30212 at commit 153cd06.

Test build #130725 has finished for PR 30212 at commit 153cd06.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130725/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35333/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35334/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35333/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35333/
Test FAILed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35334/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35334/
Test PASSed.

Test build #130724 has finished for PR 30212 at commit ee67839.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130724/
Test FAILed.

Test build #130727 has started for PR 30212 at commit d9b9fdd.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35336/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35336/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35336/
Test PASSed.

Test build #130733 has started for PR 30212 at commit d0327ad.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35342/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35342/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35342/
Test PASSed.

Test build #130727 has finished for PR 30212 at commit d9b9fdd.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
                s\"its class is $

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130727/
Test PASSed.

Test build #130733 has finished for PR 30212 at commit d0327ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class GroupingSetsV2(

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130733/
Test PASSed.

@maropu  With current code, I have test that we can replace origin GroupingSets and pass all test.

Any update?

Test build #131666 has started for PR 30212 at commit 74a22f8.

Test build #131666 has finished for PR 30212 at commit 74a22f8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
 public static class NoOpMergedShuffleFileManager implements MergedShuffleFileManager
public class RemoteBlockPushResolver implements MergedShuffleFileManager
 static class PushBlockStreamCallback implements StreamCallbackWithID
 public static class AppShuffleId
 public static class AppShufflePartitionInfo
class ExecutorSource(
  case class GetShufflePushMergerLocations(numMergersNeeded: Int, hostsToFilter: Set[String])
  case class RemoveShufflePushMergerLocation(host: String) extends ToBlockManagerMaster
trait HasMaxBlockSizeInMB extends Params
    >>> class VectorAccumulatorParam(AccumulatorParam):
            fully qualified classname of key Writable class (e.g. \"org.apache.hadoop.io.Text\")
            fully qualified classname of key Writable class (e.g. \"org.apache.hadoop.io.Text\")
            fully qualified classname of key Writable class (e.g. \"org.apache.hadoop.io.Text\")
            fully qualified classname of key Writable class (e.g. \"org.apache.hadoop.io.Text\")
class HasMaxBlockSizeInMB(Params):
trait SQLConfHelper
class Analyzer(override val catalogManager: CatalogManager)
case class UnresolvedTable(
case class UnresolvedTableOrView(
case class UnresolvedPartitionSpec(
case class ResolvedPartitionSpec(
class SubExprEvaluationRuntime(cacheMaxEntries: Int)
case class ExpressionProxy(
case class ResultProxy(result: Any)
case class ElementAt(
case class GetArrayItem(
case class GetMapValue(
case class CurrentTimeZone() extends LeafExpression with Unevaluable
abstract class LikeAllBase extends UnaryExpression with ImplicitCastInputTypes with NullIntolerant
case class LikeAll(child: Expression, patterns: Seq[UTF8String]) extends LikeAllBase
case class NotLikeAll(child: Expression, patterns: Seq[UTF8String]) extends LikeAllBase
case class Elt(
case class ParseUrl(children: Seq[Expression], failOnError: Boolean = SQLConf.get.ansiEnabled)
trait OffsetWindowFunction extends WindowFunction
class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logging
abstract class AbstractSqlParser extends ParserInterface with SQLConfHelper with Logging
class CatalystSqlParser extends AbstractSqlParser
case class AnalyzeTable(
case class AnalyzeColumn(
case class AlterTableAddPartition(
case class AlterTableDropPartition(
case class LoadData(
case class ShowCreateTable(child: LogicalPlan, asSerde: Boolean = false) extends Command
case class TruncateTable(
abstract class Rule[TreeType <: TreeNode[_]] extends SQLConfHelper with Logging
 implicit class MetadataColumnsHelper(metadata: Array[MetadataColumn])
 implicit class PartitionSpecsHelper(partSpecs: Seq[PartitionSpec])
class SparkPlanner(val session: SparkSession, val experimentalMethods: ExperimentalMethods)
class SparkSqlParser extends AbstractSqlParser
class SparkSqlAstBuilder extends AstBuilder
case class CoalesceShufflePartitions(session: SparkSession) extends Rule[SparkPlan]
class FindDataSourceTable(sparkSession: SparkSession) extends Rule[LogicalPlan]
class FallBackFileSourceV2(sparkSession: SparkSession) extends Rule[LogicalPlan]
trait PathFilterStrategy extends Serializable
trait StrategyBuilder
class PathGlobFilter(filePatten: String) extends PathFilterStrategy
abstract class ModifiedDateFilter extends PathFilterStrategy
class ModifiedBeforeFilter(thresholdTime: Long, val timeZoneId: String)
class ModifiedAfterFilter(thresholdTime: Long, val timeZoneId: String)
class ResolveSQLOnFile(sparkSession: SparkSession) extends Rule[LogicalPlan]
case class PreprocessTableCreation(sparkSession: SparkSession) extends Rule[LogicalPlan]
case class AlterTableAddPartitionExec(
case class AlterTableDropPartitionExec(
case class DropTableExec(
class V2SessionCatalog(catalog: SessionCatalog)
case class PlanDynamicPruningFilters(sparkSession: SparkSession)
  class HDFSBackedReadStateStore(val version: Long, map: MapType)
trait ReadStateStore
trait StateStore extends ReadStateStore
class WrappedReadStateStore(store: StateStore) extends ReadStateStore
abstract class BaseStateStoreRDD[T: ClassTag, U: ClassTag](
class ReadStateStoreRDD[T: ClassTag, U: ClassTag](
case class PlanSubqueries(sparkSession: SparkSession) extends Rule[SparkPlan]
class VariableSubstitution extends SQLConfHelper
abstract class JdbcDialect extends Serializable with Logging
class ResolveHiveSerdeTable(session: SparkSession) extends Rule[LogicalPlan]
class DetermineTableStats(session: SparkSession) extends Rule[LogicalPlan]

Build finished.

Test build #131677 has started for PR 30212 at commit 4b943eb.

Test build #131677 has finished for PR 30212 at commit 4b943eb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class GroupingSets(

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131677/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36267/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36277/

Test build #131730 has started for PR 30212 at commit 516afc5.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36328/

Test build #131730 has finished for PR 30212 at commit 516afc5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131730/

In postgres sql, it support

select a, b, c, count(1) from t group by a, b, cube(a, b, c);
select a, b, c, count(1) from t group by a, c, grouping sets (a, b, (a, b));

Could you remove the two PostgreSQL examples above in the PR description? I think the examples is out-of-scope of this PR and they make reviewers confused.

In postgres sql, it support

select a, b, c, count(1) from t group by a, b, cube(a, b, c);
select a, b, c, count(1) from t group by a, c, grouping sets (a, b, (a, b));

Could you remove the two PostgreSQL examples above in the PR description? I think the examples is out-of-scope of this PR and they make reviewers confused.

Yea, that is another problem.

Test build #131870 has started for PR 30212 at commit fad63db.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36467/

Test build #131882 has started for PR 30212 at commit da9480c.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36479/

Test build #131870 has finished for PR 30212 at commit fad63db.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131870/

Test build #131882 has finished for PR 30212 at commit da9480c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131882/

Test build #131924 has started for PR 30212 at commit 7e267b8.

Test build #131924 has finished for PR 30212 at commit 7e267b8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131924/

Test build #131927 has started for PR 30212 at commit fa21d9f.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36521/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36524/

Test build #131927 has finished for PR 30212 at commit fa21d9f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/131927/

Test build #132765 has started for PR 30212 at commit 40fae77.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37367/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37367/

Test build #132765 has finished for PR 30212 at commit 40fae77.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132765/

Test build #132772 has started for PR 30212 at commit 0df439c.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37374/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37374/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37374/

Test build #132772 has finished for PR 30212 at commit 0df439c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132772/

retest this please

Test build #132799 has started for PR 30212 at commit 0df439c.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37402/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37402/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37402/

Test build #132799 has finished for PR 30212 at commit 0df439c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132799/

gentle ping @maropu

Could you resolve the conflict first?

Test build #133046 has started for PR 30212 at commit 41c67a2.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37646/

Test build #133046 has finished for PR 30212 at commit 41c67a2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class SparkPod(pod: Pod, container: Container)
trait KubernetesFeatureConfigStep
public class Distributions
case class AlterTableRecoverPartitions(child: LogicalPlan) extends Command
case class AlterViewAs(
case class AlterTableSerDeProperties(
case class UncacheTable(
case class SubqueryExec(name: String, child: SparkPlan, maxNumRows: Option[Int] = None)
case class UncacheTableExec(

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133046/

retest this please

Could you resolve the conflict first?

Updated

Test build #133058 has started for PR 30212 at commit 41c67a2.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37658/

Could you check my PR for your branch? https://github.com/AngersZhuuuu/spark/pull/6

Could you check my PR for your branch? AngersZhuuuu#6

Use half-life objects to provide equivalent constructors and return resolved expression when call  method.
That's a great idea.

retest this please

Test build #133065 has started for PR 30212 at commit 53b3714.

Could you review this? @cloud-fan @viirya

Test build #133058 has finished for PR 30212 at commit 41c67a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class SparkPod(pod: Pod, container: Container)
trait KubernetesFeatureConfigStep
public class Distributions
case class AlterTableRecoverPartitions(child: LogicalPlan) extends Command
case class AlterViewAs(
case class AlterTableSerDeProperties(
case class UncacheTable(
case class SubqueryExec(name: String, child: SparkPlan, maxNumRows: Option[Int] = None)
case class UncacheTableExec(

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37665/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133058/

Test build #133069 has started for PR 30212 at commit f89bc42.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37669/

Test build #133069 has finished for PR 30212 at commit f89bc42.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133069/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37665/

Test build #133065 has finished for PR 30212 at commit 53b3714.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class Cube(groupingSetIndexes: Seq[Seq[Int]], children: Seq[Expression]) extends GroupingSet
case class Rollup(

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133065/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37665/

Test build #133073 has started for PR 30212 at commit 04ff77d.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37673/

Test build #133073 has finished for PR 30212 at commit 04ff77d.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133073/

retest this please

Test build #133102 has started for PR 30212 at commit 04ff77d.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37702/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37702/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37702/

Test build #133102 has finished for PR 30212 at commit 04ff77d.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133102/

Test build #133213 has started for PR 30212 at commit d2d8195.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37810/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37810/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37810/

Test build #133213 has finished for PR 30212 at commit d2d8195.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133213/

Test build #133472 has started for PR 30212 at commit 927f21e.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38061/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38061/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38061/

Test build #133472 has finished for PR 30212 at commit 927f21e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133472/

gentle ping @cloud-fan @HyukjinKwon @dongjoon-hyun @viirya @wangyum

Test build #133693 has finished for PR 30212 at commit 927f21e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #133693 has finished for PR 30212 at commit 927f21e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133693/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133693/

retest this please

retest this please

Test build #133750 has started for PR 30212 at commit 927f21e.

Test build #133750 has started for PR 30212 at commit 927f21e.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38338/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38338/

Test build #133750 has finished for PR 30212 at commit 927f21e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #133750 has finished for PR 30212 at commit 927f21e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133750/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133750/

gentle ping @cloud-fan @viirya @HyukjinKwon

gentle ping @cloud-fan @viirya @HyukjinKwon

Can anyone take a look?

Can anyone take a look?

Test build #134435 has started for PR 30212 at commit 646d6a2.

Test build #134435 has started for PR 30212 at commit 646d6a2.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39021/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39021/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39021/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39021/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39021/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39021/

Test build #134435 has finished for PR 30212 at commit 646d6a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test build #134435 has finished for PR 30212 at commit 646d6a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134435/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134435/

retest this please

Test build #135212 has started for PR 30212 at commit 646d6a2.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39793/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39793/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39793/

Test build #135212 has finished for PR 30212 at commit 646d6a2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135212/

@AngersZhuuuu Could you fix the conflict? I think we have much time until the next release and we need to wait more for reviews. The direction itself in this PR looks reasonable, I think.

@AngersZhuuuu Could you fix the conflict? I think we have much time until the next release and we need to wait more for reviews. The direction itself in this PR looks reasonable, I think.

Sure, updated.

Test build #135704 has started for PR 30212 at commit c45184d.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40286/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40286/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40286/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135704/

I wait another 1-2 weeks, then I'll will merge this if no one has more comments.

Can we update sql-ref-syntax-qry-select-groupby.md? It's not clear to me what's the final GROUP BY syntax after this PR.

Test build #136499 has started for PR 30212 at commit 8ac097a.

Test build #136503 has started for PR 30212 at commit 080ac6d.

Test build #136505 has started for PR 30212 at commit c60ece0.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41083/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41083/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41083/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41088/

Test build #136512 has started for PR 30212 at commit 2b7f645.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41088/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41088/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41090/

Test build #136499 has finished for PR 30212 at commit 8ac097a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41090/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136499/


refactor: ♻ rewrite part of directives in small mini libraries

See Organizing a package.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1074

Relevant Lines:
1074

💛  - Coveralls


UI Scaling Controls

I'm very much in love with how HunterPie's widgets look aesthetically, but as far as I can tell there's no configurability for the UI scale of individual widgets or their subelements--this can make some things a little hard to read at 3840x2160. Is it feasible to add controls for this?

You can rescale the widgets by pressing ScrollLock (or whatever HotKey you set for Design Mode), hovering your mouse over the widget and then scrolling up to scale the widget up and scroll down to scale the widget down, then press your design mode HotKey again to save the changes.

My apologies, I somehow missed that when reading through the documentation--works great. Closing issue.


Axelarscan doesn't show dollar value on some ITS assets

hey <@U035SH6D4CQ> just wondering why some of the assets on axelarscan dont have an associated dollar value. lmk if its anything i can help chase down
Slack Message

There's no coingecko-id for these assets in the shared assets config, so the service can't get token price to calculate dollar values. Once the file is updated, the dollar value should be displayed.


OAI-PMH cannot disseminate format when  declaration is present in payload.

If there is a  declaration (or any other declaration other than  declarations present in an XML payload, a cannot disseminate format error is thrown because the lxml parser is unable to parse strings containing any XML or DOCTYPE declaration.
The error is acceptable from a service perspective, however since the data contained within payload is technically XML we should try and disseminate it as it should be trivial.

@wegrata said this was fixed indirectly in a previous pull request and passes associated tests with the code on alpha.  Therefore recommend the ticket be closed.


How to config GPU options?

I would like to config the gpu options as
gpu_options['allow_growth'] = True
in gpflowrc under [session],  it seems not working

At the moment you can't. You have to edit the file session_manager.py and add gpu_options.allow_growth to the ConfigProto.
You could also hack it and add the recursive option to the settings before doing anything else with GPflow:
import gpflow
settings_obj = gpflow.settings.get_settings()
settings_obj.session.gpu_options = gpflow._settings._MutableNamedTuple()
settings_obj.session.gpu_options.allow_growth = True
gpflow.settings.push(settings_obj)

Hello @HeroKillerEver,
This is not a bug or performance issue, and probably you've created the issue via How-to and other issues option, which has this description: Please, post other questions on StackOverflow with gpflow tag. Could you please next time create a StackOverflow question instead? Thanks!
And, you can tweak GPU parameters using:

import gpflow
import tensorflow as tf

config = tf.ConfigProto(gpu_options=tf.GPUOptions(allow_growth=True))
gpflow.reset_default_session(config=config)

... your gpflow code

import gpflow
import tensorflow as tf

config = tf.ConfigProto(gpu_options=tf.GPUOptions(allow_growth=True))
with tf.Session(config=config).as_default():
    ... your gpflow code


Client-side usage with npm

When using Prism with npm's package manager, I manage to correctly import Prism and get it working on basic languages.
However, prism.css is not imported in the project by default, so there is no syntax coloring, despite the tokens being correctly populated in the DOM.
Also, I haven't found a way to make more "exotic" languages work as well. For instance, I have a <code class="language-handlebars"> element on a page, which is not tokenized by Prism.
I guess Prism npm package have options to import css and specific languages. I'm just wondering how to use them.

Here is the Prism reference in my package.json
{
    "dependencies": {
        "prismjs": "^1.5.1"
    }
}

And the way I'm trying to use it in my code
import Prism from 'prismjs'
Prism.highlightAll();

This produces the result explained above:

Tokenizing works for basic languages (html, javascript...)
Tokenizing does not work for other specific languages (lua, handlebars...)
For all languages, syntax coloring isn't applied (css file doesn't seem loaded)


A few issues

Ruolin
That is useful with the adding the gene_id from the ensembl annotation, much faster than my python script to do it afterwards. I'll let you know how this pans out with the downstream tools I use. There are a few other minor issues though that would help make the tool better IMO:

What would likely be a further improvement is adding the gene_name as well, this is an example from Stringtie of the field we are discussing:

gene_id "ERR188044.1"; transcript_id "ERR188044.1.1"; reference_id "NM_018390"; ref_gene_id "NM_018390"; ref_gene_name "PLCXD1"
So we have your tool's gene_id and transcript_id, plus the ref_gene_name and ref_gene_id from ensembl/ reference annotation.

If I am running the --no-quant flag, would it be possible to remove the TPM and FPKM parts from the annotation output file, they should not really be there, minor issue. We get these currently:

;FPKM "NA";Frac "NA";TPM "NA";

Usually I run stringtie in 1 directory and it outputs its .gtf files into the same directory, each named according to the original file ID. This is easier than having them all in different directories, then having to rename and move them all afterwards, before I run cuffmerge. I actually use taco (https://www.nature.com/articles/nmeth.4078) instead of cuffmerge, it is supposed to be a lot better, and my results made more sense when using this.

I don't really want any log files to be outputted, I only need the .gtf file. I have to include extra code to clean all this up. Is it possible to have a parameter to get a single .gtf and nothing else?

Thanks for your time. These are just some ideas for you to review, I am keen on using strawberry in our work.
Chris

@crj32 Thank you for your suggestions. I really like those people giving feedbacks. Please don't hesitate to do so. And for 3 and 4, I think what you need is an option for output_gtf but not an output folder, right? I will start to cracking these issues. It will probably take a few days since I am also busy with other stuff.

Yeah, just a custom named output .gtf would be nice. Right you are.

What if when no annotation, do you prefer something like ; ref_gene_id ""; ref_gene_name ""?

I don't mind because we are always using a reference, but I'd suggest making the names for these IDs the same as Stringtie uses incase any other tools require specific names for compatibility.
I think without a reference, it is probably just going to be 'gene_id' and 'transcript_id'? I think you had that already.

I've just noticed something, gene_id "ERR188044.1"; transcript_id "ERR188044.1.1", from Stringtie include the sample name in the syntax. You may have recognised this already.
This is important as when it comes to merging the gtf's, as I have an error with strawberry using Taco to merge the .gtf files, because the .gtf format is not the same as Stringtie (https://tacorna.github.io/). This is an issue for me, and I expect other people, potentially.

@crj32 I think I have completed all your suggested items. Can you have a try of the current master, commit id 09aee96 to see if that has met your needs?

My VM I do my work on is down today unfortunately so we will have to wait and is it OK to get a precompiled binary instead of the source code?

So my computer is working again, I have compiled from source and will test the new version for you. Thanks.


ask

hello im new with polymer and i try to use your element, but the pdf preview not working but when i dowload the pdf is working, this my code:

Hi @tontowi22, I'll have a look this weekend.
By the way, have you checked the js console (in developer tools of Chrome/Firefox). Is there any errors?

Reopen this issue if the problem still exists.


TraitBase isImbued and isImbuedTo

Description
Closes #815
For symmetry with imbue and imbueTo methods, rename the isValid method to isImbued.
For C++ in particular, where performance is likely a concern, constructing a TraitBase-derived instance, just to check isImbued, is wasteful. So add a static isImbuedTo (for symmetry with imbueTo), which tests a TraitsData without unnecessary temporary
construction.

[x] I have updated the release notes.
[x] I have updated all relevant user documentation.

Just a sneaky one that slipped the net:
https://github.com/OpenAssetIO/OpenAssetIO/blob/a3aeeaec4649171dbbb4a31422f354cc02c614e6/src/openassetio-python/package/openassetio/TraitBase.py#L49

Fixed in 28d23035


Fix sample projects build break due to new SDK

All sample projects are currently not building due to new SDK changes. All project files need to be edited to contain appropriate property values.

What new SDK changes?  Someone looking at the history might follow the PR to this issue if they are seeing problems with their own projects.

https://github.com/dotnet/wpf/pull/868

Yeah I just wanted something encapsulated here so someone looking at the PR changes would understand why.


Add a configuration file

Read settings for chromatic aberration and sharpening from there.

Done in 1.0.4


Windows OS scan is ridiculously slow

Directories such as AppData are slowing things down evident from the github action workflow called win-tests. Perhaps, cdxgen should accept a file with location glob patterns instead of scanning all of AppData?

This is improved with 5.2.1. Fixed a bug with directory expansion code that created numerous duplicates in the path list. Also moved the filesystem parsing behind a new --deep option.


Feature: add Http.attempt operator

Codecov Report

Merging #1070 (75bc1bb) into main (84d5799) will decrease coverage by 0.07%.
The diff coverage is 57.14%.

@@            Coverage Diff             @@
##             main    #1070      +/-   ##
==========================================
- Coverage   56.83%   56.76%   -0.08%
==========================================
  Files          73       73
  Lines        2259     2262       +3
  Branches       57       63       +6
==========================================
  Hits         1284     1284
- Misses        975      978       +3

Impacted Files
Coverage Δ

zio-http/src/main/scala/zhttp/http/Http.scala
72.38% <57.14%> (-1.66%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 84d5799...75bc1bb. Read the comment docs.


gstreamer.gst-plugins-bad: Patch openjpeg version

Motivation for this change
gstreamer.gst-plugins-bad is broken because the source contains a hard-coded path to openjpeg.
This broke in https://github.com/NixOS/nixpkgs/commit/4e5725605b321805a10466238eff4d51a6c4315d which bumped the openjpeg version from 2.1 to 2.3.
cc @fpletz @7c6f434c
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Thanks!


class inheritance from Bot

Hello,
Thanks for developing such easy-use project.
When I wrote some tests with a class inherit from Bot, the infinite inheritance occured as #170 described.
Then with locally modification of super() which suggested in #170, it could runs normally in my python3 environment.
Thus, here is the PR for reference.

Thank you very much for your work!


fix typo

after all relevant phaser

phasers?
Perhaps you can add one more commit :)

changed back to phasers...


Add ability to stream command output as a response

Similar to the #251, with only one option for streaming and based on the current version of webhook (2.7.0)
Has a mysterious issue where response gets delayed for a few seconds when opening in a browser but starts immediately when opening with curl/wget.
Reference implementation https://play.golang.org/p/PpbPyXbtEs doesn't have this issue. Does anyone have an idea why is that happening?

@milolav, please edit this PR to be against the development branch, not master.  Also, the Travis CI builds are failed due to a bug in our Travis config.  We should get fixed in development soon with #445, and then you can rebase.

@moorereason I missed the argument in the test script. Fixed that and switched PR to be against the development branch

@milolav, I still haven't taken a good look at this PR, but you should be able to rebase against development to fix the Travis CI builds.

@moorereason rebased my master branch against the current development branch and merged the change for this PR.


libcore => 2018 was merged

cc https://github.com/rust-lang/rust/pull/58702#issuecomment-483572036

Thanks for the heads up :)


[UWP] Add rendering support for tooltip property

Related Issue
Fixes #5470
Description
Implemented UWP support for tooltips. Added logic to handle correctly setting the title and tooltip properties to the UI tooltip and automation name and description. Refactored automation name/description setting and tooltip creation for easier code sharing.
This PR does not fix related issue #5579. This means that container contents of a container are still not read in the case where the title is not present.
Sample Card
Added new card at samples\v1.5\Test\TooltipTestCard.json
How Verified
Using the above test card:

Validated tooltips display correctly
Validated Name and FullDescription properties were set properly in accessibility insights
Validated via Narrator that title and tooltip properties were narrated as expected.

Microsoft Reviewers: Open in CodeFlow

Thanks @jwoo-msft!


Accessibility: Pause Button

This is a (multiple allowed):

[ ] bug
[x] enhancement
[ ] feature-discussion (RFC)

If you want to have an accessibility certification for your website you'll have to optimize your website in several ways. Regarding Swiper that would be translation, WAI-ARAI attributes e.g. for buttons (both already fulfilled) but also a pause button when autoplay is set to true. Currently no pause button is generated, for but accessibility reasons it's required and you won't receive a certification otherwise. Please implement it.

If you want to have an accessibility certification for your website

I don’t want such certificate for my website

But you're not only maintining the library for yourself? There are many people with disabilities out there that would really appreciate this feature.

Then you have to correctly write an issue.


Merge develop commits into Feature/sr_compose

What does this PR do?
Due to some conflicts of merging, auto-merge from develop->feature/sr_compose is not possible.
So I created this dedicated branch from Feature/sr_compose and merge develop by solving the conflicts.
Review checklist (to be filled by reviewers)

[ ] Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
[ ] Make sure you discussed the feature or bugfix with the maintaining team in an Issue
[ ] Make sure each commit and the PR mention the Issue number (cf the CONTRIBUTING doc)

Codecov Report
Attention: Patch coverage is 87.26953% with 145 lines in your changes missing coverage. Please review.

Project coverage is 69.71%. Comparing base (eea96af) to head (422a9d4).

Files with missing lines
Patch %
Lines

...l/recorder/mapper/CheckableCompoundButtonMapper.kt
36.17%
23 Missing and 7 partials :warning:

...ernal/persistence/datastore/DataStoreFileHelper.kt
15.00%
17 Missing :warning:

...um/internal/metric/SessionEndedMetricDispatcher.kt
67.74%
4 Missing and 6 partials :warning:

.../android/core/internal/data/upload/UploadStatus.kt
86.96%
4 Missing and 5 partials :warning:

...ernal/persistence/datastore/DatastoreFileWriter.kt
84.75%
2 Missing and 7 partials :warning:

...lay/internal/recorder/mapper/SwitchCompatMapper.kt
90.00%
2 Missing and 6 partials :warning:

...d/rum/internal/domain/scope/RumViewManagerScope.kt
68.18%
6 Missing and 1 partial :warning:

.../android/rum/internal/metric/SessionEndedMetric.kt
93.55%
0 Missing and 6 partials :warning:

.../internal/recorder/mapper/CheckedTextViewMapper.kt
55.56%
1 Missing and 3 partials :warning:

.../android/api/storage/datastore/DataStoreHandler.kt
40.00%
3 Missing :warning:

... and 24 more

Additional details and impacted files
@@                   Coverage Diff                   @@
##           feature/sr_compose    #2241       +/-   ##
=======================================================
- Coverage               82.30%   69.71%   -12.59%
=======================================================
  Files                     514      746      +232
  Lines                   18358    27647     +9289
  Branches                 2821     4696     +1875
=======================================================
+ Hits                    15108    19273     +4165
- Misses                   2488     7065     +4577
- Partials                  762     1309      +547

Files with missing lines
Coverage Δ

...n/kotlin/com/datadog/android/api/InternalLogger.kt
100.00% <ø> (ø)

...in/com/datadog/android/api/feature/FeatureScope.kt
0.00% <ø> (ø)

...atadog/android/core/configuration/Configuration.kt
97.12% <100.00%> (+0.12%)
:arrow_up:

...n/com/datadog/android/core/internal/DatadogCore.kt
81.82% <100.00%> (-0.24%)
:arrow_down:

...id/core/internal/data/upload/DataUploadRunnable.kt
96.49% <100.00%> (-0.33%)
:arrow_down:

...d/core/internal/data/upload/DataUploadScheduler.kt
100.00% <100.00%> (ø)

.../android/core/internal/logger/SdkInternalLogger.kt
88.89% <100.00%> (+0.11%)
:arrow_up:

...oid/core/internal/metrics/MethodCalledTelemetry.kt
100.00% <100.00%> (ø)

...rnal/persistence/datastore/NoOpDataStoreHandler.kt
100.00% <100.00%> (ø)

.../core/internal/persistence/datastore/ext/IntExt.kt
100.00% <100.00%> (ø)

... and 93 more

... and 255 files with indirect coverage changes


can we use only one esp32 board?

I am using only one esp32 board.
My setup is like this :
One Host (My PC ) is connected with One Access Point  ( 2.4 GHz , channel 1 ).
I am using the ESP32 as passive csi collector. I set the channel at 1.
I am not getting any output.

root@bidyut-System-Product-Name:/home/bidyut/ESP32-CSI-Tool/passive# make monitor
Toolchain path: /root/.espressif/tools/xtensa-esp32-elf/esp-2021r1-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc
Toolchain version: esp-2021r1
Compiler version: 8.4.0
Python requirements from /root/esp/esp-idf/requirements.txt are satisfied.
MONITOR
--- idf_monitor on /dev/ttyUSB0 115200 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
ets Jun  8 2016 00:22:57
rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:6380
ho 0 tail 12 room 4
load:0x40078000,len:14236
load:0x40080400,len:3820
0x40080400: _init at ??:?
entry 0x40080698
I (28) boot: ESP-IDF v4.4-dev-2128-gf45b60b28 2nd stage bootloader
I (28) boot: compile time 01:06:11
I (28) boot: chip revision: 1
I (33) boot_comm: chip revision: 1, min. bootloader chip revision: 0
I (40) boot.esp32: SPI Speed      : 40MHz
I (44) boot.esp32: SPI Mode       : DIO
I (49) boot.esp32: SPI Flash Size : 2MB
I (53) boot: Enabling RNG early entropy source...
I (59) boot: Partition Table:
I (62) boot: ## Label            Usage          Type ST Offset   Length
I (70) boot:  0 nvs              WiFi data        01 02 00009000 00006000
I (77) boot:  1 phy_init         RF data          01 01 0000f000 00001000
I (85) boot:  2 factory          factory app      00 00 00010000 00100000
I (92) boot: End of partition table
I (96) boot_comm: chip revision: 1, min. application chip revision: 0
I (103) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=171ach ( 94636) map
I (146) esp_image: segment 1: paddr=000271d4 vaddr=3ffb0000 size=038f8h ( 14584) load
I (152) esp_image: segment 2: paddr=0002aad4 vaddr=40080000 size=05544h ( 21828) load
I (161) esp_image: segment 3: paddr=00030020 vaddr=400d0020 size=7d9e0h (514528) map
I (348) esp_image: segment 4: paddr=000ada08 vaddr=40085544 size=100d4h ( 65748) load
I (375) esp_image: segment 5: paddr=000bdae4 vaddr=50000000 size=00010h (    16) load
I (386) boot: Loaded app from partition at offset 0x10000
I (386) boot: Disabling RNG early entropy source...
I (397) cpu_start: Pro cpu up.
I (398) cpu_start: Starting app cpu, entry point is 0x400811b8
0x400811b8: call_start_cpu1 at /root/esp/esp-idf/components/esp_system/port/cpu_start.c:155
I (0) cpu_start: App cpu up.
I (412) cpu_start: Pro cpu start user code
I (412) cpu_start: cpu freq: 160000000
I (412) cpu_start: Application information:
I (416) cpu_start: Project name:     passive
I (421) cpu_start: App version:      f8f95a5
I (426) cpu_start: Compile time:     Jul 20 2021 01:06:09
I (432) cpu_start: ELF file SHA256:  0884d64dc63e16d6...
I (438) cpu_start: ESP-IDF:          v4.4-dev-2128-gf45b60b28
I (445) heap_init: Initializing. RAM available for dynamic allocation:
I (452) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (458) heap_init: At 3FFB7690 len 00028970 (162 KiB): DRAM
I (464) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (471) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (477) heap_init: At 40095618 len 0000A9E8 (42 KiB): IRAM
I (484) spi_flash: detected chip: generic
I (488) spi_flash: flash io: dio
W (492) spi_flash: Detected size(4096k) larger than the size in the binary image header(2048k). Using the size in the binary image header.
I (506) cpu_start: Starting scheduler on PRO CPU.
I (0) cpu_start: Starting scheduler on APP CPU.
E (547) vfs_fat_sdmmc: slot init failed (0x103).
E (547) sd.h: Failed to initialize the card (ESP_ERR_INVALID_STATE).   If you do not have an SD card attached, please ignore this message.  Make sure SD card lines have pull-up resistors in place.
W (561) wifi_init: Failed to set default Wi-Fi event handlers (0x103)
I (568) wifi:wifi driver task: 3ffbeed4, prio:23, stack:6656, core=0
I (573) system_api: Base MAC address is not set
I (578) system_api: read default base MAC address from EFUSE
I (589) wifi:wifi firmware version: 4c9ede1
I (589) wifi:wifi certification version: v7.0
I (592) wifi:config NVS flash: enabled
I (596) wifi:config nano formating: disabled
I (600) wifi:Init data frame dynamic rx buffer num: 32
I (605) wifi:Init management frame dynamic rx buffer num: 32
I (610) wifi:Init management short buffer num: 32
I (614) wifi:Init dynamic tx buffer num: 32
I (619) wifi:Init static rx buffer size: 2212
I (623) wifi:Init static rx buffer num: 10
I (626) wifi:Init dynamic rx buffer num: 32
I (631) wifi_init: rx ba win: 6
I (634) wifi_init: tcpip mbox: 32
I (638) wifi_init: udp mbox: 6
I (642) wifi_init: tcp mbox: 6
I (645) wifi_init: tcp tx win: 5744
I (650) wifi_init: tcp rx win: 5744
I (654) wifi_init: tcp mss: 1440
I (658) wifi_init: WiFi IRAM OP enabled
I (662) wifi_init: WiFi RX IRAM OP enabled
I (668) phy_init: phy_version 4670,719f9f6,Feb 18 2021,17:07:07
I (776) wifi:mode : null
I (777) wifi:ic_enable_sniffer
type,role,mac,rssi,rate,sig_mode,mcs,bandwidth,smoothing,not_sounding,aggregation,stbc,fec_coding,sgi,noise_floor,ampdu_cnt,channel,secondary_channel,local_timestamp,ant,sig_len,rx_state,real_time_set,real_timestamp,len,CSI_DATA
--- after this no output is comming.

How long did you keep the ESP32 running? Are you sure that CSI is being generated by your host and access point? Overall, your setup of the tool sounds correct to me, there must be an issue with the host to access point part.

Thank you for your response, Sir.
Q. How long did you keep the ESP32 running?
Ans. : 4 to 5 minutes. Not working.
But after few trials, it's showing the CSI data ( In passive mode ). Sometimes It stopped and does not show any received
CSI  data.
Q.  Are you sure that CSI is being generated by your host and access point?
How do I confirm it ? will It works on AP Beacon and STA probe frame?

From my understanding beacon frames will not work, it needs to be actual DATA frames. However, I do not currently have a list of what frames will or will not generate CSI in passive mode.
Here is my request for you: do you think you can you help figure out what sort of frames will generate CSI in passive mode? That would be a huge help towards this project!

Hi StevenMHernandez,
Yes , beacon frames will not work. Even Control and Management Frames are not generating CSI Data. I tested it with different ways. After receiving the Data Frames  [ void  _wifi_csi_cb(void *ctx, wifi_csi_info_t *data) ] callback method is invoked. So we only need to data frames.
_wifi_csi_cb(void *ctx, wifi_csi_info_t *data)  method doesn't provide any frame header details. So we may not verify the frame type of extracted CSI. What do you thing ?

What ways did you test beacon, control and management frames? Did you use some specific software?

I deployed a RPI  with packet sniffer mode and using tcp dump I can see Beacon and other frames (except data frame) but at the same time no CSI data is received in the esp 32 model (esp32 is configured in same chanel No. )

I am also frequently seeing this error :
E (10578) task_wdt: Task watchdog got triggered. The following tasks did not reset the watchdog in time:
E (10578) task_wdt:  - IDLE (CPU 0)
E (10578) task_wdt: Tasks currently running:
E (10578) task_wdt: CPU 0: wifi
E (10578) task_wdt: CPU 1: IDLE
E (10578) task_wdt: Print CPU 0 (current core) backtrace
**** Any advice to  fix this Sir?

If I was to recreate your setup, all I need to run is tcpdump on the command line? Are there any other options/flags that you used? You are not actively sending specific packet types with the raspberry pi are you?
I forgot to mention: (https://github.com/StevenMHernandez/ESP32-CSI-Tool/blob/master/passive/main/main.c#L27) only allows DATA frames to work for CSI. Please add more flags and report back to let me know if this helps you.
For the watchdog error:

what is your baud rate
what is the git-hash of esp32-csi-tool
what is your esp-idf version
is this for the passive project?

Setup :
Device #1 :  Wifi Router ( without internet connectivity , channel = 1)
Device #2 :  Raspberry Pi W 0 ( in wifi monitor mode)
Device #3 :  Esp32 (in CSI Pasive mode ,channel = 1 , )
        const wifi_promiscuous_filter_t filt = {
        .filter_mask = WIFI_PROMIS_FILTER_MASK_ALL
        };

        in main.c to capture all types of frame ( control, management and Data )

Device #4 : Laptop ( UDP Server)
Device #5 : Laptop ( UDP client)
In this scenerio ,
        I run "tcpdump -i mon0" in rpi terminal where mon0 is the monitor mode interface and saw
        diffrent becoan frames and other frames . But at that time No csi data is capturing by the esp ( serial port )

        But if I connect 2 pc to that network and create a udp client server and exchange data between the client and server
        I can see the csi data captured by ESP ( serial port ), though the filter mask = WIFI_PROMIS_FILTER_MASK_ALL ( which is supposed to intercept all
        types of frames) in the esp32 board. But if I stop the UDP data communication between the client and server then CSI captured also paused.

        So I thought only Data frames can gives CSI data to us what ever the filtermask is used in "wifi_promiscuous_filter_t".

        void _wifi_csi_cb(void *ctx, wifi_csi_info_t *data)  : this is the only call back method which is called after the CSI is captured/estimated by the ESP but unfortunatly
        no parameter is here to confirm from which types of packet this CSI is estimated.

Q. You are not actively sending specific packet types with the raspberry pi are you?
NO i am not sending any packet from RPI.
Q.  all I need to run is tcpdump on the command line? Are there any other options/flags that you used?
Yes in cmd line I run . I had not used any other flag.
whenn I apply filter_mask = WIFI_PROMIS_FILTER_MASK_ALL then It should capture all types of frame in promisous mode right?
according to the API it is says that CSI is estimated whenever a packet is recived by the esp from the transmitter. I am little bit confused
with it ( only data packet can estimate CSI)
***** Actually I am giving more weight in becoan frame because if esp estimates CSI data  from  the becoan frame send by  the acess point, then there is no headace for  deploying udp server and client for data traffic. We can use only on single esp32 bord and there should not be any dependency for data traffic generation , by sending becoan frame by any access point , esp32 recives that farme and estimates its CSI.

For the watchdog error:
what is your baud rate : 115200
what is the git-hash of esp32-csi-tool : f8f95a5edca2309fd9ac1ff57828acdc8893f05d
what is your esp-idf version : ESP-IDF v4.4-dev-2128-gf45b60b28
is this for the passive project? : Yes

Hi,
Is there any documentation to find the indices of null and pilot sub carriers so that we can filter them out and only consider data sub carrier?
We need to use only data sub carriers for processing.

See (https://github.com/StevenMHernandez/ESP32-CSI-Tool/issues/26#issuecomment-890544170). ESP-IDF v4.0 is tested but >v4.0 is unknown. You can test each version (v4.0, v4.1, v4.2, v4.3, up to your current version v4.4) to figure out when this watchdog error started appearing and report back in #11. That would be helpful to improve this project. But currently, I am not working on updating the version.
I do not have specific documentation about null/pilot subcarriers currently. Null subcarriers should be obvious in the data, they are all zeros. Did you do some research on your own to figure out the pilot subcarrier indices? If so, please share what you found.
If you think it would be useful to other users of the tool, please send a pull request with this information added to the Readme.md file. Thanks!

I am using only one esp board with active_sta project. However, I am not getting any data.
I (4240) tcpip_adapter: sta ip: <IP_ADDRESS>, mask: <IP_ADDRESS>, gw: <IP_ADDRESS>
I (4241) Active CSI collection (Station): Got ip: <IP_ADDRESS>
initial wifi connection established.
sending frames.

It stops at this. I have changed WIFI_CHANNEL and my SSID in main.cc
#define EXAMPLE_ESP_WIFI_SSID      "niyamssid"
#define EXAMPLE_ESP_WIFI_PASS      "niyampassword"
#define EXAMPLE_ESP_MAXIMUM_RETRY  10

#ifdef CONFIG_WIFI_CHANNEL
#define WIFI_CHANNEL CONFIG_WIFI_CHANNEL
#else
#define WIFI_CHANNEL 1
#endif

Also when I am using one board with passive project, I also dont get any data,
W (692) wifi_init: Failed to set default Wi-Fi event handlers (0x103)
I (713) wifi: wifi driver task: 3ffbac3c, prio:23, stack:3584, core=0
I (713) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (718) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (745) wifi: wifi firmware version: 581f422
I (746) wifi: config NVS flash: enabled
I (746) wifi: config nano formating: disabled
I (746) wifi: Init dynamic tx buffer num: 32
I (750) wifi: Init data frame dynamic rx buffer num: 32
I (755) wifi: Init management frame dynamic rx buffer num: 32
I (761) wifi: Init management short buffer num: 32
I (765) wifi: Init static rx buffer size: 2212
I (769) wifi: Init static rx buffer num: 10
I (773) wifi: Init dynamic rx buffer num: 32
I (881) phy: phy_version: 4180, cb3948e, Sep 12 2019, 16:39:13, 0, 0
I (882) wifi: mode : null
I (883) wifi: ic_enable_sniffer
type,role,mac,rssi,rate,sig_mode,mcs,bandwidth,smoothing,not_sounding,aggregation,stbc,fec_coding,sgi,noise_floor,ampdu_cnt,channel,secondary_channel,local_timestamp,ant,sig_len,rx_state,real_time_set,real_timestamp,len,CSI_DATA

Do I have to set the BASE MAC ADDRESS somewhere?

@NiyamPoudel Likely there is nothing nearby that is transmitting CSI-enabled OFDM frames for the passive device to collect. I would recommend reading this thread and looking at what questions I asked in the past (i.e. https://github.com/StevenMHernandez/ESP32-CSI-Tool/issues/22#issuecomment-884219664). If you answer these questions, I am more likely able to help you out.

For now, the original problem for this github-issue is "why is there no csi when in passive-mode?".
For future readers, the answer likely is:

There are no devices transmitting CSI-enabled OFDM frames nearby (Maybe devices are not using 2.4GHz or the devices are using a different wifi channel than what you set for the ESP32)
Keep the device running for a long time to see if there are any CSI is eventually collected

I am going to close this issue since it seems the issue was solved and now it might be diverging into unrelated support-style questions. If there are any other questions, please check other issues or create your own issue.


HBASE-28128 Reject requests at RPC layer when RegionServer is aborting

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 30s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 28s
master passed

+1 :green_heart:
compile
0m 40s
master passed

+1 :green_heart:
shadedjars
4m 42s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 25s
master passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m 14s
the patch passed

+1 :green_heart:
compile
0m 40s
the patch passed

+1 :green_heart:
javac
0m 40s
the patch passed

+1 :green_heart:
shadedjars
4m 42s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 24s
the patch passed

_ Other Tests _

-1 :x:
unit
10m 15s
hbase-server in the patch failed.

28m 33s

Subsystem
Report/Notes

Docker
ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5447/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/5447

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux bb14aa1106f1 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 076b5fb27f

Default Java
Temurin-1.8.0_352-b08

unit
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5447/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt

Test Results
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5447/1/testReport/

Max. process+thread count
1533 (vs. ulimit of 30000)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5447/1/console

versions
git=2.34.1 maven=3.8.6

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 26s
Docker mode activated.

I'm surprised that it's not already there. We're usually pretty good about checking for abort/stopped status before processing, but maybe that's at a layer down.

Yea, we check this in pretty much every RSRpcServices endpoint. The problem is that's post-queue, and this adds the check pre-queue. Otherwise if the RS is fully hosed we will end up throwing CallQueueTooBigException (doesnt clear meta) when it really should be ServerAborted (does clear meta)

The test failure is unrelated. @Apache9 let me know if you have any other concerns, otherwise I'll merge tomorrow


Help Me Please!!!!!!!!!!!!!

I wanted to install pandas-profiling in my pc and was unable to do so just because of hashes i am a absolute begginer help me please.

Give us information about errors please. And which required libraries' versions you have already installed.

Install Anaconda it already installs pandas for you to use in your Jupiter notebook :D


What made Trajectories capable to control more CF?

Hi
I some have waypoints generated from Blender

Here's a csv of the path shown above
sequence_000.zip
It has: Timestamp (100ms interval), x,y,z (absolute) and r, g, b
I managed to use Crazyflie's official python lib's fadeColor for RGB and send_position_setpoint for xyz
With 1 radio, it works pretty well for 1 drone, however with more it flies a bit more wobbly and will not work for more than 4 drones (bandwidth issue)
Currently trying to use more radios (didn't help in first attempt)
I was told Crazyswarm has better communication efficiency, is it the
the pycrazyswarm in particular https://crazyswarm.readthedocs.io/en/latest/api.html
or is it Trajectories the same as the latest official lib? (in terms of efficiency/performance)
To better understand the problem, in what way could Crazyswarm handle 15+ CFs per radio compared to official python lib that could do only 3-4 per radio?
From my limited understanding, instead of blasting coordinates at every 100ms or faster, you pre-program certain trajectories (which we have no fine control over the time)
And activate them on-demand, thus greatly lowering the frequency / amount of communication required.
Or are there some fancy optimisation in the protocol level?
Could the goTo or go_to command also achieve 15+ CF per radio?
send_position_setpoint at 10hz can only go up to 3-4 CF per radio

You are correct that most of the improvement comes from pre-loading trajectories and then executing everything without the need of much communication. In the Crazyswarm, the radio is mostly used to broadcast pose/position information from the motion capture system. If you use the lighthouse or LPS localization systems, this would not be needed.
In principle, it is possible to improve the encoding for setpoints as well, but this would require modifications in the firmware and the client applications.
So if you use motion capture, I think one might be able to achieve around 8 CFs/radio if you also want to send position commands (and the position commands get encoded in a smarter way). If you use LPS/lighthouse, you should be able to get 15+ CFs/radio (if the positions commands get encoded in a smarter way). Without code changes, I think around 4 CFs/radio are realistic.


Add option to keep pagination meta data when get scenes

Thanks! But I don't think it's a good idea to change a function's return type based on arguments in general. I am closing this in favor of https://github.com/mujin/mujincontrollerclientpy/pull/55


Verify that user and root default shell are bash

This probably unnecessarily breaks everything when root shell is dash.


Create recursive fibonacci c algorithm

Algorithm works recursively and has data visualization so you can have a feel of the process, but need improvements

same implementation of an algorithm is submitted by someone else and it gets accepted
Thanks for contributing


Edit/remove group membership on user profile

There should be a burger menu

Edit role
Remove

groups can only be deleted (with confirmation)
users can be changed - should show inline form for role changing


scroll for typehead

#1168 Added scrolling
Fixed issue with IE not refreshing list when using the cross to clear the input
Fixed scrolling not showing in IE
Also fixed some of the issues you told me to fix

Huge PR, it will be hard to merge, but I will try to :)

We could create ng2-bootstrap gitter (https://gitter.im) and continue the discussion over there if you need any help.


Fix id replace in structure viewer options

HTML formatting by prettier added unexpected double quotes around the id.
Extracted from #80 since it is useful by itself.

Without this I get errors like "Error: unable to get element with id chsp-8f70a20e-a181-4ce5-a26f-8b2e93d8ecc5-atom-labels", since the regex do not match the id.
And yes, I know I should not be parsing HTML with a regex: https://stackoverflow.com/a/1732454/4692076

Merging this since https://chemiscope.org/ is currently broken ...


net/mail: characters allowed in RFC 5322 are invalid while parsing email header

What version of Go are you using (go version)?

$ go version
go version go1.20.1 darwin/amd64

Does this issue reproduce with the latest release?
Yes.
What did you do?
Create an empty directory and save code below in main.go, run go mod tidy && go run ..
Note: it contains header Custom/Header: v.
package main

import (
	"log"
	"net/mail"
	"strings"
)

func main() {
	msg := `Date: Mon, 23 Jun 2015 11:40:36 -0400
From: Gopher<EMAIL_ADDRESS>To: Another Gopher<EMAIL_ADDRESS>Subject: Gophers at Gophercon
Custom/Header: v

Message body
`

	r := strings.NewReader(msg)
	m, err := mail.ReadMessage(r)
	if err != nil {
		log.Fatal(err)
	}

	println(m.Header.Get("Custom/Header"))
}

What did you expect to see?
Print value of email header Custom/Header: v
What did you see instead?
2023/03/04 20:57:52 malformed MIME header line: Custom/Header: v
exit status 1

Function mail.ReadMessage() calls textproto.ReadMIMEHeader(), it calls internal function validHeaderValueByte() (in net/textproto) to validate characters in email header field, / in header field is considered as invalid.
The problem is, validHeaderValueByte() is designed to validate characters in HTTP header, not email header.
According to RFC 5322 "Internet Message Format", section "2.2 Header fields":
   A field name MUST be composed of printable US-ASCII characters (i.e.,
   characters that have values between 33 and 126, inclusive), except
   colon.

Here's full list of printable US-ASCII characters:
https://www.ascii-code.com/characters/printable-characters
For example, characters /, *, [, ] are allowed in email header field according to RFC doc, but it won't pass current validHeaderFieldByte() because it's disallowed in http header field. This causes parse error as shown in above sample code. I believe there're some similar issues caused by validHeaderValueByte() too.
We may need a new exported function like ReadMIMEHeader but for email (maybe name it as ReadEmailMIMEHeader()), and allow characters as defined in RFC 5322, but other code in net/textproto may need some changes too.

https://dev.golang.org/owners lists @bradfitz as the maybe-owner of net/mail?

Duplicate #39591

Duplicate #39591

Not a duplicate. This issue is for email header field, but #39591 is for header value.

It's working on version 1.19

It's working on version 1.19

Yes it works in Go v1.19, but changed in v1.20 due to the fix in this issue:
https://github.com/golang/go/issues/53188
Again, net/textproto is (mostly) designed for http, not for email.

cc @neild

The problem code in net/mail/message.go is:
    53  func ReadMessage(r io.Reader) (msg *Message, err error) {
    54  	tp := textproto.NewReader(bufio.NewReader(r))
    55
    56  	hdr, err := tp.ReadMIMEHeader()
    57  	if err != nil {
    58  		return nil, err
    59  	}
    60
    61  	return &Message{
    62  		Header: Header(hdr),
    63  		Body:   tp.R,
    64  	}, nil
    65  }

Using textproto..ReadMIMEHeader() is incorrect, as a valid header in textproto is, per the comment in that code:
   655  // validHeaderFieldByte reports whether c is a valid byte in a header
   656  // field name. RFC 7230 says:
   657  //
   658  //	header-field   = field-name ":" OWS field-value OWS
   659  //	field-name     = token
   660  //	tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" / "+" / "-" / "." /
   661  //	        "^" / "_" / "`" / "|" / "~" / DIGIT / ALPHA
   662  //	token = 1*tchar

which is incorrect for a mail header, RFC 5322 says a valid header name is:
Header fields are lines beginning with a field name, followed by a
colon (":"), followed by a field body, and terminated by CRLF.  A
field name MUST be composed of printable US-ASCII characters (i.e.,
characters that have values between 33 and 126, inclusive), except
colon.

Can someone please fix it?  ....please....

@gopherbot Please open backport to 1.20.
This stopped working due to the changes in #53188, but those changes don't apply to net/mail.


fixed #175 CAN not working with lowercase

fixed the issue #175, allowing the cangen to work successfully regardless of the capitalization of node names in the .dbc file for a project and the node name in the config.yaml file.

tested and generates with upper, mixed, and lowercase node names


Confusing content

I have been trying to find out how to use remote builds on linux using azure devops, to no avail.
[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 4dc400d6-feaf-3239-094e-3c19aae44345
Version Independent ID: aabb7db4-27fc-63c9-b019-d116450c439f
Content: Deployment technologies in Azure Functions
Content Source: articles/azure-functions/functions-deployment-technologies.md
Service: azure-functions
GitHub Login: @ggailey777
Microsoft Alias: glenga

@mamoorkhan There is another document that talks about Continuous delivery with Azure Pipelines. Please refer to the YAML build pipeline.
This section only talks about the setting that needs to be added in your configuration.

@mamoorkhan Hope the above helps and clears things up. We will now proceed to close this thread. If there are further questions regarding this matter, please reopen it and we will gladly continue the discussion.

@MayankBargali-MSFT when I use above mentioned articles, it automatically sets up the RUN_WEBSITE_FROM_PACKAGE url


[Perf: Selection, Dragging] Don’t rerender NodeRenderer and MiniMapNodes unless necessary

This PR ensures that when you’re dragging a node around the screen (or selecting it, or updating it in any other way), NodeRenderer (and MiniMapNodes) don’t rerender.
This prevents us from re-running this loop N times (N being the number of nodes) on every store update. With 500 nodes, it gets surprisingly expensive (20-40 ms with CPU throttling)!
(You can review this PR commit-by-commit.)
More details
Store-level changes
To make this possible, I had to make two significant design changes (both in 24390162e08867a7c68c309a0ebcd52dafd6d425).
First, I had to split the Node term into two: InternalNode and UserProvidedNode.

UserProvidedNode means a node object that the user stores in their state and provides via nodes, defaultNodes, or api.setNodes().
InternalNode means a node extended from the user-provided one with additional properties (computed and [internalSymbol]).

Note that this change is only 70% ready: this PR introduces these terms only in the React store and system store. If I was in charge of shipping this, I’d also update types; make api.getNodes() convert back to a user-provided node (by stripping internal properties); etc.
Second, I had to introduce a check that avoid recalculating the internal node if the user-provided node hasn’t changed:
https://github.com/iamakulov/xyflow/blob/d4484be0c39abd396c506de1be69c5d460279873/packages/system/src/utils/store.ts#L79
This was critical for memoization. Previously, any time the nodes prop would update, we’d regenerate every node object. Now, we only regenerate the changed ones.
However, this is a potential breaking change as it makes React Flow ignore the following kind of updates (previously, they were respected, even if not officially supported):
const [nodes, setNodes] = useState([...]);

useEffect(() => {
  setNodes((nodes) => {
    nodes[5].draggable = false;
    return nodes
  });
}, []);

return <ReactFlow nodes={nodes} />

Component-level changes
With store changes implemented, Nodes won’t regenerate anymore unless they actually change. This means we can now write memoization around node objects. We do this in 0186b661611347869291f2f941e8f387ce55a76c and 4fd321a8ff074a5837020ec2d4c83aaf4a678e2a for NodeRenderer (+<PHONE_NUMBER>92d92d09a62cd7c4655b9ac45f6e40 for MiniMapNodes).
The above commits:

split NodeRenderer (and MiniMapNodes) into two components instead of one
make sure NodeRenderer (and MiniMapNodes) only rerender on node ID changes
make the child component rerender when its (and only its) node updates

As a result, if, before, we were continuously rerendering the whole NodeRenderer every time you’d drag a single node:

now we only rerender a concrete NodeComponent:

Performance wins
This PR makes dragging 2-3× cheaper, from 60-75 ms per drag event to 25 ms. Compare: trace before · trace after.
Selection also gets ~1.5× cheaper, on top of https://github.com/xyflow/xyflow/pull/3667 (from 75-80 ms to 45-50 ms). Compare: trace at PR 3667 · trace with both PRs applied

Note: upon further testing, this seems to increase the initial rendering costs ~1.5 times (in development, in the stress test) – because now, for every node, we also have to render the NodeComponentWrapper (= more components to render).

The increase might be smaller in the prod build
We can maybe reduce this increase by doing some custom memoization instead of relying on memo(NodeComponentWrapper). But that will make it harder to manage memory (we’ll need to learn when a node is removed so we can remove the cached entry for that node)

The increase might be smaller in the prod build

Oh yeah – in a production build, the difference is <10% (1004 ms of scripting time at next → 1093 ms with this PR and #3667 applied).
My next PR should somewhat reduce the init time and bring that value back.

Hey @iamakulov
thanks again for your great work and the super detailed PR descriptions! Mapping just over the node ids was a great idea and really pushed the performance of dragging nodes. We took a lot of parts from this PR and it helped us to create some new refactoring ideas. After working on your changes we got rid of the NodeComponentWrapper and the wrapNode HOC and only use a simple NodeWrapper component. In addition we could remove lot of util code about nodeTypes and edgeTypes.
This PR was continued in #3727


[Refactor] Chop too large hyperscan regex pattern when logging it on an error

Why I'm doing:
When hypercan patten exceeds some threshold(30KB+), hyperscan compilation report error and log the error, the size of be log would grow crazily.
What I'm doing:
Fixes #issue
What type of PR is this:

[ ] BugFix
[ ] Feature
[ ] Enhancement
[x] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

[ ] This is a backport pr

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[x] 3.3
[x] 3.2
[x] 3.1
[x] 3.0
[x] 2.5

@mergifyio backport branch-3.3

https://github.com/Mergifyio backport branch-3.2

https://github.com/Mergifyio backport branch-3.1

https://github.com/Mergifyio backport branch-3.0

https://github.com/Mergifyio backport branch-2.5


Update code analysis rules

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (e6f3722) to head (dbb975a).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff            @@
##            master      #111   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           39        33    -6
  Lines          423       391   -32
  Branches        54        54
=========================================
- Hits           423       391   -32

Flag
Coverage Δ

unittests
100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


User Checkout Paeg #7 +Barcode/QR scanner integration #5

Issues #7, #5
I added some test data for now to see if it could display within the code. Type in the first text box to see it. Has checkout page and return to checkout page
"ks81":"Apple",
"9sj2s":"Kiwi",
"j51s":"Chips"

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 96.667%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
58

Relevant Lines:
60

💛  - Coveralls

@Palacios1337 Could you move the css formatting from your .html files into the stylesheet file?

@Palacios1337 I re-named your app to checkout to differentiate from the inventory that others have been making, I am rebuilding the links to that so you should need to worry about anything outside your functionality yet.
Static files are tricky but they should belong in /shop-inventory/checkout/static/checkout/style.css. Check out here for the tutorial: https://docs.djangoproject.com/en/5.0/intro/tutorial06/

coming back to finalize merges, it looks like the git history has been completely lost here. Investigating options.


Can't run with gpu

When I try to run with CUDA, I get next error:
Traceback (most recent call last):
  File "recognize.py", line 60, in <module>
    recognize(wav_file)
  File "recognize.py", line 21, in recognize
    results = speech2text(speech)
  File "/root/espnet/venv/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/root/espnet/venv/lib/python3.8/site-packages/espnet2/bin/asr_inference.py", line 309, in __call__
    nbest_hyps = self.beam_search(
  File "/root/espnet/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/espnet/venv/lib/python3.8/site-packages/espnet/nets/beam_search.py", line 367, in forward
    running_hyps = self.post_process(i, maxlen, maxlenratio, best, ended_hyps)
  File "/root/espnet/venv/lib/python3.8/site-packages/espnet/nets/batch_beam_search.py", line 343, in post_process
    return self._batch_select(running_hyps, remained_ids)
  File "/root/espnet/venv/lib/python3.8/site-packages/espnet/nets/batch_beam_search.py", line 46, in _batch_select
    score=hyps.score[ids],
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

S2S init by this code:
speech2text = Speech2Text(
        asr_train_config=train_config,
        asr_model_file=model_file,
        lm_train_config=lm_config,
        lm_file=lm_file,
        token_type=None,
        bpemodel=None,
        maxlenratio=0.0,
        minlenratio=0.0,
        beam_size=10,
        ctc_weight=0.5,
        lm_weight=0.3,
        penalty=0.0,
        nbest=1,
        device = "cuda"
    )

Hi. I couldn't reproduce your error. Could you specify what is your torch version and espnet version?

Hi, I use Python 3.8.10 with this libs
Package                  Version
------------------------ ------------
appdirs                  1.4.4
audioread                3.0.0
certifi                  2022.12.7
cffi                     1.15.1
charset-normalizer       3.1.0
ci-sdr                   0.0.2
click                    8.1.3
cmake                    3.26.1
ConfigArgParse           1.5.3
ctc-segmentation         1.7.4
Cython                   0.29.33
decorator                5.1.1
Distance                 0.1.3
einops                   0.6.0
espnet                   202207
espnet-tts-frontend      0.0.3
nvidia-cublas-cu11       <IP_ADDRESS>
nvidia-cuda-cupti-cu11   11.7.101
nvidia-cuda-nvrtc-cu11   11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11        <IP_ADDRESS>
nvidia-cufft-cu11        <IP_ADDRESS>
nvidia-curand-cu11       <IP_ADDRESS>
nvidia-cusolver-cu11     <IP_ADDRESS>
nvidia-cusparse-cu11     <IP_ADDRESS>
nvidia-nccl-cu11         2.14.3
nvidia-nvtx-cu11         11.7.91
torch                    2.0.0
torch-complex            0.4.3

Please try to use '1.12.0+cu116' torch, as we tested with this version.


Confusing unicode exclusion option for U+202e

Testing #138071

U+202E is RIGHT-TO-LEFT OVERRIDE but not sure what "dethgilhgih ..." means ...

It is reversed :D
Nice catch of that unicode injection attack.

Because all such control characters are invisible, the fix for #138184 also fixes this:


Cascade Parameters set in _index.md cannot be overwritten / added to in children Pages

As described in this forum topic -and confirmed by me- Cascade parameter keys set in the _index.md of a Section or Nested Section cannot be overwritten in the frontmatter of children pages.
Content tree:
/content
	/book-1
		_index.md
		/part-1
			_index.md
			/chapter-1
				featured.jpg
				index.md

Cascade parameters set in content/book-1/_index.md
cascade:
    authors: John Smith
    layout: book
    book:
        title: Example Title
        publisher: Example Publisher
        publish_date: 2020-01-01

If a user tries to add an extra parameters in chapter-1/index.md under the book: key, only the new parameters show up, replacing what was set above.
For example if a user adds the following in chapter-1/index.md
book:
    part: 1
    chapter: 1

Then all key value pairs under book: set in content/book-1/_index.md are removed and only what entered in the child page chapter-1/index.md  is returned.
I think that Hugo should allow overwriting / adding to the keys of cascaded frontmatter.

I just saw that #8050 was already opened.
So closing this in favor of that.


abnormal positive/negative regulation

Gah - I know I have asked about this at least twice, possibly three times, and perhaps I'm just really dim with regard to getting my head around the abnormal positive/negative regulation terms, but I'm really struggling with using them.
Would it be possible to migrate them to something more explicit (for instance increased/decreased transcription compared to wild type) or include a definition?
Original comment by: Antonialock

closing because out of date; problems can go in new tickets


Changing shape kicks me out of the server

Mod loader
Forge
Minecraft version
1.20.1
Woodwalkers version
4.4.3
CraftedCore version
4.2.2
Modloader version
Forge 47.2.32
The latest.log file
https://mclo.gs/x54SxPO
Issue description
When I try to change shape, it kicks me out of the server with the exception:

java.lang.RuntimeException: Attempted to load class net/minecraft/client/player/LocalPlayer for invalid dist DEDICATED_SERVER

I tried modded mob and then used commands to change to vanilla mob (wolf) and it happened again. This has been happening since the latest update to 4.4.3 :(
Steps to reproduce
No response
Other
No response

fixed as of 8090342bd54a88d6203ad9cfd123242e8e26a388


[Android]about Banner Size

Is there a way to set AdMobBannerSize in android to the device size(full-size) like version 1?
the adSize='banner' displayed like below which has white space around the ad.

User smart banners?

@UberMC  you mean "adSize='smartBanner'" ? It's the same .


PHP Fatal error when fetching outbox

When I try to remotely fetch the first page of my outbox, a PHP Fatal error occurs:
[02-Feb-2023 16:18:10 UTC] PHP Fatal error:  Uncaught TypeError: Activitypub\Model\Activity::from_post(): Argument #1 ($post) must be of type Activitypub\Model\Post, array given, called in /(path)/wordpress/wp-content/plugins/activitypub/includes/rest/class-outbox.php on line 106 and defined in /(path)/wordpress/wp-content/plugins/activitypub/includes/model/class-activity.php:48
Stack trace:
#0 /(path)/wordpress/wp-content/plugins/activitypub/includes/rest/class-outbox.php(106): Activitypub\Model\Activity->from_post(Array)
#1 /(path)/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1171): Activitypub\Rest\Outbox::user_outbox_get(Object(WP_REST_Request))
#2 /(path)/wordpress/wp-includes/rest-api/class-wp-rest-server.php(1018): WP_REST_Server->respond_to_request(Object(WP_REST_Request), '/activitypub/1....', Array, NULL)
#3 /(path)/wordpress/wp-includes/rest-api/class-wp-rest-server.php(442): WP_REST_Server->dispatch(Object(WP_REST_Request))
#4 /(path)/wordpress/wp-includes/rest-api.php(410): WP_REST_Server->serve_request('/activitypub/1....')
#5 /(path)/wordpress/wp-includes/class-wp-hook.php(308): rest_api_loaded(Object(WP))
#6 /(path)/wordpress/wp-includes/class-wp-hook.php(332): WP_Hook->apply_filters('', Array)
#7 /(path)/wordpress/wp-includes/plugin.php(565): WP_Hook->do_action(Array)
#8 /(path)/wordpress/wp-includes/class-wp.php(399): do_action_ref_array('parse_request', Array)
#9 /(path)/wordpress/wp-includes/class-wp.php(780): WP->parse_request('')
#10 /(path)/wordpress/wp-includes/functions.php(1332): WP->main('')
#11 /(path)/wordpress/wp-blog-header.php(16): wp()
#12 /(path)/index.php(7): require('/homepages/6/d8...')
#13 {main}
  thrown in /(path)/wordpress/wp-content/plugins/activitypub/includes/model/class-activity.php on line 48

Individual posts are okay, as is the outbox OrderedCollection; the error seems to be limited to the OrderedCollectionPage.

I have already fixed it in this PR https://github.com/pfefferle/wordpress-activitypub/pull/265, but I think I will release a new version with that fix, asap. but it should not cause any big issues, because the outbox is not really used anywhere yet.
Thanks for reporting!


feat: Bulk get schema + Altimate Scan

add logic to get full db catalog.
TODO - currently assuming all projects look at disjoint catalogs. There might be some way to optimize the calls but not sure if i can get that info from just hostname/db name etc.
Note - using get_catalog because it is supported in older versions as well. the newer version lets you get only certain schemas. This might be a little too heavy and we are relying on adapter-level optimizations (eg snowflake is restricting to the information_schema of that database). However, this is the same call that dbt docs generate makes so it wont be slower than that call. It does avoid all the other stuff that generate does so ideally this should be much faster.
Additionally this PR also builds on the catalog and has the starting pieces of the deep scan aka altimate scan.
Checklist

[ ] I have run this code and it appears to resolve the stated issue
[ ] README.md updated and added information about my change

Overall looks good. There are a few things that we need to take care

report the list of files skipped during the scan due to parsing or any other errors [ in future we should extend it to show a complete report]
send out count of the errors found per type and also the count of the number of files skipped

Can you either imrpove the tool tip to talk about what the scan does or add a line in the panel itself to explain what it does?


what is the most stable branch to deploy?

hey guys, i'm attempting to run this locally to try it out and I'm wondering which branch is the most stable.
i cloned the 18.x branch and tried to build the docker image and it failed.

I also had issues deploying with deploying 18.X. And the 17.X worked like a charm.
You should give it a try.


⚠ ClamAV has degraded performance

In 0e85176, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 76 ms

Resolved: ClamAV performance has improved in 1c65161.