[SPARK-4233] [SQL] WIP:Simplify the UDAF API (Interface)

Simplify the UDAF API is the first step of optimization for Aggregation (see https://issues.apache.org/jira/browse/SPARK-4366).
Currently UDAF cannot scale up when data volume grows, particularly for the distinct aggregation expressions.  This PR doesn't aim for fixing the distinct performance, but facilitate

More straightforward API for UDAF implementation
UDAF Developers will not write the distinct expression any more, like DistinctAverage is not necessary, as Average is provided, the framework will handle the distinct internally.
Row-based Aggregation Buffer
Aggregation Buffer will be stored as MutableRow with schema, it means the UDAF developers will benefit from the Catalyst Expression Evaluation framework in UDAF development.
Aggregation buffers shuffling cross the machine boundary is transparently from the UDAF developers, that will give us a chance to switch / tuning the aggregation algorithms without changing the UDAF interface.

The following sub tasks have been done:

[x] Integrated with Hive UDAF (Generic UDAF)
[x] Re-implemented the existed UDAF (e.g. max, first, last etc.)
[x] Code style issues.

Still some open issues need to be done with follow-up PRs:

Approximate aggregations
Partial aggregation for DISTINCT (before the data shuffling)
CodeGen for aggregation function (This PR has removed all of the aggregation codegen)
Sort-based aggregation (to reduce the overhead of memory footprint)
Aggregation optimization when Data Skew .

Data Gen Code:
import org.apache.spark.sql.hive.HiveContext
case class Source(key: String, value: Long)
    val sc = new SparkContext("local[6]", "TestSQLContext", new SparkConf().set("spark.shuffle.spill", "true"))
    val hc = new HiveContext(sc)

    val constant = 500000
    val data = sc.parallelize(1 to 75, 75).mapPartitions { i => {
        val rand = new java.util.Random(System.currentTimeMillis())
        new Iterator[Source] {
          var count = constant
          def hasNext = count >= 0
          def next(): Source = {
            count -= 1
            Source("Key" + rand.nextLong(), rand.nextLong())
          }
        }
      }
    }
    hc.createSchemaRDD(data).saveAsParquetFile("/tmp/source")

And the benchmark code and result:
import org.apache.spark.sql.hive.HiveContext

case class Source(key: String, value: Long)
val hc = new HiveContext(sc)
hc.parquetFile("/tmp/source").registerTempTable("source")
hc.sql("set spark.sql.shuffle.partitions=120")
hc.sql("set spark.sql.codegen=true")
hc.sql("create table IF NOT EXISTS result (c1 double, c2 double, c3 double, c4 double, c5 double)")

val q1 = "insert overwrite table result select count(value), max(value), min(value), sum(value), avg(value) from source"
val q2 = "insert overwrite table result select count(value), max(value), min(value), sum(value), avg(value) from source group by key"
val q3 = "insert overwrite table result select count(distinct value), max(value), min(value), sum(distinct value), avg(value) from source"
val q4 = "insert overwrite table result select count(distinct value), max(value), min(value), sum(distinct value), avg(value) from source group by key"

case class QueryResult(query: String, durations: Seq[Long], physicalPlan: Seq[String]) {
  override def toString() = s"""Query:$query \n Duration: ($durations)\n ${physicalPlan.mkString("\n")}"""
}
val results = new collection.mutable.ArrayBuffer[QueryResult]()
try {
  (q1 :: q2 :: q3 :: q4 :: Nil).foreach(q => {
        val physicalPlan = hc.sql(s"explain $q").collect().map(_.getString(0))
        val durations = (0 to 2).map { i =>
          val a = System.currentTimeMillis()
          hc.sql(q)
          val b = System.currentTimeMillis()
          b - a
        }
        val qr = QueryResult(q, durations, physicalPlan)
        results.append(qr)
        println(qr)
  })
} catch {
  case t: Throwable => t.printStackTrace()
} finally {
  results.foreach(println)
}

CMD: bin/spark-shell --master local[6] --jars conf/hive-site.xml --driver-memory 10g

SQL
New Implementation
Old Implementation
Old Implementation with CodeGen

Q1
(7412, 5548, 5432)
(6571, 4188, 3848)
(7821, 8637, 3894)

Q2
(287442, 205844, 283307)
(430723, 399582, 450138)
(409049, 382874, 380012)

Q3
(253209, 267187, 283325)
(81634, 109788, 117197)
(111527, 108582, 119733)

Q4
(363017, 550359, 319450)
(1909499, 1428532, 1654106)
(1336548, 1405936, 1253961)

CMD: bin/spark-shell --master local[6] --jars conf/hive-site.xml --driver-memory 5g

SQL
New Implementation
Old Implementation
Old Implementation with CodeGen

Q1
(8419, 5608, 5400)
(6632, 4339, 4166)
(7324, 4068, 3990)

Q2
(308409, 263482, 306832)
OOM
OOM

Q3
(350264, 346999, 371698)
OOM
OOM

Q4
(331502, 397524, 572660)
OOM
OOM

In general, the new implementation has better in memory usage, and faster while group by keys specified, in the meantime, we have room to optimize the non-group by key cases for the new implementation, which is supposed to have no impact with the UDAF interface.
PS: the old implementation seems doesn't support the query like select avg(distinct value) from xxx

test this please

Test build #25020 has started for   PR 3247 at commit 93df2f4.

This patch merges cleanly.

Test build #25020 has finished for   PR 3247 at commit 93df2f4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedFunction(
trait AggregateFunction
trait AggregateExpression extends Expression
abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression
case class Min(
case class Average(child: Expression, distinct: Boolean = false)
case class Max(child: Expression, distinct: Boolean = false)
case class Count(child: Expression)
case class CountDistinct(children: Seq[Expression])
case class Sum(child: Expression, distinct: Boolean = false)
case class First(child: Expression, distinct: Boolean = false)
case class Last(child: Expression, distinct: Boolean = false)
case class MinFunction(aggr: BoundReference, base: Min) extends AggregateFunction
case class AverageFunction(count: BoundReference, sum: BoundReference, base: Average)
case class MaxFunction(aggr: BoundReference, base: Max) extends AggregateFunction
case class CountFunction(aggr: BoundReference, base: Count)
case class CountDistinctFunction(aggr: BoundReference, base: CountDistinct)
case class SumFunction(aggr: BoundReference, base: Sum) extends AggregateFunction
case class FirstFunction(aggr: BoundReference, base: First) extends AggregateFunction
case class LastFunction(aggr: BoundReference, base: AggregateExpression) extends AggregateFunction
sealed case class AggregateFunctionBind(
sealed class KeyBufferSeens(
sealed trait Aggregate
sealed trait PreShuffle extends Aggregate
sealed trait PostShuffle extends Aggregate
case class AggregatePreShuffle(
case class AggregatePostShuffle(
case class DistinctAggregate(

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25020/
Test FAILed.

Test build #25043 has started for   PR 3247 at commit 74945d5.

This patch merges cleanly.

Test build #25043 has finished for   PR 3247 at commit 74945d5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedFunction(
trait AggregateFunction
trait AggregateExpression extends Expression
abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression
case class Min(
case class Average(child: Expression, distinct: Boolean = false)
case class Max(child: Expression, distinct: Boolean = false)
case class Count(child: Expression)
case class CountDistinct(children: Seq[Expression])
case class Sum(child: Expression, distinct: Boolean = false)
case class First(child: Expression, distinct: Boolean = false)
case class Last(child: Expression, distinct: Boolean = false)
case class MinFunction(aggr: BoundReference, base: Min) extends AggregateFunction
case class AverageFunction(count: BoundReference, sum: BoundReference, base: Average)
case class MaxFunction(aggr: BoundReference, base: Max) extends AggregateFunction
case class CountFunction(aggr: BoundReference, base: Count)
case class CountDistinctFunction(aggr: BoundReference, base: CountDistinct)
case class SumFunction(aggr: BoundReference, base: Sum) extends AggregateFunction
case class FirstFunction(aggr: BoundReference, base: First) extends AggregateFunction
case class LastFunction(aggr: BoundReference, base: AggregateExpression) extends AggregateFunction
sealed case class AggregateFunctionBind(
sealed class KeyBufferSeens(
sealed trait Aggregate
sealed trait PreShuffle extends Aggregate
sealed trait PostShuffle extends Aggregate
case class AggregatePreShuffle(
case class AggregatePostShuffle(
case class DistinctAggregate(

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25043/
Test FAILed.

Test build #25047 has started for   PR 3247 at commit 230e70f.

This patch merges cleanly.

Test build #25047 has finished for   PR 3247 at commit 230e70f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedFunction(
trait AggregateFunction
trait AggregateExpression extends Expression
abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression
case class Min(
case class Average(child: Expression, distinct: Boolean = false)
case class Max(child: Expression, distinct: Boolean = false)
case class Count(child: Expression)
case class CountDistinct(children: Seq[Expression])
case class Sum(child: Expression, distinct: Boolean = false)
case class First(child: Expression, distinct: Boolean = false)
case class Last(child: Expression, distinct: Boolean = false)
case class MinFunction(aggr: BoundReference, base: Min) extends AggregateFunction
case class AverageFunction(count: BoundReference, sum: BoundReference, base: Average)
case class MaxFunction(aggr: BoundReference, base: Max) extends AggregateFunction
case class CountFunction(aggr: BoundReference, base: Count)
case class CountDistinctFunction(aggr: BoundReference, base: CountDistinct)
case class SumFunction(aggr: BoundReference, base: Sum) extends AggregateFunction
case class FirstFunction(aggr: BoundReference, base: First) extends AggregateFunction
case class LastFunction(aggr: BoundReference, base: AggregateExpression) extends AggregateFunction
sealed case class AggregateFunctionBind(
sealed class InputBufferSeens(
sealed trait Aggregate
sealed trait PreShuffle extends Aggregate
sealed trait PostShuffle extends Aggregate
case class AggregatePreShuffle(
case class AggregatePostShuffle(
case class DistinctAggregate(

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25047/
Test PASSed.

@marmbrus , this PR passed the unit test, but some of details need to be discussed. Can you review this? Particularly for the UDAF interface design.
Sorry about so many code changes, as I almost rewrote all of the UDAF relevant code.

I only looked at this quickly, but I like the goals, especially the middle one.  Our current implementation is really wasteful on memory.  Some thoughts:

It would be good if you could write up a quick design doc that outlines the interfaces as right now its kind of hard to pull them out from all the other changes you have to make.
I wonder if it is possible to combine aggregate expression and aggregate function somehow.
Can you explain how the modes are used.  Do we really need them?

Other things:

Before we commit this we will have to implement the approximates.  I don't think its okay to regress in functionality here.
I'm not totally against removing the code generated version, but I'd have to see some performance tests that show we aren't regressing.

Test build #25783 has started for   PR 3247 at commit 2cae095.

This patch does not merge cleanly.

Test build #25783 has finished for   PR 3247 at commit 2cae095.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedFunction(
trait AggregateFunction
trait AggregateExpression extends Expression with AggregateFunction
abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression
case class Min(
case class Average(child: Expression, distinct: Boolean = false)
case class Max(child: Expression, distinct: Boolean = false)
case class Count(child: Expression)
case class CountDistinct(children: Seq[Expression])
case class Sum(child: Expression, distinct: Boolean = false)
case class First(child: Expression, distinct: Boolean = false)
case class Last(child: Expression, distinct: Boolean = false)
sealed case class AggregateFunctionBind(
sealed class InputBufferSeens(
sealed trait Aggregate
sealed trait PreShuffle extends Aggregate
sealed trait PostShuffle extends Aggregate
case class AggregatePreShuffle(
case class AggregatePostShuffle(
case class DistinctAggregate(

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25783/
Test FAILed.

Test build #26013 has started for   PR 3247 at commit feb00c8.

This patch merges cleanly.

Test build #26013 has finished for   PR 3247 at commit feb00c8.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedFunction(
trait AggregateFunction
trait AggregateExpression extends Expression with AggregateFunction
abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression
case class Min(
case class Average(child: Expression, distinct: Boolean = false)
case class Max(child: Expression, distinct: Boolean = false)
case class Count(child: Expression)
case class CountDistinct(children: Seq[Expression])
case class Sum(child: Expression, distinct: Boolean = false)
case class First(child: Expression, distinct: Boolean = false)
case class Last(child: Expression, distinct: Boolean = false)
sealed case class AggregateFunctionBind(
sealed class InputBufferSeens(
sealed trait Aggregate
sealed trait PreShuffle extends Aggregate
sealed trait PostShuffle extends Aggregate
case class AggregatePreShuffle(
case class AggregatePostShuffle(
case class DistinctAggregate(

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26013/
Test FAILed.

@marmbrus I've rebased to the latest master, and also updated the benchmark result, Sorry, the interface has slight different than the design doc in jira, I will update that soon, but the general idea would be the same.

Test build #26014 has started for   PR 3247 at commit 3dc1572.

This patch merges cleanly.

Test build #26014 has finished for   PR 3247 at commit 3dc1572.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
  case class Movie(movieId: Int, title: String, genres: Seq[String])
  case class Params(
class ALS extends Estimator[ALSModel] with ALSParams
 case class RatingBlock(srcIds: Array[Int], dstIds: Array[Int], ratings: Array[Float])
case class UnresolvedFunction(
trait AggregateFunction
trait AggregateExpression extends Expression with AggregateFunction
abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression
case class Min(
case class Average(child: Expression, distinct: Boolean = false)
case class Max(child: Expression, distinct: Boolean = false)
case class Count(child: Expression)
case class CountDistinct(children: Seq[Expression])
case class Sum(child: Expression, distinct: Boolean = false)
case class First(child: Expression, distinct: Boolean = false)
case class Last(child: Expression, distinct: Boolean = false)
sealed case class AggregateFunctionBind(
sealed class InputBufferSeens(
sealed trait Aggregate
sealed trait PreShuffle extends Aggregate
sealed trait PostShuffle extends Aggregate
case class AggregatePreShuffle(
case class AggregatePostShuffle(
case class DistinctAggregate(

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26014/
Test FAILed.

Test build #26076 has started for   PR 3247 at commit d251c8f.

This patch merges cleanly.

Test build #26076 has finished for   PR 3247 at commit d251c8f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class UnresolvedFunction(
trait AggregateFunction
trait AggregateExpression extends Expression with AggregateFunction
abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression
case class Min(
case class Average(child: Expression, distinct: Boolean = false)
case class Max(child: Expression, distinct: Boolean = false)
case class Count(child: Expression)
case class CountDistinct(children: Seq[Expression])
case class Sum(child: Expression, distinct: Boolean = false)
case class First(child: Expression, distinct: Boolean = false)
case class Last(child: Expression, distinct: Boolean = false)
sealed case class AggregateFunctionBind(
sealed class InputBufferSeens(
sealed trait Aggregate
sealed trait PreShuffle extends Aggregate
sealed trait PostShuffle extends Aggregate
case class AggregatePreShuffle(
case class AggregatePostShuffle(
case class DistinctAggregate(

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26076/
Test PASSed.

Can you rebase the patch? I can't merge it into master.

Sorry, @maropu I'v updated. Let's see if will break anything.

Test build #29286 has started for   PR 3247 at commit 0915435.

This patch merges cleanly.

Test build #29286 has finished for   PR 3247 at commit 0915435.

This patch fails Scala style tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

case class UnresolvedFunction(

trait AggregateFunction

trait AggregateExpression extends Expression with AggregateFunction

abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression

case class Min(

case class Average(child: Expression, distinct: Boolean = false)

case class Max(child: Expression)

case class Count(child: Expression)

case class CountDistinct(children: Seq[Expression])

case class Sum(child: Expression, distinct: Boolean = false)

case class First(child: Expression, distinct: Boolean = false)

case class Last(child: Expression, distinct: Boolean = false)

//case class Add(left: Expression, right: Expression) extends BinaryArithmetic

sealed case class AggregateFunctionBind(

sealed class InputBufferSeens(

sealed trait Aggregate

sealed trait PreShuffle extends Aggregate

sealed trait PostShuffle extends Aggregate

case class AggregatePreShuffle(

case class AggregatePostShuffle(

case class DistinctAggregate(

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29286/
Test FAILed.

Test build #29287 has started for   PR 3247 at commit 70117c4.

This patch merges cleanly.

Test build #29287 has finished for   PR 3247 at commit 70117c4.

This patch fails Scala style tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

case class UnresolvedFunction(

trait AggregateFunction

trait AggregateExpression extends Expression with AggregateFunction

abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression

case class Min(

case class Average(child: Expression, distinct: Boolean = false)

case class Max(child: Expression)

case class Count(child: Expression)

case class CountDistinct(children: Seq[Expression])

case class Sum(child: Expression, distinct: Boolean = false)

case class First(child: Expression, distinct: Boolean = false)

case class Last(child: Expression, distinct: Boolean = false)

//case class Add(left: Expression, right: Expression) extends BinaryArithmetic

sealed case class AggregateFunctionBind(

sealed class InputBufferSeens(

sealed trait Aggregate

sealed trait PreShuffle extends Aggregate

sealed trait PostShuffle extends Aggregate

case class AggregatePreShuffle(

case class AggregatePostShuffle(

case class DistinctAggregate(

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29287/
Test FAILed.

Test build #29289 has started for   PR 3247 at commit 40fed21.

This patch merges cleanly.

Test build #29289 has finished for   PR 3247 at commit 40fed21.

This patch fails Spark unit tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

case class UnresolvedFunction(

trait AggregateFunction

trait AggregateExpression extends Expression with AggregateFunction

abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression

case class Min(

case class Average(child: Expression, distinct: Boolean = false)

case class Max(child: Expression)

case class Count(child: Expression)

case class CountDistinct(children: Seq[Expression])

case class Sum(child: Expression, distinct: Boolean = false)

case class First(child: Expression, distinct: Boolean = false)

case class Last(child: Expression, distinct: Boolean = false)

sealed case class AggregateFunctionBind(

sealed class InputBufferSeens(

sealed trait Aggregate

sealed trait PreShuffle extends Aggregate

sealed trait PostShuffle extends Aggregate

case class AggregatePreShuffle(

case class AggregatePostShuffle(

case class DistinctAggregate(

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29289/
Test FAILed.

Test build #29291 has started for   PR 3247 at commit 341e708.

This patch merges cleanly.

Test build #29291 has finished for   PR 3247 at commit 341e708.

This patch fails Spark unit tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

case class UnresolvedFunction(

trait AggregateFunction

trait AggregateExpression extends Expression with AggregateFunction

abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression

case class Min(

case class Average(child: Expression, distinct: Boolean = false)

case class Max(child: Expression)

case class Count(child: Expression)

case class CountDistinct(children: Seq[Expression])

case class Sum(child: Expression, distinct: Boolean = false)

case class First(child: Expression, distinct: Boolean = false)

case class Last(child: Expression, distinct: Boolean = false)

sealed case class AggregateFunctionBind(

sealed class InputBufferSeens(

sealed trait Aggregate

sealed trait PreShuffle extends Aggregate

sealed trait PostShuffle extends Aggregate

case class AggregatePreShuffle(

case class AggregatePostShuffle(

case class DistinctAggregate(

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29291/
Test FAILed.

Test build #29310 has started for   PR 3247 at commit b539baf.

This patch merges cleanly.

Test build #29310 has finished for   PR 3247 at commit b539baf.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

case class UnresolvedFunction(

trait AggregateFunction

trait AggregateExpression extends Expression with AggregateFunction

abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression

case class Min(

case class Average(child: Expression, distinct: Boolean = false)

case class Max(child: Expression)

case class Count(child: Expression)

case class CountDistinct(children: Seq[Expression])

case class Sum(child: Expression, distinct: Boolean = false)

case class First(child: Expression, distinct: Boolean = false)

case class Last(child: Expression, distinct: Boolean = false)

sealed case class AggregateFunctionBind(

sealed class InputBufferSeens(

sealed trait Aggregate

sealed trait PreShuffle extends Aggregate

sealed trait PostShuffle extends Aggregate

case class AggregatePreShuffle(

case class AggregatePostShuffle(

case class DistinctAggregate(

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29310/
Test PASSed.

Test build #29457 has started for   PR 3247 at commit 13f4f15.

Test build #29457 has finished for   PR 3247 at commit 13f4f15.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

case class UnresolvedFunction(

trait AggregateFunction

trait AggregateExpression extends Expression with AggregateFunction

abstract class UnaryAggregateExpression extends UnaryExpression with AggregateExpression

case class Min(

case class Average(child: Expression, distinct: Boolean = false)

case class Max(child: Expression)

case class Count(child: Expression)

case class CountDistinct(children: Seq[Expression])

case class Sum(child: Expression, distinct: Boolean = false)

case class First(child: Expression, distinct: Boolean = false)

case class Last(child: Expression, distinct: Boolean = false)

sealed case class AggregateFunctionBind(

sealed class InputBufferSeens(

sealed trait Aggregate

sealed trait PreShuffle extends Aggregate

sealed trait PostShuffle extends Aggregate

case class AggregatePreShuffle(

case class AggregatePostShuffle(

case class DistinctAggregate(

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29457/
Test PASSed.

@chenghao-intel I'm also with your refactoring idea though, it's too big to merge into the master in bulk.
ISTM this patch is better to split into some small ones.
e.g.)

a patch to remove DISTINCT aggregator expression
a patch to simplify buffering codes in Aggregate
a patch to improve hive integration (Mode, or something)
...

Thought?
Anyway, I'm interested in your UDTF and UDAF refactoring activities, and so
I'd like to join that.

@maropu Glad to know you're interested with the refactoring!

But, it's probably very difficult to merge them into master piece by piece, as those tasks have many dependencies with each other, and we also have the bottom line that we can't break anything of master tests.
@maropu I will be glad to merge/review your PRs if you make the change against my repo, or @marmbrus @rxin is it possible to create a branch for this PR in apache repo?

Is it not possible to create that simple patch that removes DISTINCT aggregation expressions?
We only add distinct as a field value in AggregateExpresion, and then
SUM/COUNT switches  an aggregation strategy, DISTNCT or not, by using the current AggregationFunction implementations.

OK, I got your mean, as I put into the description of this PR, we want to make a unified UDAF interface in this PR, DISTINCT is quite critical for this purpose, we don't want to change the API definition again and again, right?

This PR is superseded by another PR and can be close, right?

Yea, I will close this for #5542


Re-registration of the point cloud mesh DM5 clone VFB_00020020

Description of the problem
The point cloud of DM5 clone VFB_00020020 is not registered within the brain close to the medulla and anterior ventrolateral protocerebrum:
vfb link

I wonder if it can be re-registered? Thanks!
This form is for VFB2 bug reports and feature requests.
If you need help, please use the forum.
Describe the bug or feature request in detail.
Always include a code snippet, screenshots, and any relevant data to help us understand your issue.
Browser

[x] All of them
[ ] Chrome
[ ] Firefox
[ ] Internet Explorer

OS

[x] All of them
[ ] Windows
[ ] macOS
[ ] Linux
[ ] Android
[ ] iOS

Sorry for the delay was on holiday, I'll take a look at these ASAP.


Docker image wont start

When using the docker-compose.yml file, on up I see.
$ docker-compose up
Recreating docker_mongo_1...
Recreating docker_app_1...
Attaching to docker_mongo_1, docker_app_1
mongo_1 | 2015-10-12T19:54:45.724+0000 I JOURNAL  [initandlisten] journal dir=/data/db/journal
mongo_1 | 2015-10-12T19:54:45.724+0000 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
app_1   |
app_1   | ><EMAIL_ADDRESS>prestart /usr/src/app
app_1   | > migroose
app_1   |
mongo_1 | 2015-10-12T19:55:02.885+0000 I JOURNAL  [initandlisten] preallocateIsFaster=true 74
app_1   |
app_1   | /usr/src/app/node_modules/mongodb/lib/server.js:228
app_1   |         process.nextTick(function() { throw err; })
app_1   |                                             ^
app_1   | Error: connect ECONNREFUSED
app_1   |     at exports._errnoException (util.js:746:11)
app_1   |     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1010:19)
app_1   |
app_1   | npm ERR! Linux 3.16.0-4-amd64
app_1   | npm ERR! argv "node" "/usr/local/bin/npm" "start"
app_1   | npm ERR! node v0.12.7
app_1   | npm ERR! npm  v2.14.1
app_1   | npm ERR! code ELIFECYCLE
app_1   | npm ERR<EMAIL_ADDRESS>prestart: `migroose`
app_1   | npm ERR! Exit status 1
app_1   | npm ERR!
app_1   | npm ERR! Failed at the<EMAIL_ADDRESS>prestart script 'migroose'.
app_1   | npm ERR! This is most likely a problem with the lets-chat package,
app_1   | npm ERR! not with npm itself.
app_1   | npm ERR! Tell the author that this fails on your system:
app_1   | npm ERR!     migroose
app_1   | npm ERR! You can get their info via:
app_1   | npm ERR!     npm owner ls lets-chat
app_1   | npm ERR! There is likely additional logging output above.
app_1   | npm ERR! Linux 3.16.0-4-amd64
app_1   | npm ERR! argv "node" "/usr/local/bin/npm" "start"
app_1   | npm ERR! node v0.12.7
app_1   | npm ERR! npm  v2.14.1
app_1   | npm ERR! path npm-debug.log.7da2e9feb6af80ea1c45d89105a542fd
app_1   | npm ERR! code EACCES
app_1   | npm ERR! errno -13
app_1   |
app_1   | npm ERR! Error: EACCES, open 'npm-debug.log.7da2e9feb6af80ea1c45d89105a542fd'
app_1   | npm ERR!     at Error (native)
app_1   | npm ERR!  { [Error: EACCES, open 'npm-debug.log.7da2e9feb6af80ea1c45d89105a542fd']
app_1   | npm ERR!   errno: -13,
app_1   | npm ERR!   code: 'EACCES',
app_1   | npm ERR!   path: 'npm-debug.log.7da2e9feb6af80ea1c45d89105a542fd' }
app_1   | npm ERR!
app_1   | npm ERR! Please try running this command again as root/Administrator.
app_1   |
app_1   | npm ERR! Please include the following file with any support request:
app_1   | npm ERR!     /usr/src/app/npm-debug.log
docker_app_1 exited with code 1
Gracefully stopping... (press Ctrl+C again to force)
Stopping docker_mongo_1...

This means it's not able to connect to your Mongo server.

I had the same problem, it looks to be a race condition whereby docker-compose doesn't wait for a dependant container to fully start before starting the parent.  So, specifically, docker-compose is starting the docker_mongo_1 container but it hasn't started listening for connections before the docker_app_1 containers starts and tries to connect to it.  docker_app_1 tries once, and if that fails (connect ECONNREFUSED), it bombs out.
The problem is detailed in more depth, along with potential solutions at the docker website
I think the best solution would be for the sdelements/lets-chat image to be modified to include an ability to either wait for MongoDB to be listening on the target port or retry the connection a few times, rather than bailing out after the first attempt.  Perhaps something as simple as wait-for-it.sh might do the trick.
I fuddled around it in the short term by removing my lets-chat container before starting the rest (docker-compose rm app; docker-compose up) - and usually the delay in building the container was long enough for MongoDB to get started.


Switch Persistence [2.5/4]: Establish Switch-Sphinx-Wallet APIs for Switch Persistence

This PR melds #775 and #777, by exposing the primary APIs needed to facilitate the upcoming additions to make the switch persistent. The changes are exposed in a way such that nearly all of the code is dead, but can still be built, compiled and tested using our existing test suite.
My thinking is that we can use this as a meeting point off which to base the remaining changes internal to the switch.

Oops thanks @Roasbeef, I did forget to update the docs. Fixing now

@Roasbeef updated the godocs, PTAL


about code

Hi,
Thanks for your wonderful work, I wonder whether this repo is the official code of
“Multivariate Time-series Anomaly Detection via Graph Attention Network“

No, it's my implementation. Hope you find it useful.
On Mon, Apr 22, 2024 at 2:59 AM remu～ @.***> wrote:

Hi,
Thanks for your wonderful work, I wonder whether this repo is the official
code of
“Multivariate Time-series Anomaly Detection via Graph Attention Network“
—
Reply to this email directly, view it on GitHub
https://github.com/mangushev/mtad-gat/issues/8, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKEIYQEVCK2LMPH3WHAMHRTY6S7NVAVCNFSM6AAAAABGSIDM7CVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2TKOBVG44DEMI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

No, it's my implementation. Hope you find it useful.
…
On Mon, Apr 22, 2024 at 2:59 AM remu～ @.> wrote: Hi, Thanks for your wonderful work, I wonder whether this repo is the official code of “Multivariate Time-series Anomaly Detection via Graph Attention Network“ — Reply to this email directly, view it on GitHub <#8>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKEIYQEVCK2LMPH3WHAMHRTY6S7NVAVCNFSM6AAAAABGSIDM7CVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2TKOBVG44DEMI . You are receiving this because you are subscribed to this thread.Message ID: @.>

I see, thanks

have a fun. experiment with it. You will have to play with false positives
and false negatives by changing thresholds . There is another project in my
list of projects to automatically calculate "initial" threshold values.
On Wed, Apr 24, 2024 at 9:16 PM remu～ @.***> wrote:

No, it's my implementation. Hope you find it useful.
… <#m_4675979528794331068_>
On Mon, Apr 22, 2024 at 2:59 AM remu～ @.> wrote: Hi, Thanks for your
wonderful work, I wonder whether this repo is the official code of
“Multivariate Time-series Anomaly Detection via Graph Attention Network“ —
Reply to this email directly, view it on GitHub <#8
https://github.com/mangushev/mtad-gat/issues/8>, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKEIYQEVCK2LMPH3WHAMHRTY6S7NVAVCNFSM6AAAAABGSIDM7CVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2TKOBVG44DEMI
https://github.com/notifications/unsubscribe-auth/AKEIYQEVCK2LMPH3WHAMHRTY6S7NVAVCNFSM6AAAAABGSIDM7CVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI2TKOBVG44DEMI
. You are receiving this because you are subscribed to this thread.Message
ID: @.>
I see, thanks
—
Reply to this email directly, view it on GitHub
https://github.com/mangushev/mtad-gat/issues/8#issuecomment-2076198871,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKEIYQASTKIXKXGJSXPKXC3Y7BROFAVCNFSM6AAAAABGSIDM7CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZWGE4TQOBXGE
.
You are receiving this because you commented.Message ID:
@.***>


Add gzip compression

About this change - What it does
This is a currently unfinished draft implementation of GZip compression
Resolves: https://github.com/aiven/guardian-for-apache-kafka/issues/195
Why this way
What has been done

Adjustments to the Source/Flow/Sink so that streams are compressed if the configuration for compression is enabled
Cli option for compression which also adds a subcommand called gzip to enable compression
Key creation with the json.gz extension if compression configuration is enabled

What still needs to be done

As described in the linked issue, handle the case where someone restarts Guardian with compression setting changed and hence having to suspend applying the change in the compression setting until the current key/object is completed.

i.e. if Guardian is currently backing up a file with the key 0.json without compression and then Guardian is restarted with compression setting enabled, we still have to finish 0.json without any compression and only start compressing for the next key which would be 1.json.gz
This means we also need to detect if there is any currently running backup by ignoring the file extension since this is the first time that Guardian is supporting multiple extensions. Specifically when someone starts Guardian and it goes through the process of seeing if there is any current uncompleted backups, that process needs to lookup keys without the extension. I believe S3's prefix is designed for this specific usecase

Confirm that changing compression levels between restarts of Guardian has no impact
Restore functionality which needs to detect if a file/object is compressed (i.e. filename/key has the .gz extension) and then decompress before doing the current restore flow.
Testing all of the above cases
Documentation

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

26 of 38 (68.42%)  changed or added relevant lines in 1 file are covered.
2 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-1.3%) to 74.419%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

core-backup/src/main/scala/io/aiven/guardian/kafka/backup/BackupClientInterface.scala
26
38
68.42%

Files with Coverage Reduction
New Missed Lines
%

core-restore/src/main/scala/io/aiven/guardian/kafka/restore/KafkaProducer.scala
1
72.73%

core-s3/src/main/scala/io/aiven/guardian/kafka/s3/Config.scala
1
90.63%

Totals

Change from base Build<PHONE_NUMBER>:
-1.3%

Covered Lines:
288

Relevant Lines:
387

💛  - Coveralls

So I have just updated the PR with a lot more work done, basically there is a full implementation of GZip resolving the issues in the PR description as well as some added tests to see if it works.
Unfortunately I encountered an issue in upstream akka, the unsafeDataVia function I added to akka-streams appears to be having issues with flows that cannot have truncation and gzip compression happens to be one of those flows. Upstream issue is at https://github.com/akka/akka/issues/31522

So the gzip compression PR is finally complete, originally PR post has been updated with the details.

Note that the tests have passed previously, now running tests is due to rebase of https://github.com/aiven/guardian-for-apache-kafka/pull/265 (see https://github.com/aiven/guardian-for-apache-kafka/actions/runs/3142309373)

Definitely pushes my limits of Akka Streams, second pair of eyes would be helpful, but I think it looks good overall. My personal observation - compression seems to be creeping into every place, may be a bit challenging to add different types (fe zstd, ...), in any case - good one @mdedetrich

My personal observation - compression seems to be creeping into every place, may be a bit challenging to add different types (fe zstd, ...), in any case - good one @mdedetrich

Thanks! The abstractions I added in my PR (one of the reasons why it happens to be a bit bigger than normal) would make adding compressions later on easier however there definitely needs to be some code organization done, i.e. abstracting out the compression in its own classes/traits rather than putting it back into BackupClientInterface.


Terraform version 1.0 support

https://www.hashicorp.com/blog/announcing-hashicorp-terraform-1-0-general-availability
terraform version 1.0 has been announced. Would be great if this project could support it. I should be available next month to contribute changes so creating and issue here to centralize info related and stuff that should be done.

Could you please give some more details here? At least the plan was generated with Terraform 1.0

For some reason CI jobs seems also failing for 1.x


Is Tab Butler still maintained?

The last modification was more than six months ago, is this extension still maintained?

Hey @Terkyz Yes, we are still actively working on Tab Butler. We are hoping to fully release v1 in the coming weeks with new features and updated UI!


Changing layout

Hey :)
I've got a problem with layout of wallet.
I have a little problem when i click this right bar

the options apper like this

but is put to close when your not with cursor on that ...  and I have to click it again,  :'( and be very carrefull with my mouse :)

Yes that is normal functionality of the wallet, if you don't like having to use that menu, then simply expand the wallet by resizing it or hitting maximize.


TS-4096: ts.now() should return subsecond info for ts_lua plugin

@bgaff and @zwoop , pls review

@jpeach , can you take a quick look on this as well?

This still returns time in seconds, since the arithmetic is the same. If you change the time units should you add a new API to preserve compatibility?

@jpeach
Yes. the time is still in seconds. But instead of returning an integer all the time, it is now a "lua_Number", which is like a "double".
In some sense, the data type returned is still the same ("lua_Number"). The unit is also not changed.
should I still add a new API instead? since all the operations on the return value will still work the same way.

I guess I still don't get it. This API already returned a Lua number, so the only semantic change I can see is that there is no longer any potential issue with truncation due to time_t. That doesn't sound like the issue you are describing and I don't see anything about sub-second timestamps here?

@jpeach
Before the change the lua number is always an integer (e.g.<PHONE_NUMBER>). Now after the change the lua number will contain the fraction of the second as well. (e.g.<PHONE_NUMBER>.123)

Oh I see, that makes sense. +1

ha. my bad english


RestBinding to non-existing url should not return 500 but 404

Describe the bug
When creating a custom operation for a single record (like Patient/), the RestBinding will throw a 500 if a client requests a record that does not exist.
To Reproduce

Create a web or function pipeline with a binding that points to a FHIR server (+ an authenticator)
Send a request to the FHIR server through the pipeline for a record that doesn't exist (like Patient/123).

Expected behavior
There should at least be an option to not fail on HTTP status codes between 202 and 499 but to pass this through. Especially for 300-400. I don't think "EnsureStatusSuccessful in the RestBinding is correct
Actual behavior

Agree. Removing "EnsureStatusSuccessful" from RestBinding would simply return response what to do with the non-200 status code, which should be the behavior.

As Suggested we have removed the "EnsureStatusSuccessful." So if patient is not there it will return 404 or if patient is deleted then 410.

@vmsuthar56 - is this live? 😄 If so can you close the issue?


Add option to supply "url" to static adaptor

Is your feature request related to a problem?
yes.
When using qwik add to add 'static html" option, one needs to know to change the following:
plugins: [
      staticAdaptor({
        origin: 'https://yoursite.qwik.dev',
      }),
    ],

To make all the "meta" links work
Describe the solution you'd like
Add an option to the CLI that asks the developer about the url of the website
Describe alternatives you've considered
adding it to the output log after generating (I'll do that anyway as it's an easier fix for now)
Additional context
No response

Going to hold off on this for now since we're instead printing out the message. If it continues to be a problem we can address this again.


Can't download punkt anymore

Starting this morning, nltk.download('punkt') results in this error:
  File "/Users/seandodd/code/Sites-API/env/lib/python3.5/site-packages/nltk/downloader.py", line 664, in download
    for msg in self.incr_download(info_or_id, download_dir, force):
  File "/Users/seandodd/code/Sites-API/env/lib/python3.5/site-packages/nltk/downloader.py", line 534, in incr_download
    try: info = self._info_or_id(info_or_id)
  File "/Users/seandodd/code/Sites-API/env/lib/python3.5/site-packages/nltk/downloader.py", line 508, in _info_or_id
    return self.info(info_or_id)
  File "/Users/seandodd/code/Sites-API/env/lib/python3.5/site-packages/nltk/downloader.py", line 875, in info
    self._update_index()
  File "/Users/seandodd/code/Sites-API/env/lib/python3.5/site-packages/nltk/downloader.py", line 825, in _update_index
    ElementTree.parse(compat.urlopen(self._url)).getroot())
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/xml/etree/ElementTree.py", line 1184, in parse
    tree.parse(source, parser)
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/xml/etree/ElementTree.py", line 596, in parse
    self._root = parser._parse_whole(source)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 23, column 143

Sorry about it code was broke on the nltk_data side. nltk/nltk_data#70 patched it.

Thanks


Fix markAsWebTransaction check

I'm using this class as a basis for a middleware for https://github.com/Baldinof/roadrunner-bundle.
I guess you meant method_exists instead of class_exists.

Fixed in #8


API key errors in api_search

Currently using data_gathering.api_search with a bad API key does not alert the user that the Planet API has returned an error message (a 401 response).
Instead, a user would not encounter an error until trying to extract geometry features from the returned json:
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[10], line 6
      4 print(domain_geometry)
      5 result = dg.api_search(item_type, apiKey)
----> 6 geojson_data, gdf = dg.downloadable_PlanetIDs(result, domain_geometry)
      7 id_list = dg.id_gemoetry_lists(geojson_data, gdf)
      8 order_urls = dg.submit_orders(id_list, item_type, bundle_type, apiKey)

File ~/git/PyPlanetSCA-Python-Library/src/planetsca/data_gathering.py:175, in downloadable_PlanetIDs(result, domain_geometry)
    173 def downloadable_PlanetIDs(result, domain_geometry):
    174     geojson_data = result.json()
--> 175     gdf = gpd.GeoDataFrame.from_features(geojson_data["features"])
    177     # Add a new column to 'gdf' with the intersection area
    178     gdf["intersection_area"] = gdf["geometry"].intersection(domain_geometry).area

KeyError: 'features'

We should add a feature to the api_search function so that it will alert the user if it returns anything other than a 200 response.

For example, this would print out the json response (and any error messages from the planet API) if the response code isn't OK:
if result.status_code != 200:
    print("Planet API response not OK: ", result.json())

Make two error messages to catch no API and incorrect API

Just finished this and merged into main. There's two catches before the main code is run for if there is no API or the API is not a string and using an invalid API.


Add Debugging Guide

People often ask about how to debug SAM templates. Add a DEBUGGING guide them through

What are you looking for?
I went through #20 and found your reference to aws cloudformation create-change-set and I plan on adding an npm test command to help validate changes.


Migration under VB.Net

Hy at all developer
I use the EF7 RC1 Final. Migration under VB is not possible.
I have created a new project and added the NuGet packages SQLServer and Command.
I added the classes like the wiki article and make an initial.migration and make update-database over the PakageManagerConsole. In thy VB Project was created *.cs files.
Then i have added the propertie "Comment" to the Post Class and do a second migration but it created again a complete migration like the initialmigration. AND after a update-database in my DB ist not a column with name "Comment".
Sorry for my bad english, my mainlanguage is german.
MigrationTest.zip

OK, sorry. I have found this: https://github.com/aspnet/EntityFramework/issues/4064
Can deleted


ENH: Speed up set_bipolar_reference

Reference issue
Fixes #8718
What does this implement/fix?
This improves the performance of adding a larger number (>25) of bipolar channels as suggested by @jasmainak.
EDIT: A gist to measure the speed can be found here
Additional information
I encountered some difficulties with how to best organize the creation of the new channels with the new method and handling Info while staying conservative. I think a large part of the performance improvement comes from moving add_channels outside the loop as done with the last commit 29f609d2. Before, even with the new matrix-multiplication-method the performance was much smaller with n>25.
At the moment, the new bipolar-channels are always appended (like when drop_refs=False) and don't take the place of the anode as in the previous implementation. If that is important I could add a reorder_channels afterwards with bipolar channels taking place of the anodes again.
As the bipolar-channels are not taking the same place as the anodes, I had to change the test a bit for now to pass.
There was also an additional attribute of the UpdateChannelsMixin called _projector besides picks which seemed to be outdated too with changed channels in Info.
Was it on purpose, that the bipolar reference of epochs was created with the mean of the anode-epochs?
ref_data = data[..., ref_from, :].mean(-2, keepdims=True)

In the new implementation it is created with the anode-signal of each single-epoch but could also be changed to the epochs mean.
ref_inst._data = np.asarray([multiplier.dot(ep) for ep in inst._data])

I changed the location of the bipolar-channel from zero to the location of the cathode as suggested by @jasmainak . I wondered, why every other 'chs'-attribute should still be taken from anode. Couldn't it be the anode-location instead or all the other info-attributes taken from the cathode?
What do you think about these changes?

can you share some performance benchmarks? Say if I have 180 channels, how long does it take to compute bipolar with old and new implementation for a) raw b) epochs c) evoked

can you share some performance benchmarks? Say if I have 180 channels (so 179 anode-cathode pairs), how long does it take to compute bipolar with old and new implementation for a) raw b) epochs c) evoked

I uploaded a test-file for speed.
There is an error for epochs with this test-file using the old implementation which is now fixed. That is why I excluded the epochs from the test for now.
Raw and Evokeds alone should demonstrate the drastic improve in performance too:
Before fix:
Creation of 26 bipolar channels for Raw&Evoked took: 38.687 s
After fix:
Creation of 26 bipolar channels for Raw&Evoked took: 1.801 s
I set the number to 26 because (at least on my laptop) the computation time seems to increase exponentially above ~20. I don't really understand why, I assume it is because of some kind of overhead introduced by the add_channels-code when called multiple times in a row.
Can you reproduce this behaviour?

Excellent, thanks for the benchmarks! You can edit your original description and add it there so it is not lost amongst the comments. And we can update it when you make more optimizations. 180 was the number of channels in my dataset and how I discovered the problem. You can see the dramatic improvement also with 26 channels :)
I think at this point, you can also try to use snakeviz or line profiler to see which lines are the slowest.
I think we should aim for sub-second speed, referencing should be instantaneous!

A quick question:
I don't really understand, why electrode-information is taken from the anode, but location should be taken from the cathode.
Is this a convention?

it should not be.
the chs['loc'] should ideally contain the 2 locations with the 2nd location
being the reference used.

it should not be. the chs['loc'] should ideally contain the 2 locations with the 2nd location being the reference used.
…

Ok thank you, now all channel-info including location is copied from anode

In this gist I uploaded the protocols of profiling before and after the most recent improvements. The major part of the time was indeed still taken by the multiple call of "add_channels" and the creation of an instance for each new bipolar-channel.
Now, as suggested, only one instance for the new rereferenced channels is created which is then added once to the original instance.
Performance before recent improvements:
Creation of 26 bipolar channels took:
Raw: 1.642 s
Epochs: 2.073 s
Evoked: 0.803 s

Creation of 180 bipolar channels took:
Raw: 11.286 s
Epochs: 14.424 s
Evoked: 5.513 s

Perfomance after recent improvements:
Creation of 26 bipolar channels took:
Raw: 0.165 s
Epochs: 0.275 s
Evoked: 0.054 s

Creation of 180 bipolar channels took:
Raw: 0.358 s
Epochs: 0.633 s
Evoked: 0.055 s

@marsipu what is the reason for [ci skip] here?  The code changes are substantial, they should be tested.  Can you git commit --amend to remove the [ci skip] from your last commit, and then git push --force?  Or alternatively, git commit --allow-empty -m "trigger CIs" and then git push.

@marsipu what is the reason for [ci skip] here? The code changes are substantial, they should be tested. Can you git commit --amend to remove the [ci skip] from your last commit, and then git push --force?

Oh I am sorry, I thought it was good practice to skip ci for minor changes to save resources for other PRs. But I see now, I probably should have had at least documentation ci again for cb2e3d0. There was ci for all major changes before. Is the ci result lost for every new push made?
I will trigger ci again as you instructed.

When is it allowed/advisable to skip any ci?

It is generally speaking not avisable to skip CIs.  The one exception that I'm willing to put in writing is this: if all you changed is documentation (tutorial, example, and/or docstring), then [skip azp][skip github] is acceptable.
The reason for this is that the CIs can catch problems unrelated to your changes, that cropped up due to new versions of our dependencies being released.  The sooner we catch those, the sooner we can fix them before real users encounter them.  Two such examples came up just yesterday: #9321 and https://github.com/pydata/pydata-sphinx-theme/pull/395.
The other case where you might do [ci skip] is if your PR is a draft PR and you know that the commits you're pushing aren't yet enough to fix the bug / implement the new feature, and you don't want to waste energy by running the CIs on something you already know will fail.

It is generally speaking not avisable to skip CIs. The one exception that I'm willing to put in writing is this: if all you changed is documentation (tutorial, example, and/or docstring), then [skip azp][skip github] is acceptable.

Thank you for clarification, I will respect that from now on.

There seems to be an error for test_report.py::test_scraper, which doesn't seem to be related to this PR, right?
I could maybe fix this with exist_ok=True in the line throwing the error, but I am not sure if this defies the purpose of the test.

I've opened #9332 to deal with the failing scraper tests.

Close-reopen cycle to restart all CIs

Thanks @marsipu !


Setting parameter overwrite to false results in the undocumented exception: System.ArgumentException: BlockBlobClient.OpenWrite only supports overwriting

Type of issue
Missing information
Description
I assume this is really a bug in the BlockBlockClient OpenWriteAsync function. The overwrite parameter is a boolean.  So it should accept true or false.
Using Azure.Storage.Blobs nuget package v12.19.1.
Page URL
https://learn.microsoft.com/en-us/dotnet/api/azure.storage.blobs.blobclient.openwriteasync?view=azure-dotnet
Content source URL
https://github.com/Azure/azure-docs-sdk-dotnet/blob/master/xml/Azure.Storage.Blobs/BlobClient.xml
Document Version Independent Id
44dca6af-43a8-616c-1e76-7b6dcb1d7da4
Article author
@azure-sdk
Metadata

ID: 3218cb5f-ac7d-e212-5cb8-717ed6ea95b5
Service: azure

Seems identical to #24831 - an issue from 2021. Maybe xmldoc of the OpenWrite methods should be improved to highlight that passing false is not supported.


Switch to imgui docking branch, enable docking features and hopefully magic viewports too

At least enable docking and throw away the old stacked/tabbed interfaces.
Ideally, if someone can get it working, enable the magic viewports too.

Did it in just under a year in #233.


Add a dynamically-generated CHANGELOG.md

See https://github.com/conventional-changelog/standard-version.

Done. See 6cb16e1.


Minor corrections on ojects doc

Closing, but feel free to re-submit with more details.  As it is, though, I think the first change is incorrect for the way the code is currently structured, and the second change is unnecessary (both forms are valid PHP syntax).


Project banner module

fix #3049

First version of banner generated from project name (truncated to 17chars if too long).
It could eventually be improved:

by taking content from a dedicated module parameter.

by building the banner from another acsii generator : for example using this API or this maven plugin or any standard library (if found).

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Matthieu BONNOT seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

opened for too long, let's close this and discuss if needed


undefined is not an object (evaluating 't.decodedData[-260].v[0]')

The following error message is displayed in a red rectangle on the bottom of the page when I click Add to Wallet.
undefined is not an object (evaluating 't.decodedData[-260].v[0]')
I have the same error when using the camera and also when scanning the PDF.
It was working for vaccination QR codes (tested a couple of days ago).
It failed for the negative PCR-test result QR code (tested today, 08.07.21).
All QR codes I tested were officially generated by the authorities in Romania.
Tested on: https://covidpass.marvinsextro.de/

Which browser are you using? Safari?

You can't create wallet passes for test or recovery certificates at the moment. Does it work with vaccination certificates?

Which browser are you using? Safari?

Yes. In both cases: when it was working and when it failed.

You can't create wallet passes for test or recovery certificates at the moment. Does it work with vaccination certificates?

Yes. Thanks. Tomorrow we have a flight to Germany. Maybe PCR-test certificates support is available by then. :)

The corresponding issue for this is #5.

I will close this with reference to #5.

@valids You should now be able to add your test certificates. See #34

@valids You should now be able to add your test certificates. See #34

@marvinsxtr Thanks a lot. It works. Guys, did you sleep? I am leaving to DE now…


Adds Tint & Shade Generator to Color Picker Tools

This is just a link addition for https://maketintsandshades.com under Color Picker Tools.
Checklist

[x] I made something good today 💐.
[x] I put links in the correct format: [Tool](link) — description.
[x] My description starts with the lower-case letter and ends with the full stop.
[x] I put the tool in the alphabetical order.

Optional

[x] I added the appropriate labels: free, open-source or mac.
[x] I put the tool only to one category.
[x] I followed Contribution Guidelines.

Added! sorry for the delay in adding.


Adding IP address/network restriction in API Backends

Hi,
I was talking with our Chief Security Officer explaining the functionalities from API-Umbrella.
He was concerned about the API Backends restrictions and we explain that you currently can restrict a API Backend with a API user (API Key) by IP address. But in some scenarios you could've an API Backend that you wish to make it more secure.
So, what about adding the feature of the source IP address or network restriction in the API Backend?
Thanks

Apologies for the delay in response. If I'm understanding you correctly, do you want to restrict an entire API backend so that it's only accessible to certain client IP addresses? There are a couple of options to achieve that right now:

As you noted, we do currently support restricting specific API keys so they can only be used from specific IPs or IP ranges. You could use that in combination with role restrictions to enforce this for an entire API backend at the API Umbrella level. Although, that's sort of a roundabout way to achieve IP based restrictions for an entire API backend, so it might not be ideal.
We do pass the original IP address to the API backend via the X-Forwarded-For header. So if you wanted to implement this IP restriction logic in your API backend, that should be doable.

But I think there is probably a good use-case for IP restrictions at the API backend level to make this easier directly inside API Umbrella. When we recently implemented the IP restrictions for API keys, I think we actually hooked up most of the pieces necessary for easily allowing API backends to also have their own IP restrictions. I think the main thing we didn't do was simply expose that functionality in the web admin interface for API backends. So I think this should hopefully be pretty easily doable at some point in the near future.

We have heard this same requirement for IP whitelisting and blacklisting several times in customer cases. Are there any plans to implement the restriction based on IP?


Branch dev-<IP_ADDRESS>88 not found

dupe of https://github.com/sailro/EscapeFromTarkov-Trainer/issues/293


⚠ Bofrost.lu has degraded performance

In f3aacd7, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 1160 ms

Resolved: Bofrost.lu performance has improved in 6217afd after 20 minutes.


Upgrade golang to 1.17.6

Upgrade Golang to the latest 1.17.x (1.17.6) to address the following vulnerabilities

CVE-2021-41772
CVE-2021-41771

This fixes https://github.com/signalfx/splunk-otel-collector/issues/1063
Signed-off-by: Dani Louca<EMAIL_ADDRESS>

I think there are a few other references that could be updated:
https://github.com/signalfx/splunk-otel-collector/blob/main/.github/workflows/deps-test.yml#L15
https://github.com/signalfx/splunk-otel-collector/blob/main/.github/workflows/build-and-test.yml#L18
https://github.com/signalfx/splunk-otel-collector/blob/main/.github/workflows/windows-test.yml#L32
https://github.com/signalfx/splunk-otel-collector/blob/main/.github/workflows/integration-test.yml#L18

I think there are a few other references that could be updated: https://github.com/signalfx/splunk-otel-collector/blob/main/.github/workflows/deps-test.yml#L15 https://github.com/signalfx/splunk-otel-collector/blob/main/.github/workflows/build-and-test.yml#L18 https://github.com/signalfx/splunk-otel-collector/blob/main/.github/workflows/windows-test.yml#L32 https://github.com/signalfx/splunk-otel-collector/blob/main/.github/workflows/integration-test.yml#L18

Changed the gh actions and updated the PR
Thanks


tls: add missing stats for signature algorithms.

While there, refresh supported cipher suites and add more warnings.
Signed-off-by: Piotr Sikora<EMAIL_ADDRESS>

/backport because of ENVOY_BUG in #14534.

/backport because of ENVOY_BUG in #14534.


Update standard-for-success-accreditation-tutorial.md

I've made the necessary changes that I'm able to.  The image 'standard-for-success-accreditation-tutorial/configuration.png' is incorrect and I don't see a way to replace it.  I've attached the replacement image here.  Thanks

@kborrow : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Hi, just wondering if the changes were accepted.  I have a client waiting on this SSO setup, but don't want to open it up to them until the tutorial is corrected.
Thanks,
KATHY BORROW
Software Engineer
<EMAIL_ADDRESS>844-SFS-EVAL Ext. 713
standardforsuccess.comhttp://www.standardforsuccess.com/

From: PRMerger19 @.>
Sent: Thursday, May 27, 2021 8:58 AM
To: MicrosoftDocs/azure-docs @.>
Cc: Kathy Borrow @.>; Mention @.>
Subject: Re: [MicrosoftDocs/azure-docs] Update standard-for-success-accreditation-tutorial.md (#76008)
@kborrowhttps://github.com/kborrow : Thanks for your contribution! The author(s) have been notified to review your proposed change.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/pull/76008#issuecomment-849612568, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKGJEOEPX535JRECPHPZUDTPY6WHANCNFSM45UHO74A.

@jeevansd Will you please review the changes in this PR?
Thanks!

It has been nearly two weeks since I added minimal changes to the tutorial.  Can someone please look them over.  My client is becoming impatient, and rightfully so.
Thank you,
KATHY BORROW
Software Engineer
<EMAIL_ADDRESS>844-SFS-EVAL Ext. 713
standardforsuccess.comhttp://www.standardforsuccess.com/

From: Andrea Courtright @.>
Sent: Tuesday, June 1, 2021 4:17 PM
To: MicrosoftDocs/azure-docs @.>
Cc: Kathy Borrow @.>; Mention @.>
Subject: Re: [MicrosoftDocs/azure-docs] Update standard-for-success-accreditation-tutorial.md (#76008)
@jeevansdhttps://github.com/jeevansd Will you please review the changes in this PR?
Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/pull/76008#issuecomment-852417972, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKGJEKLFSSV3A5Z6PB7YXTTQU55PANCNFSM45UHO74A.

Hi @kborrow - Image files that are referenced from a Markdown file in the repo must exist in (be added to) the repo, and not be
hosted elsewhere. Are you able to add an image file by updating your working branch via a command-line tool like GitBash? (Unfortunately, images can't be added to a pull request by editing the article in GitHub.)
If not, we can move this commit to the private repository and create an image file from a screenshot of the file in GitHub Preview.

Hi Kristine,
I'm not sure i follow how/where to add it using GitBash. If you can provide some instruction for that ill be happy to try.
Thank you,
Kathy
Kathy Borrow

From: Kristine Toliver @.>
Sent: Tuesday, June 8, 2021 6:48:45 PM
To: MicrosoftDocs/azure-docs @.>
Cc: Kathy Borrow @.>; Mention @.>
Subject: Re: [MicrosoftDocs/azure-docs] Update standard-for-success-accreditation-tutorial.md (#76008)
Hi @kborrowhttps://github.com/kborrow - Image files that are referenced from a Markdown file in the repo must exist in (be added to) the repo, and not be
hosted elsewhere. Are you able to add an image file by updating your working branch via a command-line tool like GitBash? (Unfortunately, images can't be added to a pull request by editing the article in GitHub.)
If not, we can move this commit to the private repository and create an image file from a screenshot of the file in GitHub Preview.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/pull/76008#issuecomment-857236256, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKGJEIFEC7SC2YXD53F6ZDTR2M43ANCNFSM45UHO74A.

@kborrow The steps are described in the Microsoft Docs Contributor Guide:
https://docs.microsoft.com/en-us/contribute/get-started-setup-tools
and
https://docs.microsoft.com/en-us/contribute/get-started-setup-local
It's a fair bit of overhead, but it's a good things to know how to do. It's your choice; we can do it on our end.
Note that when you create the image file to add to the pull request, the file naming standard for the repository is to use only lowercase letters, numbers, and hyphens, and to separate words and numbers with hyphens. You might name your image configure-test-user.png and put it in the folder ./media/standard-for-success-accreditation-tutorial/, like this one that's already referenced in the file:

This is the location of that file in the GitHub repository: https://github.com/MicrosoftDocs/azure-docs/tree/master/articles/active-directory/saas-apps/media/standard-for-success-accreditation-tutorial

Hi Kristine,
Since this is more overhead than I'm hoping to install currently to replace a single image, could you please replace the existing image with the attached?  I did add the red borders around the input areas in keeping with the style of the other images in the document.  I would great appreciate it.
Thank you,
KATHY BORROW
Software Engineer
<EMAIL_ADDRESS>844-SFS-EVAL Ext. 713
standardforsuccess.comhttp://www.standardforsuccess.com/

From: Kristine Toliver @.>
Sent: Tuesday, June 8, 2021 7:49 PM
To: MicrosoftDocs/azure-docs @.>
Cc: Kathy Borrow @.>; Mention @.>
Subject: Re: [MicrosoftDocs/azure-docs] Update standard-for-success-accreditation-tutorial.md (#76008)
@kborrowhttps://github.com/kborrow The steps are described in the Microsoft Docs Contributor Guide:
https://docs.microsoft.com/en-us/contribute/get-started-setup-tools
and
https://docs.microsoft.com/en-us/contribute/get-started-setup-local
It's a fair bit of overhead, but it's a good things to know how to do. It's your choice; we can do it on our end.
Note that when you create the image file to add to the pull request, the file naming standard for the repository is to use only lowercase letters, numbers, and hyphens, and to separate words and numbers with hyphens. You might name your image configure-test-user.png and put it in the folder ./media/standard-for-success-accreditation-tutorial/, like this one that's already referenced in the file:
[image]https://user-images.githubusercontent.com/19416435/121271803-50f89b00-c879-11eb-9847-396b5aecc1fa.png
This is the location of that file in the GitHub repository: https://github.com/MicrosoftDocs/azure-docs/tree/master/articles/active-directory/saas-apps/media/standard-for-success-accreditation-tutorial
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/pull/76008#issuecomment-857266396, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKGJENUCVLUQLC5T7GKE7TTR2UATANCNFSM45UHO74A.

@ktoliver : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@kborrow Understood. I'd like to get this merged for the next publishing cycle, so I've removed the image reference for now. I expect to be able to add the image separately later today. I'll add a note here and at-mention you when I've done so. But we can get the content updates published meanwhile. Thanks!

Hi @kborrow - I updated the configuration.png image in the tutorial. Is it what you were anticipating?
https://docs.microsoft.com/azure/active-directory/saas-apps/standard-for-success-accreditation-tutorial
Thank you.

Yes, Kristine, it looks good. Thank you!
Kathy Borrow

From: Kristine Toliver @.>
Sent: Wednesday, June 9, 2021 8:42:51 PM
To: MicrosoftDocs/azure-docs @.>
Cc: Kathy Borrow @.>; Mention @.>
Subject: Re: [MicrosoftDocs/azure-docs] Update standard-for-success-accreditation-tutorial.md (#76008)
Hi @kborrowhttps://github.com/kborrow - I updated the configuration.png image in the tutorial. Is it what you were anticipating?
https://docs.microsoft.com/azure/active-directory/saas-apps/standard-for-success-accreditation-tutorial
Thank you.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/azure-docs/pull/76008#issuecomment-858194763, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKGJEOUNAWVY2B3SNZXHUDTSADAXANCNFSM45UHO74A.


ENH: improve error message when trying to fit an ICA on empty epochs

fixes https://github.com/mne-tools/mne-study-template/pull/41#issuecomment-518920458
cc @jasmainak
Previous error triggered in np.hstack:
ValueError: need at least one array to concatenate

New error message triggered earlier, in _fit_epochs:
RuntimeError: Tried to fit ICA with epochs, but none were found: epochs.events is "[]".

Excellent progress. As you can see, I’m still catching up :)
On Wed 14 Aug 2019 at 15:15, Eric Larson<EMAIL_ADDRESS>wrote:

Merged #6660 https://github.com/mne-tools/mne-python/pull/6660 into
master.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mne-tools/mne-python/pull/6660?email_source=notifications&email_token=ADY6FIX3KW2D24UHHILN7JLQERKWXA5CNFSM4ILUYGK2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOTBZHE4I#event-2557637233,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADY6FIVFKM5KXGZCF23OZJ3QERKWXANCNFSM4ILUYGKQ
.

--
Sent from my iPhone


CustomShaders: make FragmentInput struct more consistent for Model and VoxelPrimitive

Feature
A CustomShader for a VoxelPrimitive uses an undocumented form of the FragmentInput struct.
Comparison to the FragmentInput struct used for Model
The FragmentInput struct for voxels differs from the  FragmentInput struct used for a CustomShader attached to a Model in several key ways:
Metadata statistics
Statistics for voxel metadata are appended to the Metadata struct. For example, to access the minimum value of a metadata property, a user must use code like this:
float valMin = fragmentInput.metadata.statistics.${property}.min;

Since #10683, these statistics for Model have been moved to a top-level MetadataStatistics struct. We should also add a MetadataClass struct to contain the noData value, as was done for Model in #10520.
Attributes
Some standard attributes such as positionEC and normalMC (renamed as surfaceNormal) are hidden inside the voxel sub-struct. These should be moved to FragmentInput.attributes for consistency with Model.
Voxel-specific values
Some properties of the voxel sub-struct do not have direct parallels with Model. For example, travelDistance is useful for accurate opacity calculations. Others such as tileIndex and sampleIndex are conceptually similar to FeatureIds, but we may not want to use the same terms since FeatureId properties are aligned with specific glTF extensions that aren't mirrored in the 3DTILES_content_voxels extension.
What needs to change
Most of the required changes will be in processVoxelProperties.js and VoxelFS.glsl. Also the Custom Shader Guide will need to be updated for any properties that remain different for voxels.
These changes will break many existing voxel shaders. But the long-term benefits include:

Simplified documentation
Faster onboarding for new users of CesiumJS voxel rendering
Easier maintenance of voxel code and docs

To consolidate issues, I'll add a action item to update the Custom Shader Guide to this.

Some understandable confusion around this was reported in https://github.com/CesiumGS/cesium/issues/12124.


Consider option to control cache on ivalidation

Currently, the cache is just dropped on each update.
It's possible to make it configurable to stay available forever in case of problems.
Upd: it looks to me that I found the best way to implement it is using the builtins, a good example is in _perl_modules sources and following _store_cache, _retrieve_cache, and _cache_invalid functions

That would be nice, but as is I love this project! Thank you for making it.  Really seems like something that could be submitted upstream to zsh as well.
At the moment I'm using HOCO_FUNCTIONS to automatically add some hosts used by with my personal aliases from zsh history to host completion. It's pretty janky but it works great.  Thank you!

function __listRecentHosts() {
  gtail -n 20000 ~/.zsh_history | ggrep -E ";(certs|certOwner|dnsAll)\s+\w{2,}" | hck -d ";" -f 2 | hck -d "\s+" -f "2" | sort | guniq
}
HOCO_FUNCTIONS=(__listRecentHosts)
source ~/.config/hoco.zsh

You can try a new version from #4 before it's merged.


We never test the actual files in template/

Relating to #213 we never test the actual files in template/, we only test files from test_template/ . Although they should be identical where needed, it's easy to miss something, and suddenly they diverge. I don't like this pattern at all. Maybe do some fancy trickery with paths to make the running of test_example/ in the github actions use files from template/? Just spitballing here, not given it much thought yet.

Morph this into having several subrepos to use for testing (each fulfill and test its own usage)


Post-v1.0.0 literature

This thread records literature that needs to be added to the next version. This may include literature that was published prior to this version but that was not incorporated.

[x] Gee, B. M., & Sidor, C. A. (2024). Diminutive temnospondyls from the lower and middle Fremouw Formation (Lower Triassic) of Antarctica. Journal of Vertebrate Paleontology, e2407183. https://doi.org/10.1080/02724634.2024.2407183 [online 2024/11/11]
[x] Osterling Arias, A. F., Mooney, E. D., Bevitt, J. J., & Reisz, R. R. (2024). A new trematopid from the lower Permian of Oklahoma and new insights into the genus Acheloma. PloS one, 19(10), e0309393. https://doi.org/10.1371/journal.pone.0309393 [online 2024/10/17]
[x] Huttenlocker, A. K., Douglass, R., Lungmus, J. K., Oliver, K., Pardo, J. D., & Small, B. J. (2024). Report of a Diverse Vertebrate Body Fossil Assemblage in the Maroon Formation (Carboniferous–Permian), Eagle County, Colorado, USA. Annals of Carnegie Museum, 90(2), 139-160. https://doi.org/10.2992/007.090.0204 [online 2024/07/31]
[x] So, C., Kufner, A. M., Pardo, J. D., Edwards, C. L., Price, B. R., Bevitt, J. J., ... & Lovelace, D. M. (2024). Fossil amphibian offers insights into the interplay between monsoons and amphibian evolution in palaeoequatorial Late Triassic systems. Proceedings B, 291(2033), 20241041. https://doi.org/10.1098/rspb.2024.1041 [online 2024/10/30]

[ ] Mooney, E. D., Maho, T., Philp, R. P., Bevitt, J. J., & Reisz, R. R. (2024). Paleozoic cave system preserves oldest-known evidence of amniote skin. Current Biology, 34(2), 417-426. https://doi.org/10.1016/j.cub.2023.12.008

[ ] Rinehart, L.F. Lucas, S.G., & Heckert, A.B. (2024). LAMY AMPHIBIAN QUARRY, A LATE TRIASSIC METOPOSAUR-DOMINATED BONEBED IN NEW MEXICO. [ResearchGate]

Werneburg, R., & Schoch, R.R. (2024). A new large stereospondylomorph temnospondyl from the early Permian (Rotliegend, Lower Goldlauter Fm.) of the Thuringian Forest, Central Germany. Semana, 39, 29-53. [ResearchGate]

Prino, A., Witzmann, F., Schwermann, A. H., Sander, P. M., Garbay, L., & Konietzko-Meier, D. (2024). How not to disappear completely: new Stereospondyli fossils from the Rhaetian, Upper Triassic of Bonenburg, North Rhine-Westphalia and their implications for the Late Triassic extinction of Stereospondyli. Acta Palaeontologica Polonica, 69(4), 687-712.


Optimint stopped before successful block submission

After DALC error (see: celestiaorg/dalc#21) I had to stop my optimint aggregator. In such case, after the restart, block will not be stored in DA.
Aggregator should keep disk-persisted queue of blocks that should be stored in DA. This way it could recover from situation like described before. This could also be used to avoid to continue block creation in case of DALC errors/timeouts (see #200, #201).

Fixed by: #1568

Fixed by: https://github.com/rollkit/rollkit/pull/1568


video disabled due to region lock shows Transcript/subtitle disabled.

To Reproduce
Steps to reproduce the behavior:
What code / cli command are you executing?
A user tried extracting this video
https://www.youtube.com/watch?v=kZsVStYdmws
This video is available in only some regions (e.g. Hong Kong, Taiwan) but not for the others (e.g. United States).
Therefore, it works in local (Hong Kong) but after deployment (to a US server), it will shows
Subtitles are disabled for this video
This code can reproduce that, it worked if using VPN for HK region. Doesn't work for US
video_id = "kZsVStYdmws"
YouTubeTranscriptApi.list_transcripts(video_id)

Which Python version are you using?
Python 3.10.8
Which version of youtube-transcript-api are you using?
youtube-transcript-api 0.6.0
Expected behavior
Describe what you expected to happen.
I think it is okay that region which disabled the video cannot fetch transcript, but the exception is confusing that I troubleshot for a while to understand why it happened.
Potentially, it is because it entered raise TranscriptsDisabled part. Therefore maybe adding one more exception handling helps.
    def _extract_captions_json(self, html, video_id):
        splitted_html = html.split('"captions":')

        if len(splitted_html) <= 1:
            if video_id.startswith('http://') or video_id.startswith('https://'):
                raise InvalidVideoId(video_id)
            if 'class="g-recaptcha"' in html:
                raise TooManyRequests(video_id)
            if '"playabilityStatus":' not in html:
                raise VideoUnavailable(video_id)

          **Here, added exception**

            **raise TranscriptsDisabled**(video_id)

Actual behaviour
it will shows Subtitles are disabled for this video for disabled video region even the subtitle is enabled.

I will respect whether you fix it or not. Thanks for handling

Hi @michaelthwan,
thank you for reporting. I agree: this is not something we can do anything about, but a more descriptive error message would be nice. I am currently a bit short on time to implement this myself, but I will put it on the list and contributions will be very much welcome! 😊

Hi @michaelthwan,
thank you for reporting. I agree: this is not something we can do anything about, but a more descriptive error message would be nice. I am currently a bit short on time to implement this myself, but I will put it on the list and contributions will be very much welcome! 😊

@jdepoix  I finally had some down time, taking a look at this issue.
As far as what YouTube identifies this error as its still considered "Video unavailable" for the main reason, but has subreason text that displays The uploader has not made this video available in your country
In the browser, in place of the video not loading due to a region lock we get a black background with white text showing:

Video unavailable
The uploader has not made this video available in your country

In the HTML we end up with this to search against
"playabilityStatus":{"status":"UNPLAYABLE","reason":"Video unavailable","errorScreen":{"playerErrorMessageRenderer":{"subreason":{"runs":[{"text":"The uploader has not made this video available in your country"}]}

We could do a new error message class such as this? To keep it somewhat inline with whats in the response of YouTube.
# file: youtube_transcript_api/_errors.py

class VideoUnplayable(CouldNotRetrieveTranscript):
    CAUSE_MESSAGE = 'The video has not been made available in your country'

Though it would be another search for an exact string match against html such as
def _extract_captions_json(self, html, video_id):
    splitted_html = html.split('"captions":')

    if len(splitted_html) <= 1:
        if video_id.startswith('http://') or video_id.startswith('https://'):
            raise InvalidVideoId(video_id)
        if 'class="g-recaptcha"' in html:
            raise TooManyRequests(video_id)
        if '"playabilityStatus":' not in html:
            raise VideoUnavailable(video_id)

        # add something like this
        if 'The uploader has not made this video available in your country' in html:
            raise VideoUnplayable(video_id)

@jdepoix Interestingly enough we could also add an Age-related error class as well. Although it seems we could get around the age-related retrieval of a transcript since you can pull a transcript regardless if you are logged in or not. To do that would require adding logic around my findings in #110. But until we have that workaround implemented we could at least throw an appropriate error a very similar way as the country/region lock since the HTML to match on for that lives in the same spot and looks like this.
"playabilityStatus":{"status":"LOGIN_REQUIRED","reason":"Sign in to confirm your age","errorScreen":{"playerErrorMessageRenderer":{"subreason":{"runs":[{"text":"This video may be inappropriate for some users."}]}

This would let us also sign off #111  until a workaround is implemented.

Hi @crhowell, thanks for looking into this and sorry for the late reply!
It looks like the data in "playabilityStatus" could generally be useful to provide more helpful exceptions and error messages! We could add a exception type for each status (LoginRequired, VideoUnplayable) which render playabilityStatus.reason and playabilityStatus.errorScreen.playerErrorMessageRenderer.subreason.runs as part of the error message. However, just looking for a natural language string in the html definitely is too fragile, as this probably will be in a different language depending on the locale. But isn't this part of the json we are parsing in json.loads(splitted_html[1].split(',"videoDetails')[0].replace('\n', '')) anyways? In that case we could just check what the status is and throw the corresponding exception, while passing in the reason/subreason. If it is not part of the json we are currently parsing, I guess we should find a way to parse it, since everything else will be very fragile.

@jdepoix  Well its branched logic in there based on whether or not splitted_html has an index 1 or not.
Basically if we split the html html.split('"captions":') on captions. If that List has a length  less than or equal to 1. We will ALWAYS raise an exception and json.loads never runs.
Otherwise, that means if we have more than 1 index position our list, we do try to parse the 1st index position.
But in our case for these specific errors, from what ive inspected via debug breakpoint we do not have more than 1 index position so we would never hit the json.loads side of the branching, we always raise the exception which leaves us back with the fragile in html statement.

You guys are very helpful. Thank you @crhowell @jdepoix

Hi @crhowell, yeah, that makes sense, but this should be solvable 😊
if len(splitted_html) <= 1:
        if video_id.startswith('http://') or video_id.startswith('https://'):
            raise InvalidVideoId(video_id)
        if 'class="g-recaptcha"' in html:
            raise TooManyRequests(video_id)
        splitted_html = html.split('"playabilityStatus":')
        if len(splitted_html) <= 1:
            raise VideoUnavailable(video_id)

        playability_status_json = json.loads(
            splitted_html[1].split(',"WHAT_EVER_THE_NEXT_PROPERTY_IS')[0].replace('\n', '')
        )

        # ... handle playability_status_json ...

        # fallback if we don't know the status
        raise TranscriptsDisabled(video_id)

@jdepoix I can throw an initial pass PR together for this I have a partial solution already. Ill test it against Age/Region error cases as well as the valid working cases so we can see what kind of "reason" shows up when everything is working fine and transcripts are retrievable.
Ill tag you for review on it once submitted.

I'm experiencing the same problem with this library. My setup:

VPS hosted on Hetzner (German IP): Unable to retrieve transcripts
Local machine (Spain IP): Can download transcripts without any problem

This suggests that the issue is indeed related to region-based restrictions or how YouTube is responding to requests from different geographic locations. The current error message ("Subtitles are disabled for this video") is misleading and made troubleshooting difficult.
I'm looking forward to the implementation of a more descriptive error handling system that can differentiate between truly disabled subtitles and region-based restrictions. This would greatly improve the user experience and make it easier to diagnose and handle these issues in our applications.
In the meantime, is there a recommended workaround for handling region-locked videos? Would using a proxy or VPN be a viable solution for production environments facing this issue?

Hi @mihailmariusiondev, this most likely is not an issue of your region being blocked, but the IP of your cloud provider being blocked. Have a look #303 to find more about this issue.

If anyone is still struggling with this. I have made a working solution using oauth2 that doesn't uses proxies and avoids Youtube ban. You can use it for free on Apify: https://apify.com/invideoiq/video-transcript-scraper. You only pay for Apify usage, however Apify gives you free credit of 5$ which will give you around 5000 transcripts


WIP do not merge, vs2017/vswhere support

So I opted to use the vswhere tool to locate various VS 2017 installations that match certain criterias (i.e. has Visual C++ tooling). However, I've made it optional, if you specify an installation path manually it will use that instead. I found documentation regarding the usage of vswhere and have included it here.
This is incomplete but I wanted feedback from @deplinenoise or @kayru whether you think the direction is fine.
Also, trying to get rid of as much cruft as possible this time. Again, this is incomplete but it gets the compiler running.

If vswhere isn't installed, do you think it's worth while attempting to detect vs location by some other means? Perhaps following the same detection strategy as cmake?

@kayru well, there is PATH, as long as you use it and put things there, it will work. I have multiple additional PATHs setup on my machine. like %USERPROFILE%\bin and C:\bin if you put vswhere there (on my machine) it will work.
IMAO, the design philosophy outlined in the manual goes against it. i.e. don't guess or at least I interpret this as we should be mindful about introducing guesswork to locate vswhere, although the perf argument isn't really applicable here it could end up confusing new users. Ideally, I think we'd want the DAG generation pass to actually depend on these files so that invalidates when for example vswhere changes but that would require us to parse PATH and attempt to resolve the file ourselves. All this would further complicate the toolchain, I'm not against it but it's not essential.
A very simple thing to do would be to of course continue parameterizing the toolchain so that the exact location of vswhere could be specified externally with PATH search behavior as default.
I'm not familiar with the CMake detection strategy. Anything specific that you'd think could be useful?

I think we can go simple here. I bet 99.99% of VS users install to the default location.
I propose:

By default, assume default install location
Allow install location to be overridden via some global environment var (TUNDRA_VS2017_ROOT)
Check that some well-known file is present under install location, otherwise error out and reference above environment variable for the super-infrequent case that someone has installed VS elsewhere

Alright, but I think Microsoft has killed the idea of a default install location unless you imply that this is somehow BuildTools or Community. I would default to BuildTools but that's just me.
C:\Program Files (x86)\Microsoft Visual Studio>dir /b
2017
Installer
Shared

C:\Program Files (x86)\Microsoft Visual Studio>cd 2017

C:\Program Files (x86)\Microsoft Visual Studio\2017>dir /b
BuildTools
Community

If Visual C++ is installed you'd find it in the VC\Auxiliary\Build sub dir.
C:\Program Files (x86)\Microsoft Visual Studio\2017>dir /b BuildTools\VC\Auxiliary
Build
VS

C:\Program Files (x86)\Microsoft Visual Studio\2017>dir /b Community\VC\Auxiliary
VS

The well-known file we need to check for regardless has to be Microsoft.VCToolsVersion.default.txt
C:\Program Files (x86)\Microsoft Visual Studio\2017>type BuildTools\VC\Auxiliary\Build\Microsoft.VCToolsVersion.default.txt
14.11.25503

@deplinenoise made another pass at the vs2017 tooling. I don't know if you think this is crazy or not but I thought it be nice to not have to reverse engineer how to set up the entire environment and instead copy things from the vcvarsall.bat file. So that's what I did and this actually works.
This can be further improved by filtering out things that don't differ from host environment. One thing which is a departure from how we've done this in the past is that the path to the compiler and linker is not absolute and it's is using path instead to find the right command.
Now I just wanted to try this and see if it would work and it does. It builds things just fine like this. Not sure if it's a good future-proof method but it does assume as much about the internals of the Visual C++ tooling, it just expects the vcvarsall.bat file to do it's thing.

When will this make it in? Works great for me!
I implemented a small optimization: https://github.com/katacode/tundra/commit/119ad4c8e2da4d877bed07a34e297ee83b187040


Adding ProgressHud to UITableViewController

If you add the ProgressHUD to a UITableViewController it does not look good. Of course its inside the dragable tableview and is kind of positioned "behind" the divider lines, making them show above it. Any possibilities of adding a workaround for this, or is the solution just to not use it on UITableViewControllers?

@kautaleksei  Thanks


gstreamer receives http 410

After some inactivity (a few hours), some Qobuz tracks cannot be played in Mopidy. In the debug log I see the following (the url is anonymized):
[MainThread] mopidy.audio.gst Got ERROR bus message: error=GLib.Error('Gone', 'gst-resource-error-quark', 5) debug='../ext/soup/gstsouphttpsrc.c(1690): gst_soup_http_src_parse_status (): /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin5/GstSoupHTTPSrc:source:\nGone (410), URL: https://streaming-qobuz-std.akamaized.net/file?uid=********&eid=********&fmt=7&profile=raw&app_id=*********&cid=*******&etsp=**********&hmac=***************************, Redirect to: (NULL)'
The issue doesn't affect every track, it's random. I've noticed that within an album, I get 410 from the Qobuz CDN for around 10% of the songs. After restarting Mopidy, the issue gets resolved.
How I tested it: I copied the URLs to another host (the faulty URL with http 410, and the working URL after the restart). When tested with curl, the faulty URL also receives a 410 error on the other host. The only difference between the two URLs is in the etsp and the hmac parameters.

IIRC this issue is solved in dev branch. You can try installing it with pip install git+https://github.com/vitiko98/mopidy-qobuz@dev
I didn't merge it to main yet because I don't have a Qobuz account to test (for now).


Control Bug

Challenge http://freecodecamp.com/challenges/waypoint-remove-classes-from-an-element-with-jquery has an issue. Please describe how to reproduce it, and include links to screenshots if possible.
.
It has to control if the user has really removed the btn-default class. You can in the screenshot that it accepts the code even though i have not inserted btn-default in the removeClass() function

Thanks @harry9656,
Confirmed, challenge is marked completed with incorrect answer $('button').removeClass();


Increase response timeout

My ollama server is a little bit slow but works fine, the issue is that if id doesnt answer after a few seconds spacellama just gives up, so maybe a configurable delay would help slower llama instances

good idea. Adding it to the new version