[SPARK-9043] Serialize key, value and combiner classes in ShuffleDependency

ShuffleManager implementations are currently not given type information for
the key, value and combiner classes. Serialization of shuffle objects relies
on objects being JavaSerializable, with methods defined for reading/writing
the object or, alternatively, serialization via Kryo which uses reflection.
Serialization systems like Avro, Thrift and Protobuf generate classes with
zero argument constructors and explicit schema information
(e.g. IndexedRecords in Avro have get, put and getSchema methods).
By serializing the key, value and combiner class names in ShuffleDependency,
shuffle implementations will have access to schema information when
registerShuffle() is called.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Jenkins, test this please.

There was a timeout fetching from the git repo. Having Jenkins try again.

Merged build triggered.

Merged build started.

Test build #37268 has started for   PR 7403 at commit 8c980c4.

I don't think that we can make this change as-is since we can't break binary compatibility for stable public APIs like PairRDDFunctions.

@JoshRosen What suggestions do you have for making the key, value and combiner class names available in the ShuffleDependency in a way that is binary compatible?
I had assumed that this would be part of the next major release where binary changes might be allowed.

Also, the comment in the DeveloperApi annotation reads...
/**
 * A lower-level, unstable API intended for developers.
 *
 * Developer API's might change or be removed in minor versions of Spark.
 *
 * NOTE: If there exists a Scaladoc comment that immediately precedes this annotation, the first
 * line of the comment must be ":: DeveloperApi ::" with no trailing blank line. This is because
 * of the known issue that Scaladoc displays only either the annotation or the comment, whichever
 * comes first.
 */

The PairRDDFunctions classes isn't labeled DeveloperApi so what is the assumed level of stability?
Luckily, the original PairRDDFunctions has ClassTags for the key and value; however, the combine* methods didn't have info on the combiner classes. Not sure how to work around that. Ideas?

Test build #37268 has finished for   PR 7403 at commit 8c980c4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

@massie no labeling means it should be stable across the entire Spark 1.x release...

Merged build triggered.

Merged build started.

Test build #37380 has started for   PR 7403 at commit 1be4402.

@JoshRosen @rxin I just pushed an update that reverts the PairRDDFunctions ClassTag implicits to their original form.
The only remaining sticking point is the three combineByKey methods in PairRDDFunctions which require a ClassTag in order to get type information about the combiner class. Does adding an implicit to a method break binary compatibility? (I'm certain that removing an implicit would)

Test build #37380 has finished for   PR 7403 at commit 1be4402.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]],
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Merged build finished. Test FAILed.

To answer my own question, adding a ClassTag, of course, breaks binary compatibility since it changes the method signature. Sorry about the unnecessary comment chatter.

Merged build triggered.

Merged build started.

Test build #37423 has started for   PR 7403 at commit 5c58b4d.

I just pushed an update (5c58b4df9b) which ensures that we keep binary compatibility in PairRDDFunctions. The old combineByKey methods now call into the new combineByKey2 methods passing in a null for the combiner ClassTag.
If this is an approach that is agreeable, I'll expand the tests and finalize the work.
Any suggestions on a better name of the new combiner methods? I just called it combineByKey2 but I'll sure there's a better name.
Thanks for the review help @JoshRosen and @rxin.

Test build #37423 has finished for   PR 7403 at commit 5c58b4d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]],
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Merged build finished. Test PASSed.

There is another approach that would work here too.
Currently, RDD has following implicit method for the PairRDDFunctions...
object RDD {

  def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
                                 (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null): PairRDDFunctions[K, V] = {
    new PairRDDFunctions(rdd)
  }

Code compiled with this looks like the following,
val rdd: com.github.massie.RDD = new com.github.massie.RDD((ClassTag.apply(classOf[scala.Tuple2]): scala.reflect.ClassTag));
      massie.this.RDD.rddToPairRDDFunctions(rdd, (ClassTag.apply(classOf[java.lang.String]): scala.reflect.ClassTag), (ClassTag.apply(classOf[java.lang.String]): scala.reflect.ClassTag), scala.math.Ordering$String).combineByKey()

To keep binary compatibility, we just need to keep the rrdToPairRDDFunctions, remove the implicit and create a new implicit, e.g.
object RDD {

  def rddToPairRDDFunctions[K, V](rdd: RDD[(K, V)])
                                 (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null): PairRDDFunctions[K, V] = {
    new PairRDDFunctions(rdd)
  }

  implicit def rddToNewPairRDDFunctions[K, V](rdd: RDD[(K, V)])
                                 (implicit kt: ClassTag[K], vt: ClassTag[V], ord: Ordering[K] = null): NewPairRDDFunctions[K, V] = {
    new NewPairRDDFunctions(rdd)
  }
  ...
}

Old code will still run since the rddToPairRDDFunction exists and PairRDDFunctions has not been modified. New code will build against the new NewPairRDDFunctions class which included the ClassTag information in the combineByKey methods.
As a small side note, the PairRDDFunctions would wrap the NewPairRDDFunctions so even though we don't have copy-paste. The combineByKey functions in PairRDDFunctions would pass in a null ClassTag. That means that old applications will work as before, but not provide combiner class information to the shuffle.
The advantage of this approach is that the method names rename the same (no need for a combineByKey2 or whatever).

Merged build triggered.

Merged build started.

Test build #37988 has started for   PR 7403 at commit 5aba066.

Test build #37988 has finished for   PR 7403 at commit 5aba066.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

This is ready for review when someone has the time. All unit tests pass.

Only classes marked @DeveloperApi or are private[spark] have been altered to include ClassTags (breaking compatibility)
The PairRRDFunctions have three new sortByKeyWithClassTag methods that mirror the original sortByKey signatures
All calls to sortByKey have been moved to sortByKeyWithClassTag. Only users that call sortByKey directly will not have combiner class type information provided to the shuffle.
I've added a warning message when sortByKey is used
As an aside, I wrote the code to use the implicit trick I talked about and it turned out to complicate the code quite a lot.

Merged build triggered.

Merged build started.

Test build #38310 has started for   PR 7403 at commit d07b771.

Test build #38310 has finished for   PR 7403 at commit d07b771.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]],
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Merged build finished. Test FAILed.

Jenkins, test this please.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Jenkins, test this please.

Merged build triggered.

Merged build started.

Test build #38367 has started for   PR 7403 at commit d07b771.

Test build #38367 has finished for   PR 7403 at commit d07b771.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]],
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Merged build finished. Test PASSed.

Jenkins tests pass. The other failures were Jenkins hiccups.

Merged build triggered.

Merged build started.

Test build #38403 has started for   PR 7403 at commit bf0344e.

Merged build triggered.

Merged build started.

Test build #38406 has started for   PR 7403 at commit 59abe01.

Test build #38403 has finished for   PR 7403 at commit bf0344e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]],
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Merged build finished. Test FAILed.

Test build #38406 has finished for   PR 7403 at commit 59abe01.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]],
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Merged build finished. Test FAILed.

The error isn't related to this PR...
[info] *** 1 SUITE ABORTED ***
[error] Error: Total 21, Failed 0, Errors 1, Passed 20
[error] Error during tests:
[error] 	org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDSuite

... looks like something related to Kinesis backed streaming.

Jenkins, test this please.

Merged build triggered.

Merged build started.

Test build #38414 has started for   PR 7403 at commit 59abe01.

Test build #38414 has finished for   PR 7403 at commit 59abe01.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]],
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Merged build finished. Test FAILed.

Still failing with...
[info] KinesisBackedBlockRDDSuite:
[info] Exception encountered when attempting to run a suite with class name: org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDSuite *** ABORTED *** (3 seconds, 596 milliseconds)
[info]   com.amazonaws.services.kinesis.model.LimitExceededException: This request would exceed the shard limit for the account<PHONE_NUMBER>25 in us-east-1. Current shard count for the account: 50. Limit: 50. Number of additional shards that would have resulted from this request: 2. Refer to the AWS Service Limits page (http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html) for current limits and how to request higher limits. (Service: AmazonKinesis; Status Code: 400; Error Code: LimitExceededException; Request ID: 69b67101-3273-11e5-a3de-05ea4744cd4e)
[info]   at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1077)
[info]   at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:725)
[info]   at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)
[info]   at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)
[info]   at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2479)
[info]   at com.amazonaws.services.kinesis.AmazonKinesisClient.createStream(AmazonKinesisClient.java:481)
[info]   at org.apache.spark.streaming.kinesis.KinesisTestUtils.createStream(KinesisTestUtils.scala:84)
[info]   at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDSuite$$anonfun$beforeAll$1.apply$mcV$sp(KinesisBackedBlockRDDSuite.scala:46)
[info]   at org.apache.spark.streaming.kinesis.KinesisFunSuite$class.runIfTestsEnabled(KinesisFunSuite.scala:41)
[info]   at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDSuite.runIfTestsEnabled(KinesisBackedBlockRDDSuite.scala:25)
[info]   at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDSuite.beforeAll(KinesisBackedBlockRDDSuite.scala:44)
[info]   at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
[info]   at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDSuite.beforeAll(KinesisBackedBlockRDDSuite.scala:25)
[info]   at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
[info]   at org.apache.spark.streaming.kinesis.KinesisBackedBlockRDDSuite.run(KinesisBackedBlockRDDSuite.scala:25)
[info]   at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462)
[info]   at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671)
[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:294)
[info]   at sbt.ForkMain$Run$2.call(ForkMain.java:284)
[info]   at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[info]   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[info]   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[info]   at java.lang.Thread.run(Thread.java:745)

... which is unrelated to this PR. All other tests pass.

Jenkins, test this please.

Merged build triggered.

Merged build started.

Test build #38562 has started for   PR 7403 at commit 59abe01.

Test build #38562 has finished for   PR 7403 at commit 59abe01.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]],
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Merged build finished. Test FAILed.

Jenkins, test this please.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Jenkins, test this please.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Jenkins, test this please.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Jenkins, test this please.

Merged build triggered.

Merged build started.

Test build #38751 has started for   PR 7403 at commit adb3de5.

Test build #38751 has finished for   PR 7403 at commit adb3de5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]],
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](
case class Abs(child: Expression) extends UnaryExpression with ExpectsInputTypes

Merged build finished. Test PASSed.

@rxin @JoshRosen Sorry for all the Jenkins noise on this PR. Shane and I (mostly Shane!) tracked down the underlying issue on Jenkins that was causing the PRB to fail.
All the unit tests for this PR pass. Please let me know if you see anything else you see that needs to be done before merging this.
Thanks for the review help. I appreciate it.

@rxin @JoshRosen A week has passed without comment. When you have a moment, it would be great to hear your feedback on this PR.

I should mention that this is the last change to Spark Core that I need to make for the Parquet shuffle to work.

I'm a little overloaded on other reviews / tasks right now, so it may take me some time to be able to review this. @rxin, can you triage w.r.t. prioritization for 1.5 / risk, etc?

Thanks, @JoshRosen. Appreciate the response.

@rxin I look forward to your comment.

This PR only adds ClassTags to the following @DeveloperApi classes: Dependency, CoGroupedRDD, and ShuffledRDD.
The existing methods in PairRDDFunctions have not been changed, instead, there are three new sortByKeyWithClassTag methods that mirror the original sortByKey signatures
All calls to sortByKey by higher-level APIs have been moved to sortByKeyWithClassTag. Only users that call sortByKey directly will not have combiner class type information provided to the shuffle. (And it's easy for shuffle implementation to handle that however they choose)

According to the source for DeveloperApi, a Developer API is unstable and can change between minor releases.

Merged build triggered.

Merged build started.

Test build #41754 has started for   PR 7403 at commit c8d5fa8.

Test build #41754 has finished for   PR 7403 at commit c8d5fa8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]],
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41754/
Test PASSed.

@rxin All tests pass after the combinerClassName is changed to Option[String]. Let me know if there's anything else that needs to be changed to merge this.

Merged build triggered.

Merged build started.

Test build #41893 has started for   PR 7403 at commit 56792cb.

Merged build triggered.

Merged build started.

Test build #41897 has started for   PR 7403 at commit 41d2a3c.

@andrewor14 I noticed that I missed a few of the style fixes that you recommended. I just pushed 41d2a3c which fixes them.
Thanks for the reviewing this PR. I appreciate it.

Test build #41897 has finished for   PR 7403 at commit 41d2a3c.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41897/
Test FAILed.

Merged build finished. Test FAILed.

Test build #41893 has finished for   PR 7403 at commit 56792cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](@transient var rdds: Seq[RDD[_ <: Product2[K, _]]],
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41893/
Test PASSed.

@rxin and @andrewor14 - Is there anything more that needs to be done before this PR is ready to be merged? I've made all recommended changes.
There is an open question about having the class names be (a) public variables on a Dependency or (b) parameters passed to registerShuffle. Option A would be better if we want in the future to see the Dependency tree with all class types and option B is better if we don't want to have the class names be public. Of course, option B would require a change to the ShuffleManager API and this PR.
This PR is ready to go for Option A. Let me know your thoughts.

I just rebased this on top of upstream/master in order to fix some conflicts that arose as master changed underneath this PR.
@rxin @andrewor14 This PR has been open for almost two months. Can you please let me know if you see any remaining work that needs to be done before merging?

Merged build triggered.

Merged build started.

Test build #42210 has started for   PR 7403 at commit ed1afac.

@massie The changes here look fine. The only thing I'm still not sure about is the fact that everything is public. You pointed out that ShuffleDependency is DeveloperApi, but things like combineByKeyWithClassTag are not and are fully public. My view is that if in the initial version of this patch we keep as many things private, we can always expose more of these things gradually in the future, but we can't go the other way. E.g. your other patch #7265 doesn't require any of these things to be public.
Other than that, I don't have strong opinions one way or the other about this patch. I think it's a good addition but I'm wary of the many public API changes in this patch. If @rxin thinks it's in a good shape then we should merge it.

Test build #42210 has finished for   PR 7403 at commit ed1afac.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class SparkHadoopWriter(jobConf: JobConf)
class BlockRDD[T: ClassTag](sc: SparkContext, @transient val blockIds: Array[BlockId])
class CoGroupedRDD[K: ClassTag](
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](
class ZippedWithIndexRDD[T: ClassTag](prev: RDD[T]) extends RDD[(T, Long)](prev)
class DefaultSource extends RelationProvider with DataSourceRegister
class Checkpoint(ssc: StreamingContext, val checkpointTime: Time)
abstract class InputDStream[T: ClassTag] (ssc_ : StreamingContext)
abstract class ReceiverInputDStream[T: ClassTag](ssc_ : StreamingContext)

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42210/
Test FAILed.

Merged build finished. Test FAILed.

Thanks for the response, @andrewor14. I'll update this PR to make the class names private. As you say, we can make them public in a future PR, if needed. You're right that #7265 doesn't need them to be public.

Merged build triggered.

Merged build started.

Test build #42220 has started for   PR 7403 at commit 2906e74.

Test build #42220 has finished for   PR 7403 at commit 2906e74.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class SparkHadoopWriter(jobConf: JobConf)
class BlockRDD[T: ClassTag](sc: SparkContext, @transient val blockIds: Array[BlockId])
class CoGroupedRDD[K: ClassTag](
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](
class ZippedWithIndexRDD[T: ClassTag](prev: RDD[T]) extends RDD[(T, Long)](prev)
 case class Instance(w: Double, a: Vector, b: Double)
class DefaultSource extends RelationProvider with DataSourceRegister
class WeibullGenerator(
class IndexToString(JavaTransformer, HasInputCol, HasOutputCol):
class Checkpoint(ssc: StreamingContext, val checkpointTime: Time)
abstract class InputDStream[T: ClassTag] (ssc_ : StreamingContext)
abstract class ReceiverInputDStream[T: ClassTag](ssc_ : StreamingContext)

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42220/
Test PASSed.

Merged build finished. Test PASSed.

The class names are private now and all tests pass.
To provide a little more explanation of why the combineByKeyWithClasstag methods are needed (instead of just naming them combineByKey.
class PlaySuite extends SparkFunSuite {

  def combine[C](c: C): Unit = {
    println("Running combine without ClassTag")
  }

  def combine[C](c: C)(implicit ct: ClassTag[C]): Unit = {
    println("Running combin with ClassTag")
  }

  test("Example") {
    combine(42)
  }

}

Causes the compiler to throw the error:
Error:(34, 5) ambiguous reference to overloaded definition,
both method combine in class PlaySuite of type [C](c: C)(implicit ct: scala.reflect.ClassTag[C])Unit
and  method combine in class PlaySuite of type [C](c: C)Unit
match argument types (Int)
    combine(42)
    ^

There's unfortunately no way to add ClassTags and not break compatibility without have these new methods.
I hope that you find this PR is ready to merge now. Please let me know if you see anything else that needs to be done.

@massie This looks OK to me. The only thing is that I find the name combineByKeyWithClassTag awkward, and because it's a public API we won't be able to remove it in the future. I understand that it won't compile if you just call it combineByKey, but I wonder if we should at least mark these @DeveloperApi or something so we're not fully committed to keeping the signatures exactly as they currently are in the future.
I'll defer the judgment to @rxin.

@andrewor14 I agree the name combineByKeyWithClassTag is awkward. I'm open to changing it to a less awkward name, if you or anyone else can suggest one.
We can't make the combineByKeyWithClassTag methods private, since developers may want to use them directly (as they can with the combineByKey methods), although most likely use the higher-level methods.
I hope that in Spark 2.0, when we're able to fix this API, we can simply add the ClassTag to combineByKey and have no need for the *withClassTag methods.

Let's add an experimental annotation to it, and then merge this.

Merged build triggered.

Merged build started.

Test build #42297 has started for   PR 7403 at commit adcdfaf.

Test build #42297 has finished for   PR 7403 at commit adcdfaf.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class ShuffleDependency[K: ClassTag, V: ClassTag, C: ClassTag](
class CoGroupedRDD[K: ClassTag](
class ShuffledRDD[K: ClassTag, V: ClassTag, C: ClassTag](

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42297/
Test PASSed.

Merged build finished. Test PASSed.

LGTM.

I've merged this. Thanks @massie


Add Rabbit mq cluster operator as an add-on

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?
The ability to install the rabbit mq cluster operator just by enabling it via the eks blueprint add-ons parameters.
Describe the solution you would like
The solution should be able to opt in to install the rabbit mq cluster operator via eks blueprint add-on parameters. The solution should be able to create the IRSA with the required IAM policies for the operator to fulfil its functions.

Describe alternatives you have considered

Additional context

Team, I believe I could start working on this feature request. Could you please assign the task to me?

What is the motivation for this?

Users should be able to deploy this operator today using the helm_releases argument https://github.com/aws-ia/terraform-aws-eks-blueprints-addons/blob/bd1df9c3ff3feab4c8ec5f704fb1acdab01b483f/tests/complete/main.tf#L189-L219
For now, I am going to close this issue - thank you!

@bryantbiggs  thanks for pointing this out , i didn't this way existed .


Update RayService CR to integrate with Ray Nightly

Why are these changes needed?
The ray nightly changes the serve endpoints. This pr applies the new schema.
Manual test results:

Related issue number

Checks

[X] I've made sure the tests are passing.
Testing Strategy

[X] Unit tests
[X] Manual tests
[ ] This PR is not tested :(

@edoakes @DmitriGekhtman Please review this pr and see if we can merge it. Thanks


Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
Repository problems
These problems occurred while renovating this repository.

WARN: Failed to assign reviewer

Open
These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

[ ] chore(deps): update dependency argoproj/argo-helm to v5.31.1
[ ] chore(deps): update dependency metallb/metallb to v0.13.9
[ ] chore(deps): update dependency bitnami-labs/sealed-secrets to v2.8.2
[ ] chore(deps): update dependency kubernetes/ingress-nginx to v4.6.0
[ ] Click on this checkbox to rebase all open PRs at once

Detected dependencies
regex

argocd-config/base/argocd.yaml

argoproj/argo-helm 5.31.0

argocd-config/base/ingress-nginx.yaml

kubernetes/ingress-nginx 4.3.0

argocd-config/base/metallb.yaml

metallb/metallb 0.13.7

argocd-config/base/sealed-secrets.yaml

bitnami-labs/sealed-secrets 2.7.0

.github/renovate.json5

aquaproj/aqua-renovate-config 1.5.2

aqua.yaml

aqua.yaml

ahmetb/kubectx v0.9.4
argoproj/argo-cd v2.7.1
helm/helm v3.11.3
ahmetb/kubectx/kubens v0.9.4

aqua.yaml

aquaproj/aqua-registry v3.159.0

aqua.yaml

kubernetes-sigs/kustomize v5.0.1

duplicated


[Direct Line] Conversation PostActivityAsync never sends message other types go through?

Bot Info

App ID: 4dexpertbot
SDK Platform: .NET
Active Channels: Teams, Skype for Business, DirectLine
Deployment Environment: Azure App service

Issue Description
Skype for Business and the test chat on dev.botframework.com work. As does the emulator locally.
Directline Posts system messages just fine.
However my app service does not receive any of the messages I'm sending.
Code Example
I have multiple bots eventually and I already have oAuth enabled on my service so I created an ICredentialProvider:
    public class BotCredentialProvider : ICredentialProvider
    {
        private readonly Dictionary<string, string> Credentials;

        public BotCredentialProvider()
        {
            Credentials = new Dictionary<string, string>();
            Credentials.Add("MicrosoftExpertBotAppId".GetApplicationVariable(), "MicrosoftExpertBotAppPassword".GetApplicationVariable());
        }

        public Task<string> GetAppPasswordAsync(string appId)
        {
            return Task.FromResult(this.Credentials.ContainsKey(appId) ? this.Credentials[appId] : null);
        }

        public Task<bool> IsAuthenticationDisabledAsync()
        {
            return Task.FromResult(false);
        }

        public Task<bool> IsValidAppIdAsync(string appId)
        {
            return Task.FromResult(this.Credentials.ContainsKey(appId));
        }
    }

This works. (or appears to)
Here's the apicontroller:
    [BotAuthentication(CredentialProviderType = typeof(BotCredentialProvider))]
    public class ExpertBotController : ApiController
    {
        static ExpertBotController()
        {
            var builder = new ContainerBuilder();
            builder.Register(c => ((ClaimsIdentity)HttpContext.Current.User.Identity).GetCredentialsFromClaims())
                .AsSelf()
                .InstancePerLifetimeScope();
            builder.Update(Conversation.Container);
        }

        /// <summary>
        /// Receive a message from the user.
        /// </summary>
        /// <param name="activity"></param>
        /// <returns></returns>
        [HttpPost]
        public async Task<HttpResponseMessage> Post([FromBody] Activity activity)
        {
            try
            {
                if (activity.Type == ActivityTypes.Message)
                {
                    await Conversation.SendAsync(activity, MakeQuestionDialog);
                }
                else
                {
                    await HandleSystemMessage(activity);
                }

                return Request.CreateResponse(HttpStatusCode.OK);
            } catch (Exception ex)
            {
                return Request.CreateErrorResponse(HttpStatusCode.BadRequest, ex.UnwindException());
            }
        }

...
}

This works for the working cases and for system messages. It fails (i.e. it never gets into the if statement) for messages.
Here's the client code:
var DirectClient = new DirectLineClient(RestClient.EXPERTBOT_TOKEN);

var Conversation = await DirectClient.Conversations.StartConversationAsync(ResponseCancellation.Token);

            new Thread(async () => await ReadBotMessagesAsync(ResponseCancellation.Token)).Start();

            await DirectClient.Conversations.PostActivityAsync(Conversation.ConversationId, new Activity
            {
                From = new ChannelAccount(App.Client.Token.AccessToken, App.Client.Name),
                Text = QuestionText,
                Type = ActivityTypes.Message
            });

Neither the new thread Reading Bot Messages nor the actual api Post method get called when the PostActivityAsync is called.
Reproduction Steps

Fire up the code above and it doesn't call and there are no errors in the dev.botframework.com page for DirectLine (or anything else), the bots work on Skype for Business and in the test on that page and in the Emulator and Directline sends system messages just not actual messages.

Expected Behavior
Should send activities with the message being sent and work exactly the same as Skype for Business etc.
Actual Results
System messages go through fine, actual messages do not.

The Microsoft Bot Framework team prefers that how to questions be submitted on Stack Overflow. The official Bot Framework Github repo  is the preferred platform for submitting bug fixes and feature requests.

I have closed this issue. Please feel free to resubmit your question over on Stack Overflow and one of our support engineers will help you out as soon as possible.


[Bug] NULL is returned instead of 0 on max-aggregation with docs of value 0 only

CrateDB version:
4.4.1
Environment description:
CrateDB Cloud

Problem description:
NULL is returned instead of 0 on max-aggregation with docs of value 0 only

Steps to reproduce:

CREATE TABLE t1 (
    x TEXT,
    y DOUBLE PRECISION
);

INSERT INTO t1 VALUES ('a',0),('a',0),('a',0),('b',1),('b',1),('b',1),('c',0),('c',1),('b',2),('b',3);

SELECT x, y FROM t1;

SELECT x, MAX(y) FROM t1 GROUP BY x;

Thank you for the report and reproduction steps. This will be fixed with the next 4.4 release.


Mapbox dependency

  node_modules/@geoman-io/maplibre-geoman-free/dist/maplibre-geoman.es.js:4:20:       4 │ import * as yl from "mapbox-gl";         ╵                     ~
The last version of geoman-maplibre is including mapbox instead of maplibre.

Hi mskzo, yes we are working on adding compatibility to mapbox. However this should not be exposed in the library. Thanks for pointing this out so that we can fix it.

The last version of geoman-maplibre is including mapbox instead of maplibre.

Fixed in the last commits.
The plugin now supports two base maps: MapLibre and Mapbox (very alpha stage).
But it requires avoiding any base map-specific imports across the whole project, for example, it's impossible to use TS types from Mapbox and MapLibre in places other than the MapAdapter.
So we have to re-invent some types like map events, etc.

👋 hello still getting
   node_modules/@geoman-io/maplibre-geoman-free/dist/index-DTETc3Nq.js:6:19:
      6 │ import * as g from "mapbox-gl";
        ╵                    ~~~~~~~~~~~

with version 0.2.12 is this expected? cheers

It's a side effect of new way of TS types exporting, it's fixed in the latest commits, will be available soon.
As a quick-fix "mapbox-gl" could be installed: npm install --save mapbox-gl.
Pros: In the latest commits all geoman types will be available without exceptions.

Solved


support multiple evaluators

support multiple evaluators

Coverage remained the same at 71.831% when pulling 154506fe068b7bb8c9d26907fe175b19c42105bb on oikomi:patch-3 into 6a608a789930232ff05f4f04aa31eea845f99b20 on kubeflow:master.

/assign @gaocegege


Remove all references to COARSE location

So that we don't have to declare a new permission that we will not use in the long term https://github.com/e-mission/e-mission-docs/issues/1046#issuecomment-2021951729

Closing this since it was a red herring. The real issue was that the activebeacon library declares the COARSE location permission in its AndroidManifest.xml. We can handle this by either accepting the new location permission, or removing it in our AndroidManifest.xml. I have taken the second approach for now.
https://github.com/e-mission/e-mission-phone/pull/1135#issuecomment-2022050839


Odd behaviour or maybe I am using it wrongly

I am using filepond to upload pictures for real estate listings.
When I create new listings everything works fine, I upload everything to Amazon S3 internally and no complaints. The odd situation arises when I allow the user to modify the listing, obviously, I have to show all the images previously uploaded and I am doing it this way:
<fieldset class="filepond" id="file"               files="fieldset">         <ul>             @foreach($images as $asset)                 <li>                     <label>                         <input value="{{ $asset }}" data-type="local" checked type="checkbox" style="display:none;">                     </label>                 </li>             @endforeach         </ul>
What filepond is doing is like uploading everything again, you can read the uploading and takes ages. That's not normal because the main thing I need is to show the images inside the filepond container and that should be something straightforward, I need to have the X button to allow the user to delete the picture and all the functionality rearranging pictures, etc.
This is code in js: (I know is a bit messy but works, for some reason wasn't showing the fading background behind the text, it was loading the image but not showing, the behaviour, as I said, was odd and for sure it's my fault.
`load: (source, load, error, progress, abort, headers) => {
// This is the AWS S3 URL https://bucket...../image.jpg
let fetchURL = getURL('get-file')+'/'+elemFile.getAttribute('identifier')+'/';
                fetch(fetchURL + source)
                    .then(response => response.blob())
                    .then(blob => {
                        load(blob)
                        var uri = URL.createObjectURL(blob);
                        var img = new Image();

                        // I had to do this code in order to show to show the image, the image was there but hidden.
                        let elemText = Array.from(document.querySelectorAll('span')).find(el => el.textContent === source);
                        let panel = elemText.parentElement.parentElement.parentElement.parentElement.querySelectorAll('.filepond--panel')[0];
                        img.src = uri;
                        img.setAttribute("style", "position:absolute; top: 0; border-radius: 8px;");

                        [].forEach.call(panel.querySelectorAll('.filepond--item-panel'), function(div) {
                            div.setAttribute("style", "display: none !important;");
                        });
                        // I had to do this code to position the background underneath the text.
                        const newDiv = document.createElement("div");
                        newDiv.setAttribute("style", "z-index: 1; width:100%; position:absolute; top: 0; border-radius: 8px; height: 50px; background-image: linear-gradient(rgba(0,0,0,0.25), rgba(0,0,0,0));");
                        panel.appendChild(newDiv);
                        panel.appendChild(img);

                    });
            }`

Can anyone help me with this issue, absolutely lost with this issue. I want filepond loads the image from the URL showing the text and background and keep the functionality on the X button to delete the picture.
Million thanks

Why not use the image preview plugin?

Thank you, I will have a look :)

Closed because inactive.


⚠ Bofrost.de has degraded performance

In 5ad6472, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1629 ms

Resolved: Bofrost.de performance has improved in 00f946a after 8 minutes.


PLAT-86939: Update Moonstone Icon (updated sizes)

Checklist

[x] I have read and understand the contribution guide

[x] A CHANGELOG entry is included

[ ] At least one test case is included for this feature or bug fix

[x] Documentation was added or is not needed

[ ] This is an API breaking change

Issue Resolved / Feature Added
Update with Moonstone Icon with sizes fixed to fit within the margins

Before

After

Forgot ... @teckliew let's tack on a change log as well. Otherwise :shipit:

We're going to ask about the remaining overflowing icon glyph, which we think is intentional, but want to verify it A) is intentional B) whether it should be aligned with its twin icon: (A)-(B) and (A)-(B)x

LIke Blake said, just gave feed back about repeat and repeatoff icons in the ticket, and if that's good to go, we can push this pr ahead.


Issue with trailingSlash and prerendered routes

Describe the bug
I have a website with prerendered pages running via adapter-node. The problem is that users sometimes use trailing slash in URLs. In this case, the loaded page looks like it has no css loaded, but with the correct content. I would expect a redirect to the URL without a trailing slash.
What is the expected behavior in this case?
https://sveltekit-trailing-slash.vercel.app/test

https://sveltekit-trailing-slash.vercel.app/test/

Reproduction
Demo: https://sveltekit-trailing-slash.vercel.app/
Repository: https://github.com/yellowby/sveltekit-trailing-slash
Logs
No response
System Info
System:
    OS: Windows 10 10.0.22621
    CPU: (16) x64 Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
    Memory: 38.98 GB / 63.93 GB
  Binaries:
    Node: 18.13.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.19 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 8.19.3 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: Spartan (44.22621.1105.0), Chromium (109.0.1518.69)
    Internet Explorer: 11.0.22621.1
  npmPackages:
    @sveltejs/adapter-auto: ^1.0.2 => 1.0.2
    @sveltejs/adapter-node: ^1.1.4 => 1.1.4
    @sveltejs/kit: ^1.0.0 => 1.3.1
    svelte: ^3.54.0 => 3.55.1
    vite: ^4.0.0 => 4.0.4

Severity
annoyance
Additional Information
No response

It looks like you mean adapter-vercel rather than adapter-node?
The expected behaviour is that /test/ redirects to /test, so that relative asset paths are correct. It looks like the routes config generated by adapter-vercel is incorrect though.

@Rich-Harris Sorry for the confusion (should post screenshots from localhost and adapter-node). The issue affects both adapter-node and adapter-vercel.

It looks like you mean adapter-vercel rather than adapter-node?
The expected behaviour is that /test/ redirects to /test, so that relative asset paths are correct. It looks like the routes config generated by adapter-vercel is incorrect though.

Should the trailing slash redirect logic be moved from the adapters to instead be part of server.respond?
Seems like an issue that affects any adapter that doesn't implement this logic itself.

It is part of server.respond — adapters just need to be careful about not responding with foo/index.html for /foo or foo.html for /foo/, and letting SvelteKit handle the request instead. Unfortunately in the adapter-node case I think this might involve hacking around sirv


Adds support to Markdown Component for rendering Widgets (and other things...)

As requested by Illia in the post found here.
Writing {{path/to/widget~blockHeight}} in the markdown (blockHeight is optional) will call a provided onPath function. The onPath function can then render the desired Widget/Thing via the path.

Great work! @elliotBraem I also plan to work on the widget rendering in markdown this week, but it's awesome you have done this.
One suggestion for passing the props to the widget, maybe we can use the query parameter format like mob.near/widgets/Profile?accountId=root.near.

As per @think-in-universe 's suggestion, we can use the query format like mob.near/widgets/Profile?accountId=root.near for embedding widgets with props.
{{mob.near/widget/Profile?accountId=root.near}}

Rebased to dev @evgenykuzyakov
Yes, you can try it out here : https://everything.dev/#/efiz.near/widget/test.markdown

I like it! And I think I prefer it as well -- you got me thinking about how claiming {{}} probably wasn't the best idea; I like to be able to write {{SOMETHING}} sometimes and I'm sure others do too. Something like bos:// is unique. Want to create a PR from your branch instead? @think-in-universe

@elliotBraem sure. Since it's an open question, we can ask people to vote their favorite syntax, and we can support multiple formats if needed.
yeah. I can submit my PR later today for you and Eugene to review. I will update the BOS URL regex to be more generic, so it supports not only widgets but also other standards like things.

Curious if we should use near:// instead bos://?
Because the idea is that you retrieve component data from NEAR blockchain.
I was thinking for when there is native browser for BOS, it would be near:// URLs and also mobile could route to it.

@ilblackdragon Good point for the native browser and mobile app routing / deep linking.
My current thought is that bos:// scheme makes it easier for deep linking into BOS browser or app on desktop and mobile, just like Slack's deep linking. It's usually harder for the desktop / mobile apps to maintain a list of schemes (e.g. near://, eth://, cosmos://, ...) that keep changing or growing.
Besides making near and eth schemes by themselves, we can also make them part of BOS URL.
bos://<chain?>/<domain?>/<src>?<query>#<fragment>

chain is optional, default to be near, can also be inferred from domain in some cases (such as social.near, lens.eth)
domain is optional, default to be social.near (contract ID) for near
illegal chain and domain will fallback to default ones.

The below URLs are valid following the above pattern:

bos://near/social.near/mob.near/widget/Profile?accountId=root.near
bos://social.near/mob.near/widget/Profile?accountId=root.near (chain = near)
bos://mob.near/widget/Profile?accountId=root.near  (chain = near, contractId = social.near)
bos://mob.near/widget/Profile  (chain = near, contractId = social.near)
bos://near.org/mob.near/widget/Profile?accountId=root.near   (chain = near, contractId = social.near)
bos://near.social/#/mob.near/widget/Profile?accountId=root.near   (chain = near, contractId = social.near)
bos://eth/lens.eth/alice.lens/widget/Profile?accountId=bob.lens
bos://eth/0x28af365578586eD5Fd500A1Dc0a3E20Fc7b2Cffa/alice.lens/widget/Profile?accountId=bob.lens
bos://lens.eth/alice.lens/widget/Profile?accountId=bob.lens (chain = eth)

@ilblackdragon another option is that we don't limit the schemes in VM, but allow components and gateway to make their own decisions, though consensus are preferred to come up a standard way.
@elliotBraem I have submitted my updated implementation in this PR: https://github.com/NearSocial/VM/pull/90  which implements a onURL handler for processing URLs in the markdown. Markdown components that wants to support widget rendering can use this function to build their own solutions.

I think this PR can be deprecated now. It can be achieved with onLink from #132
Example:
const onLink = (props) => {
  // TODO: Parse `props.href`
  return <a {...props} />;
};

const text = `
[**Link1**](bos://mob.near/widget/Bla?foo=bar)
`;

return <Markdown text={text} onLink={onLink} />;


Binding variable after comma operator does not work

❯ jq -n '[1, 2 as $x | $x + 1]'
[
  1,
  3
]

❯ xq -n '[1, 2 as $x | $x + 1]'
Error: compile query

Caused by:
    ParseError(UnrecognizedToken(Loc { line: 0, col: 6, byte_idx: 6 }, "Keyword(\"as\", As)", ["\"!=\"", "\"%\"", "\"%=\"", "\")\"", "\"*\"", "\"*=\"", "\"+\"", "\"+=\"", "\",\"", "\"-\"", "\"-=\"", "\"/\"", "\"//\"", "\"//=\"", "\"/=\"", "\":\"", "\";\"", "\"<\"", "\"<=\"", "\"=\"", "\"==\"", "\">\"", "\">=\"", "\"]\"", "\"and\"", "\"catch\"", "\"elif\"", "\"else\"", "\"end\"", "\"or\"", "\"then\"", "\"|\"", "\"|=\"", "InterpolationEnd"]))

❯ xq --version
xq 0.1.9-dcd6c4a024b54b3b2d0dc7ed90a349d42c11aa67

This looks like a regression due to switching to lalrpop.
❯ xq -n '[1, 2 as $x | $x + 1]'
[
  1,
  3
]

❯ xq --version
xq 0.1.9-95558180fabd53143a91d72e226780939461f8c5

Ah good catch! I actually knew that label and binding patterns are no longer be in a child of operators and ignore it since it complicates the grammar definition a lot, but yeah looks like I should've implemented it.

Hmmmmmm handling try . as $a in lalrpop would be further difficult.
It may be easier to make lexer insert markers at the end of the scope of label, def and ... as ..., and maybe try (without catch)....


Two column layout

Is there a built-in way to have a two column (or three, four maybe six column) layout with ngx-formly?
I tried to set a fieldGroup with fieldGroupClassName "row" on the outside, and inside I wanted to use two fieldGroups with fieldGroupClassName "col-sm-6". The fields inside of the columns would have col-sm-12, so that they would line up vertically.
However, this doesn't work correctly.
Is it possible to have two columns, where fields are vertically aligned per column? It would be nice if the column width could be set like "col-sm-4" for each column

have you checked the example https://formly.dev/examples/bootstrap-specific/advanced-layout ?

Yes ... but this layout is still row oriented. The fields of a group are in one row
I would like to have two (or more) field groups. Each group in a column, vertically aligned. The grouped fields constitute a logical group (e.g. address information, car information etc.)

https://stackblitz.com/edit/angular-yp4t9a

Ah interesting. I thought I had to use fieldGroupClassName for fieldGroups. Obviously, it would have worked with just className
What is the exact difference between those two?

Each field group is rendered using the following component tree where className is applied to formly-field and fieldGroupClassName to formly-group component:
<formly-field [class]="fieldGroup.className">
 <formly-group [class]="fieldGroup.fieldGroupClassName">
  <formly-field [class]="child1.className"></formly-field>
  <formly-field [class]="child2.className"></formly-field>
 </formly-group>
</formly-field>

Thank you, it works fine now


NETWORK_ERR: XMLHttpRequest Exception 101

I did find this: https://github.com/ariya/phantomjs/issues/10985 and I went through the entire thing but it was not able to help me.
I'm using PhantomJS with CasperJS, and attempting to make a GET request for a JSON object. On the .send() method, I'm getting the NETWORK_ERR: XMLHttpRequest Exception 101 error.  After digging through this, I believe that the issue is most likely with PhantomJS and not CasperJS.
PhantomJS version:
1.8.2, 1.9.2, 1.9.8, 2.0.0
How to reproduce:

Using CasperJS and this file: https://gist.github.com/melissanoelle/704796109351ed828abe
Run casperjs --ssl-protocol=tlsv1 --ignore-ssl-errors=true --cookies-file=/tmp/cookies.txt --web-security=no test home_page.js
See error:

A-0666:front-end-functional-tests madamaitis$ casperjs --ssl-protocol=tlsv1 --ignore-ssl-errors=true --cookies-file=/tmp/cookies.txt --web-security=no test home_page.js
2016-02-24 14:09:06.733 phantomjs[51669:19647671] *** WARNING: Method userSpaceScaleFactor in class NSView is deprecated on 10.7 and later. It should not be used in new applications. Use convertRectToBacking: instead.
Test file: home_page.js
# Home page body tests.
XMLHttpRequest cannot load https://www.limelight.com/get_strings?page_name=home_page. Origin file:// is not allowed by Access-Control-Allow-Origin.
FAIL Error: NETWORK_ERR: XMLHttpRequest Exception 101
#    type: uncaughtError
#    file: home_page.js:90
#    error: {"line":90,"sourceId":252873456,"sourceURL":"home_page.js","stack":"Error: NETWORK_ERR: XMLHttpRequest Exception 101\n    at home_page.js:90\n    at /usr/local/Cellar/casperjs/1.1-beta3/libexec/modules/tester.js:1064\n    at /usr/local/Cellar/casperjs/1.1-beta3/libexec/bin/bootstrap.js:60\n    at begin (/usr/local/Cellar/casperjs/1.1-beta3/libexec/modules/tester.js:1093)\n    at home_page.js:141","stackArray":[{"sourceURL":"home_page.js","line":90},{"sourceURL":"/usr/local/Cellar/casperjs/1.1-beta3/libexec/modules/tester.js","line":1064},{"sourceURL":"/usr/local/Cellar/casperjs/1.1-beta3/libexec/bin/bootstrap.js","line":60},{"function":"begin","sourceURL":"/usr/local/Cellar/casperjs/1.1-beta3/libexec/modules/tester.js","line":1093},{"sourceURL":"home_page.js","line":141}],"code":101,"message":"NETWORK_ERR: XMLHttpRequest Exception 101","name":"NETWORK_ERR"}
#    stack: not provided
FAIL 1 test executed in 0.134s, 0 passed, 1 failed, 0 dubious, 0 skipped.

Details for the 1 failed test:

In home_page.js:90
  Home page body tests.
    uncaughtError: Error: NETWORK_ERR: XMLHttpRequest Exception 101

CasperJS's documentation said to add the flag --web-security=no, and that this error would be solved. I looked through CasperJS's source code, and found that it was just piping that along directly to PhantomJS. So, then I looked through PhantomJS's source code for where the web-security flag is used and found this:
https://github.com/ariya/phantomjs/blob/b483dd673a1ca589ff10c5f73dfea1e43bfa3225/test/ghostdriver-test/java/src/test/java/ghostdriver/BaseTest.java#L103-L108
Java is certainly not my strongest language, and I'm certainly not very familiar with the codebase for this, but it looks as if the ability to turn off web security has been... commented out?
However, there's also a C++ file here:
https://github.com/ariya/phantomjs/blob/3e8e04e3d6965f73f85dcf0fab7da82a43da689b/src/config.cpp#L804-L806
That seems to allow it to be set... however it sets a variable called m_webSecurityEnabled that based on a search of the repo I don't see being used?
It was at this point that I decided that perhaps this would be better handled by asking someone more familiar with this project for some help. If this is actually an error with CasperJS, I'm happy to go open this up over there. It is sometimes difficult to be completely sure where to open an issue at. Thanks in advance.
Expected output
That the XMLHttpRequest object can retrieve the data at the specified endpoint without creating the NETWORK_ERR: XMLHttpRequest Exception 101 error.
My OS
Mac 10.10.3
How I installed PhantomJS
via Homebrew

Is this something that I should be opening over at CasperJS? How can I get some attention on this issue here?

Hi @melissanoelle,
thanks for this complete report. As CasperJS is mostly a wrapper around phantomJS or SlimerJS, I think it's an phantomJS issue.
For instance, you should try to use SlimerJS instead of phantomJS: if it works as expected, it's not a CasperJS issue ^^
Mickaël

Thanks a lot for filing this issue! But could you please try again with the latest PhantomJS?
I see that you used 2.0.0, but this version doesn't have a fix for your issue (https://github.com/ariya/phantomjs/commit/d938eddaf2039cc4d8a90704f736129edb8e1260)
It was landed in later versions.
Thank you!

@mickaelandrieu -- I attempted to install and use Slimer, but it goes up to FF version 38 and I have 44 installed. After messing around with it for a few hours I decided to drop it.
@Vitallium -- I'm having trouble installing the latest version from source and then using it in the command line. I followed the instructions here: https://github.com/ariya/phantomjs/tree/master/deploy but when I attempted to run the make clean... bit, it failed on me:
A-0666:phantomjs madamaitis$ make clean && cd src/qt && make clean && cd ../..
cd src/ && ( test -e Makefile.phantomjs || /Users/madamaitis/repos/phantomjs/src/qt/qtbase/bin/qmake /Users/madamaitis/repos/phantomjs/src/phantomjs.pro -o Makefile.phantomjs ) && /Applications/Xcode.app/Contents/Developer/usr/bin/make -f Makefile.phantomjs clean
rm -f qrc_phantomjs.cpp qrc_ghostdriver.cpp
rm -f moc_phantom.cpp moc_callback.cpp moc_webpage.cpp moc_webserver.cpp moc_networkaccessmanager.cpp moc_cookiejar.cpp moc_filesystem.cpp moc_system.cpp moc_env.cpp moc_terminal.cpp moc_config.cpp moc_childprocess.cpp moc_repl.cpp moc_qcommandline.cpp
rm -f webpage.moc
rm -f phantom.o callback.o webpage.o webserver.o main.o utils.o networkaccessmanager.o cookiejar.o filesystem.o system.o env.o terminal.o encoding.o config.o childprocess.o repl.o crashdump.o mongoose.o linenoise.o utf8.o qcommandline.o phantomjs_plugin_import.o qrc_phantomjs.o qrc_ghostdriver.o moc_phantom.o moc_callback.o moc_webpage.o moc_webserver.o moc_networkaccessmanager.o moc_cookiejar.o moc_filesystem.o moc_system.o moc_env.o moc_terminal.o moc_config.o moc_childprocess.o moc_repl.o moc_qcommandline.o
rm -f *~ core *.core
make: *** No rule to make target `clean'.  Stop.

I figured "well, I haven't previously built this, so it's probably ok" so I moved on to verifying the version, but the terminal didn't recognize phantomjs and so failed when I attempted to run phantomjs --version. (/usr/local/bin/phantomjs doesn't exist)
I also tried the Mac zip here: http://phantomjs.org/download.html but I wasn't sure how to run it with CasperJS because all it did was open an terminal window with a phantomjs> prompt.
What else can I try/is there a different set of directions I should be following?
Alternatively, is the person who packages this for Homebrew still active? I don't know anything about packaging for Homebrew, and the highest version it has is 2.0.0.

@melissanoelle you can use Homebrew. We actively working with Homebrew maintainers. It contains the latest release - 2.1.1. Try to do brew update first.

@istr so no error 101, right?

@istr thank you!

If I'm able to do phantomjs -v in the command line and get back 2.1.1, and then I run the tests with casperjs --ssl-protocol=tlsv1 --ignore-ssl-errors=true --cookies-file=/tmp/cookies.txt --web-security=no test home_page.js is it possible that it might still be defaulting to 2.0.0 somewhere or somehow?
Using what @istr tried, I did this:
A-0666:front-end-functional-tests madamaitis$ ENGINE_EXECUTABLE=~/usr/local/bin/phantomjs-2.1.1 /usr/local/bin/casperjs --ssl-protocol=tlsv1 --ignore-ssl-errors=true --cookies-file=/tmp/cookies.txt --web-security=no test home_page.js
2016-03-01 11:06:16.717 phantomjs[3256:23010188] *** WARNING: Method userSpaceScaleFactor in class NSView is deprecated on 10.7 and later. It should not be used in new applications. Use convertRectToBacking: instead.
Test file: home_page.js
# Home page body tests.
XMLHttpRequest cannot load https://www.limelight.com/get_strings?page_name=home_page. Origin file:// is not allowed by Access-Control-Allow-Origin.
FAIL Error: NETWORK_ERR: XMLHttpRequest Exception 101
#    type: uncaughtError
#    file: home_page.js:90
#    error: {"line":90,"sourceId":251710192,"sourceURL":"home_page.js","stack":"Error: NETWORK_ERR: XMLHttpRequest Exception 101\n    at home_page.js:90\n    at /usr/local/Cellar/casperjs/1.1-beta3/libexec/modules/tester.js:1064\n    at /usr/local/Cellar/casperjs/1.1-beta3/libexec/bin/bootstrap.js:60\n    at begin (/usr/local/Cellar/casperjs/1.1-beta3/libexec/modules/tester.js:1093)\n    at home_page.js:141","stackArray":[{"sourceURL":"home_page.js","line":90},{"sourceURL":"/usr/local/Cellar/casperjs/1.1-beta3/libexec/modules/tester.js","line":1064},{"sourceURL":"/usr/local/Cellar/casperjs/1.1-beta3/libexec/bin/bootstrap.js","line":60},{"function":"begin","sourceURL":"/usr/local/Cellar/casperjs/1.1-beta3/libexec/modules/tester.js","line":1093},{"sourceURL":"home_page.js","line":141}],"code":101,"message":"NETWORK_ERR: XMLHttpRequest Exception 101","name":"NETWORK_ERR"}
#    stack: not provided
FAIL 1 test executed in 0.239s, 0 passed, 1 failed, 0 dubious, 0 skipped.

Details for the 1 failed test:

In home_page.js:90
  Home page body tests.
    uncaughtError: Error: NETWORK_ERR: XMLHttpRequest Exception 101

@mickaelandrieu -- would CasperJS default to using a different version of PhantomJS or something?

Your environment variable ENGINE_EXECUTABLE=~/usr/local/bin/phantomjs-2.1.1 looks fishy.
Do you really have /usr/local... relative to your home?
A simple invocation of casperjs without any arguments will give you the full information, e.g.
$ ENGINE_EXECUTABLE=~/bin/phantomjs-2.0.0 bin/casperjs
CasperJS version 1.1.0-beta4 at {censored}, using phantomjs version 2.0.0
Usage: casperjs [options] script.[js|coffee] [script argument [script argument ...]]
       casperjs [options] test [test path [test path ...]]
       casperjs [options] selftest
       casperjs [options] __selfcommandtest

Options:

--verbose   Prints log messages to the console
--log-level Sets logging level
--help      Prints this help
--version   Prints out CasperJS version
--engine=name Use the given engine. Current supported engine: phantomjs and slimerjs

Read the docs http://docs.casperjs.org/

$ bin/casperjs
CasperJS version 1.1.0-beta4 at {censored}, using phantomjs version 1.9.6
Usage: casperjs [options] script.[js|coffee] [script argument [script argument ...]]
       casperjs [options] test [test path [test path ...]]
       casperjs [options] selftest
       casperjs [options] __selfcommandtest

Options:

--verbose   Prints log messages to the console
--log-level Sets logging level
--help      Prints this help
--version   Prints out CasperJS version
--engine=name Use the given engine. Current supported engine: phantomjs and slimerjs

Read the docs http://docs.casperjs.org/

Ah, good catch.
Thanks, I'm getting a different error now.
And, I'd like to point out here that after inspecting the output of casperjs --help it shows that it's been on PhantomJS of 1.8.2 this entire time even though I have 2.1.1 installed on my system, so the above error (now edited) can only be applicable for 1.8.2.
Closing this, thanks guys. I appreciate the help.


Unable to Control Aux Switch - Reference Error and Plugin Crash

Describe The Bug:
When attempting to control (open or close) the Flo by Moen using the aux switch either via HomeBridge web UI or Apple Home iOS app, the control fails and causes HomeBridge to restart.
To Reproduce:
1- Configuring Flo by Moen plugin with control and aux switch enabled.
2- Verifying polling and control using standard smart control tile using the HomeBridge web UI or Apple Home app to control.
3- Attempt to control valve with aux switch.
Expected behavior:
Valve opens and closed when using aux switch.
Logs:
[14/07/2024, 15:41:11] Homebridge v1.8.3 (HAP v0.12.2) (Homebridge B773) is running on port 51633.
[14/07/2024, 15:41:11] [Flo-by-Moen] Flo Info: Using local cache Flo token.
[14/07/2024, 15:41:11] [Flo-by-Moen] Flo Info: Token will refresh in 10 hour(s) and 51 min(s).
[14/07/2024, 15:41:11] [Flo-by-Moen] Initializing Flo devices...
[14/07/2024, 15:41:12] [Flo-by-Moen] Flo device updates complete, background polling process started.
Device will be polled each 1 min(s) 30 second(s).
[14/07/2024, 15:42:01] ReferenceError: FLO_VALVE_CLOSE is not defined
at FloOptionSwitch.setAuxSwitch (/var/lib/homebridge/node_modules/homebridge-flobymoen/accessories/optionSwitch.js:115:43)
at On. (/var/lib/homebridge/node_modules/homebridge-flobymoen/accessories/optionSwitch.js:60:52)
at On.emit (node:events:518:28)
at /var/lib/homebridge/node_modules/homebridge/node_modules/hap-nodejs/src/lib/Characteristic.ts:2601:16
at new Promise ()
at On. (/var/lib/homebridge/node_modules/homebridge/node_modules/hap-nodejs/src/lib/Characteristic.ts:2599:14)
at step (/var/lib/homebridge/node_modules/homebridge/node_modules/tslib/tslib.js:195:27)
at Object.next (/var/lib/homebridge/node_modules/homebridge/node_modules/tslib/tslib.js:176:57)
at /var/lib/homebridge/node_modules/homebridge/node_modules/tslib/tslib.js:169:75
at new Promise ()
[14/07/2024, 15:42:01] Got SIGTERM, shutting down Homebridge...
[14/07/2024, 15:42:01] [Homebridge UI] [HapClient] [<IP_ADDRESS>:51633 (0E:66:AA:CB:B7:73)] Failed to set value for Irrigation Valve Valve Switch.
[14/07/2024, 15:42:06] [HB Supervisor] Homebridge Process Ended. Code: 143, Signal: null
[14/07/2024, 15:42:11] [HB Supervisor] Restarting Homebridge...
[14/07/2024, 15:42:11] [HB Supervisor] Starting Homebridge with extra flags: -I -P /var/lib/homebridge/node_modules --strict-plugin-resolution
[14/07/2024, 15:42:11] [HB Supervisor] Started Homebridge v1.8.3 with PID: 7624
[14/07/2024, 15:42:13] Loaded config.json with 0 accessories and 2 platforms.
[14/07/2024, 15:42:13] Loaded 2 cached accessories from cachedAccessories.
[14/07/2024, 15:42:13] ---
[14/07/2024, 15:42:13] Loaded plugin<EMAIL_ADDRESS>[14/07/2024, 15:42:13] Registering platform 'homebridge-flobymoen.Flo-by-Moen'
[14/07/2024, 15:42:13] ---
[14/07/2024, 15:42:13] Loading 2 platforms...
[14/07/2024, 15:42:13] [Flo-by-Moen] Initializing Flo-by-Moen platform...
[14/07/2024, 15:42:13] [Flo-by-Moen] Starting communication with Flo portal
Show the Homebridge logs here, remove any sensitive information.

Plugin Config:

{
    "name": "Flo-by-Moen",
    "auth": {
        "username": <hidden>,
        "password": <hidden>
    },
   "disableCache": false,
    "showTemperatureAndHumidity": true,
    "enableValveControl": true,
    "showHealthTestSwitch": false,
    "showAuxSwitch": true,
    "treatWarningAsCritical": false,
    "surpressWaterNotification": false,
    "deviceRefresh": 90,
    "offlineTimeLimit": 4,
    "sleepRevertMinutes": 120,
    "retryErrorDisplay": 3,
    "showFloTemperature": true,
    "showGPMPSIasLight": false,
    "platform": "Flo-by-Moen"
}

Screenshots:

Environment:

Plugin Version:  v1.0.11
Homebridge Version: Homebridge v1.8.3 (HAP v0.12.2) (Homebridge B773)
Node.js Version: v20.12.2
NPM Version:
Operating System: Raspbian

Thank you!  After reviewing the changelog and reading the history on enhancement number 12, I learned that the ability to control the valve via automations and scenes was already possible using the Eve app.  Since then I have accomplished what I was intending to do using the Eve app.  I do like the idea of being able to work with automations and scenes within the Apple Home app but don't see it as a requirement since it's possible within the Eve app.  I recommend in the description for the showAuxSwitch option in the configuration option guide a comment be added about the possibility of using other HomeKit applications outside of the Apple Home app to get better control of the Flo valve with scenes and automations.  This could help others who might overlook the changelog and not think about the possibility that functionality could be different between HomeKit applications.  This is something I knew of but forgot about when troubleshooting this issue.  Thank you very much for you work!  It is much appreciated!

Good suggestion and will add it to the description  when I release this fix.  I personally just use the Eve App myself to automate the valve and it work flawlessly -- Hence why I miss this simple issue, since this was really out of request from community. Happy automating.

Corrected in version 1.0.12


Auto init of all model properties

Motivation
Currently, the developer must write code to initialize fields of content elements, even though they're supposed to be empty.
ArticleModel stronglyTypedElements = new ArticleModel
{
    Title = "On Roasts",
    PostDate = new DateTime(2017, 7, 4),
    Personas = new[] { },
    TeaserImage = new AssetIdentifier[]{ },
    MetadataOgImage = new AssetIdentifier[] { },
    MetadataTwitterImage = new AssetIdentifier[] { }
};

Design guidelines
It would be nice to have a support for an automatic initialization of empty fields, so as a developer I don't have to do it manually. For example, I want to update the content model:
ArticleModel stronglyTypedElements = new ArticleModel
{
    Title = "On Roasts",
    PostDate = new DateTime(2017, 7, 4)
};

Reference

GetContentItemVariantUpsertModel

@petrsvihlik @JanLenoch  I'd like to help with this enhancement. If it's OK, and since you already groomed, could you give me more details?
Thanks.

Hi @yrmartinez , I sent you an invitation to the project. I'll be able to assign the issue to you once you accept it.
Can you please be more specific about what should we clarify about this issue?
Basically, we need to make sure that the models (two code blocks in the description) get serialized into a JSON with all it's properties.
It may be as easy as changing the NullValueHandling but it can be also a bit more tricky. We need to cover the serialization with unit tests and test the serialized models against the live API.

Hi @petrsvihlik, is this something that still needs to be worked on, seeing as this issue was opened in 2018?

Hi, I'm no longer on the team. Tagging the code owners @Kentico/developer-experience

Hi @NammoPillai, thank you for your interest.
To be honest, I was not sure whether this issue is still valid so I tried it. I was able to upsert a variant with just a subset of the elements of the content item.
I've got an Article content type with two elements (title and description). Both of them are Text elements.
When I tried to upsert variant with just a single element (title) specified, the upsert action worked without errors, and the item was also updated in Kontent. I believe this issue might have been fixed during some refactoring or migration to .NET Core but I'm not sure whether is this the exact scenario.
@lukasturek , could you confirm this was the desired scenario?
My Article model
    public partial class Article
    {
        public const string Codename = "article";
        public const string DescriptionCodename = "description";
        public const string TitleCodename = "title";

        public string Description { get; set; }
        public IContentItemSystemAttributes System { get; set; }
        public string Title { get; set; }
    }

Upsert logic
        static async System.Threading.Tasks.Task Main(string[] args)
        {
            ManagementOptions options = new ManagementOptions
            {
                ProjectId = "40...",
                ApiKey = "ew..."
            };

            // Initializes an instance of the ManagementClient client
            ManagementClient client = new ManagementClient(options);

            // Elements to update
            Article stronglyTypedElements = new Article
            {
                Title = "On Roasts1"
            };

            // Specifies the content item and the language variant
            ContentItemIdentifier itemIdentifier = ContentItemIdentifier.ByCodename("hello");
            LanguageIdentifier languageIdentifier = LanguageIdentifier.ByCodename("default");
            ContentItemVariantIdentifier identifier = new ContentItemVariantIdentifier(itemIdentifier, languageIdentifier);

            // Upserts a language variant of a content item
            ContentItemVariantModel<Article> response = await client.UpsertContentItemVariantAsync<Article>(identifier, stronglyTypedElements);
        }

@makma This is exactly the desired behavior. I'm not aware of this fix, but glad to hear that it's working now. Closing the issue.
@NammoPillai Thank you for trying to help though, really appreciate it!


WFLY-6185 additional permissions to DeploymentOverlayCLITestCase for …

…the case of privileged blocks in VFS
https://issues.jboss.org/browse/WFLY-6185

@ctomc would this still be needed after your VFS permissions fix? (I don't really know the details)

I don't see anything in what this test is doing that looks like the app should need these permissions, so let's not grant them and hide the problem.

I was so engrossed in JBVFS-204 that I did not notice that even VirtualFilePermissions should not be needed. I will take a look at it again.

@istudens Please rebase

Superseded by https://github.com/wildfly/wildfly-core/pull/1753.


IDVA6-1556: Add cookie consent banner with i18n support

Implement cookie consent banner in cookie_consent_banner.njk
Add cookie-related translations to English and Welsh locale files
Update banner text to use localized strings
Include new keys for analytics cookies acceptance/rejection messages

CI: No security warnings found


Types for a Wrapper component of a styled component

Hello together,
im trying to wrap a styled component with a functional component and don't know how to type the functional component to preserve the types/props of the styled component and the corresponding html tag.
You can have a look at the following code sandbox: https://codesandbox.io/s/styled-components-base-forked-pylrdz?file=/src/ButtonWrapper.tsx
If you need more input, please let me know.
thanks in advance
bmsuseluda

Here's a sample for making Button as a common component.
Please go through this...
Create your theme object and pass it inside your Themeprovider where you can decide your theme colors like primary, secondary etc. which is used in below given example

import React, { forwardRef, PropsWithChildren, ForwardedRef, ReactElement } from 'react';

import styled, { css } from 'styled-components';

import Mixin from '@styles/Mixin';

interface ButtonProps extends React.ButtonHTMLAttributes<HTMLButtonElement> {
    children: React.ReactNode;
    primary?: boolean;
    secondary?: boolean;
    round?: boolean;
}

const Button = forwardRef<HTMLButtonElement, ButtonProps>(
    (props: PropsWithChildren<ButtonProps>, ref?: ForwardedRef<HTMLButtonElement>): ReactElement | null => {
        return (
            <Styled.Button {...props} type={props.type || 'button'} ref={ref}>
                {props.children}
            </Styled.Button>
        );
    }
);

Button.displayName = 'Button';
export default Button;

const Styled = {
    Button: styled.button<ButtonProps>`
        border: 1px solid;
        font-family: inherit;
        font-size: 1.6rem;
        font-weight: 500;
        text-transform: capitalize;
        letter-spacing: inherit;
        cursor: ${props => (props.disabled ? null : 'pointer')};
        ${props => (props.round ? Mixin.BorderRadius.Circle : Mixin.BorderRadius[2])}

        \&:hover {
            filter: brightness(0.95);
        }

        ${props => {
            switch (true) {
                case props.primary:
                    return css`
                        background: ${props.theme.Primary};
                        color: ${props.theme.White};
                        border-color: ${props => props.theme.Primary};
                    `;
                case props.secondary:
                    return css`
                        background: ${props.theme.Secondary};
                        color: ${props.theme.White};
                        border-color: ${props => props.theme.Secondary};
                    `;
                default:
                    return css`
                        background: ${props.theme.White};
                        color: ${props.theme.Primary};
                        border-color: ${props => props.theme.ColorBorder};
                    `;
            }
        }}
    `,
};

thanks for your response, but with this approach i would lose the types of the styled component, e.g. the as prop.
In my codesandbox example i use the as prop to switch the underlying HTML tag from button to anchor and therefor need the href prop. If i use this styled component without the wrapper the types reflect this behavior and by the way, this is awesome. With this issue i ask how to forward this typing mechanism of styled component to the wrapper.


frontend check for space between tokens is missing

This is my expression:

nenn(e) (mir) (die) (uhr)zeit

When I hit the mic button, I get this error:

I know that I could write nenn(e mir die uhr)zeit but then I have only 1 optional phrase, but each of them should be optional.
The error is caused by: https://rest.picovoice.ai/rhn/35c5a6b8-59c9-403b-8b4e-740a1acbaec3/wasm/de
Request method: GET
Status code: 500

can you give steps to reproduce this from scratch (i.e. an empty context)? I will then look into a fix/explanation.

First I thought, I get this error because there are multiple optional phrases next to each other. Now I have just realized that I get the same error when the expression is like

Are you fast(er than)

So the error always appears when there is no space between a word/character and the optional phrase, in this case fast(er than) causes the error. When I set a space between, the error will be gone.

you do need the space. the front end should o the check. I keep this open for the fix in front end.

Just as a side note: I get the same error when my expression is like this:

Do you have [enough, strong, ] power

The reason why I use the second comma, is because I want the choices to be optional. I have also tried it with Do you have ([enough, strong]) power but that gives me another error, which is

Expression Parsing Error: no viable alternative at input '(['

UI should now give better feedback now when invalid optionals are entered.


Persistent cookie

According to your documentation:

MaxAge=0 means no 'Max-Age' attribute specified.

However, the dependency boj/redistore breaks this ability in the commit referenced here:
https://github.com/boj/redistore/commit/715bb322255772642fde27cc4a31e1d0089086c5
This is a basic part of any session service, how do we implement this feature in your package?

Oh, of course. Welcome to PR


feat: add PolicyDefinitionNewControllerApi

What this PR changes/adds
Introduce PolicyDefinitionNewControllerApi and implements createPolicyDefinition method
Why it does that
new protocol
Further notes

I kept this PR as small as possible because it gives JsonLD capabilities to the Jersey extension in its entirety. This because we can't think of having an EDC component running that's not able to interpret JsonLDs. So the TypeManagerContextResolver looks for the json-ld registered object mapper.
checks are failing now, but in the meantime I'd like to verify the concept (will fix them later on)

Linked Issue(s)
Part of #2768
Checklist

[ ] added appropriate tests?
[ ] performed checkstyle check locally?
[ ] added/updated copyright headers?
[ ] documented public classes/methods?
[ ] added/updated relevant documentation?
[x] assigned appropriate label? (exclude from changelog with label no-changelog)
[x] formatted title correctly? (take a look at the CONTRIBUTING and Etiquette for pull requests for details)

@ndr-brt Can we give JSON-LD capabilities by mapped context instead of the entire Jersey instance?

Codecov Report
Patch coverage has no change and project coverage change: -1.79 :warning:

Comparison is base (c4fc8a7) 64.79% compared to head (701b27c) 63.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2776      +/-   ##
==========================================
- Coverage   64.79%   63.00%   -1.79%
==========================================
  Files         953      957       +4
  Lines       19561    19599      +38
  Branches     1169     1171       +2
==========================================
- Hits        12674    12348     -326
- Misses       6434     6806     +372
+ Partials      453      445       -8

Impacted Files
Coverage Δ

...pi/configuration/DspApiConfigurationExtension.java
93.33% <ø> (ø)

.../org/eclipse/edc/web/jersey/JerseyRestService.java
90.00% <ø> (-5.00%)
:arrow_down:

...g/eclipse/edc/web/jersey/ObjectMapperProvider.java
0.00% <ø> (ø)

...pse/edc/web/jersey/TypeManagerContextResolver.java
75.00% <0.00%> (-25.00%)
:arrow_down:

...n/java/org/eclipse/edc/jsonld/JsonLdExtension.java
0.00% <0.00%> (ø)

...ava/org/eclipse/edc/jsonld/util/JacksonJsonLd.java
0.00% <0.00%> (ø)

...nagement/policy/PolicyDefinitionApiController.java
100.00% <ø> (ø)

...anagement/policy/PolicyDefinitionApiExtension.java
0.00% <0.00%> (-100.00%)
:arrow_down:

...ement/policy/PolicyDefinitionNewApiController.java
0.00% <0.00%> (ø)

...anagement/policy/model/PolicyDefinitionNewDto.java
0.00% <0.00%> (ø)

... and 1 more

... and 28 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

@ndr-brt Can we give JSON-LD capabilities by mapped context instead of the entire Jersey instance?

yes, you're right, I just wanted to provide a unique way to register an object mapper, not really having it registered for all the contexts automatically. I adapted the PR accordingly


huge packages with long module names problem (sh: Argument list too long)

Hi,
huge packages with long module names can not linked correctly.
System

Nuitka version, full Python version, flavor, OS, etc. as output by this
command.

python -m nuitka --version

1.0.6
Commercial: None
Python: 3.9.0 (default, Nov 15 2020, 14:28:56)
Flavor: Anaconda Python
Executable: /home/user/anaconda39/envs/py39/bin/python
OS: Linux
Arch: x86_64
Distribution: Ubuntu (based on Debian) 20.04.4

How did you install Nuitka and Python

pip in ananconda env

python -m pip freeze

Nuitka==1.0.6

## Example
### create mymodule

"""
python -m nuitka --module m.py --include-package=mymodule
"""
def create_mymodule(n=2000):
    if not os.path.exists("mymodule"):
        os.mkdir("mymodule")

    with open(os.path.join("mymodule", "__init__.py"), "w") as fh:
        i = 1
        fh.write(f"""
import module{i:0200d}

print(module{i:0200d}.f())
""")

    for i in range(n):
        with open(os.path.join("mymodule", f"module{i:0200d}.py"), "[Shell Command Line Length Limitations](https://www.cyberciti.biz/faq/argument-list-too-long-error-solution/)w") as fh:
            fh.write(f"def f():return f'module{i:0200d}'")

if __name__ == '__main__':
    create_mymodule(2000)

create module m.py

import mymodule.module000001

print(mymodule.module000001.f())

nuitka command
python -m nuitka --module m.py --include-package=mymodule
Error
Nuitka-Options:INFO: Used command line options: --module m.py --include-package=mymodule
Nuitka:INFO: Starting Python compilation with Nuitka '1.0.6' on Python '3.9' commercial grade 'not installed'.
Nuitka:INFO: Completed Python level compilation and optimization.
Nuitka:INFO: Generating source code for C backend compiler.
Nuitka:INFO: Running data composer tool for optimal constant value handling.
Nuitka:INFO: Running C compilation via Scons.
Nuitka-Scons:INFO: Backend C compiler: gcc (gcc).
Nuitka-Scons:INFO: Backend linking program with 2010 files (no progress information available).
scons: *** [/home/user/project/tests/m.cpython-39-x86_64-linux-gnu.so] sh: Argument list too long```

### Shell Command Line Length Limitation

```getconf ARG_MAX
2097152

Thanks, that is a nice report. There is code in Nuitka against it already:
# Hack to make Scons use tempfile for gcc linking, to avoid line length limits,
# which can make linking fail with many modules otherwise. Most needed on Windows,
# but useful on other platforms too.
if env.gcc_mode:
    makeGccUseLinkerFile(source_dir=source_dir, source_files=source_files, env=env)
```

Because of that gcc command lines should always be short. It looks as if it has no effect in your case (anymore?).

The example code has a few pasting mistakes, but don't worry, I got it.

I wonder if you have a different scons version in use, that the inline copy one, and that is why. I would be interested in the "LINKCOM" value of the scons-report.txt in the .build folder.

No worries, I reproduced it even with my Debian Buster.

So this now works on Linux for me, but sadly for mingw it's not a string, but a generator with code behind that. Might have to monkey patch around there.

This was fixed with 1.1 release, but I forgot to update this.


Feature: Send email upon Event activity

To help user-engagement we greatly need to increase the number of emails (notifications) we send to our users so they are included in the conversation and re-engaged to use the site.
This feature request would request the following...
• Upon an new event being created email all users that are in that community
• Upon an update to this event (sent max of once per-day) to all attendees of the event
• X hours before an event's start time, suggested 1 hour hardcoded for now, perhaps adjustable in the future
Ideally, this email will have an .ics attachment for ease of automated integrations with calendaring systems, google calendar, etc.

Would v1 include some sort of event reminder as well? Either an .ics email when hitting attending, or an email 24 hours before the start of event?
Or would we want to leave that for v2?

Note: Per feedback/discussion in Slack updated the feature request's specifications
See: https://couchersorg.slack.com/archives/C025XSM3NVB/p1651611905607559

This is done


Diff Report Chart Selection and Filtering

Diff report chart selection is broken.
Generates issue of innerHTML undefined on null object. The reason is that plot in naarad.js has changed and takes in 6 arguments rather than 5.
In report.html for Diff report:
<select id="select-chart-1" onchange="plot('select-chart-1','charting-div-1',0,false,'csv-url-div-1')">

In naarad.js which is common to normal reporting:
function plot(selector_id, reset_selector_id, div_id, colorset_id, advanced_source, url_div)

Quick hotfix is to just add the reset_selector_id to be the other selection box, i.e. select-chart-2 for the given example.
But I think for diff reports it'd be better to have the second selection box in this case select the same metric so both charts of same metric are shown. This would require some changes in functionality.
Second, we should add filtering as seen in normal reporting.
Summary:

Fix to have plot called correctly in Diff's report.html.
Add filtering on the selection boxes to Diff's report.html.

This issue ticket will refer to only the bug fix now since we want to have metrics to be able to be plotted. Currently the charting in diff report doesn't even work.

This was merged.


Navigating to initial backend website does not work

In page: Tutorial: Authenticate and authorize users end-to-end in Azure App Service
In the 'Check your changes' section, you should be able to browse the to freshly deployed backend website.  When you do, the web site comes up, but it hangs reading any data (you see 'Loading', but that's it).  In the debugger, the hang appears to be:
    [HttpGet]
    public async Task<ActionResult<IEnumerable<TodoItem>>> GetTodoItem()
    {
        var data = await _client.GetStringAsync($"{_remoteUrl}/api/Todo");  /* here */
        return JsonConvert.DeserializeObject<List<TodoItem>>(data);
    }

Some additional configuration is missing, I believe.
The frontend website appears to work.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8ff46628-b5dc-8b02-02b6-8f510d8059b0
Version Independent ID: 24d09b70-d144-d152-ac5a-c49219a5d8e8
Content: Tutorial: Authenticate users E2E - Azure App Service
Content Source: articles/app-service/app-service-web-tutorial-auth-aad.md
Service: app-service
GitHub Login: @cephalin
Microsoft Alias: cephalin

@RotateAt60MPH Thanks for the feedback! We are currently investigating and will update you shortly.


Supporting partial schema resolution using import schemaLocation hint path

Added a new Validator.AddSchema() signature so the XmlSchemaSet can resolve imports instead of supplying a XmlReader that apparently can't follow hints from schemaLocation.
Will refactor a bit in a near future towards DRY.
Added a very rough test project just to make sure the new signature has the same output as the existing one (used test data from the previous NMatrix.Schematron WinForms test app. Tests SHOULD be improved soon.

Looks good, but plz switch to using xunit + xunit.runner.visualstudio nugets for tests, and a regular class library project, rather than MS's testing framework and project type.
Thanks!

I'll reopen this so the build triggers

Ended up being unnecessary. Github is so magical sometimes I forget...

I will. I'm a bit on a tight schedule and commuting between tasks, that's
why used default tools out of the box.
I'll give it some love soon. Promise.
Thanks for accepting the PR.
On Thu, Dec 17, 2015 at 10:49 PM, Daniel Cazzulino<EMAIL_ADDRESS>
wrote:

Looks good, but plz switch to using xunit + xunit.runner.visualstudio
nugets for tests, and a regular class library project, rather than MS's
testing framework and project type.
Thanks!
—
Reply to this email directly or view it on GitHub
https://github.com/kzu/Schematron/pull/1#issuecomment-165606656.


Updates for lightning 0.10.0

Before submitting

[ ] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
[ ] Did you read the contributor guideline, Pull Request section?
[ ] Did you make sure to update the docs?
[ ] Did you write any new necessary tests?

What does this PR do?
Fixes # (issue).
PR review
Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃

@williamFalcon I think this one is ok, but it seems full testing has been broken since #248 😢 .


Integrated terminal fails to recover to previous state when vscode is closed

Type: Bug
I normally run 5 terminals open at the same same time.  I always have these open. They are the first thing I open and I never close them. I if I need another, then I open another terminal window. The bug is that when I open vs code there is only terminal open, seemly a random one of the five I noramaly have. The bug is intermittent. It happens most commonly when I my linux system crashes but not always. It also occurs when I shut down the system normally.
VS Code version: Code 1.75.1 (441438abd1ac652551dbe4d408dfcec8a499b8bf, 2023-02-08T21:35:30.018Z)
OS version: Linux x64 5.15.0-60-generic snap
Modes:
Sandboxed: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i5-10400H CPU @ 2.60GHz (8 x 1426)

GPU Status
2d_canvas: unavailable_softwarecanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: disabled_softwaremultiple_raster_threads: enabled_onopengl: disabled_offrasterization: disabled_softwareraw_draw: disabled_off_okskia_renderer: enabled_onvideo_decode: disabled_softwarevideo_encode: disabled_softwarevulkan: disabled_offwebgl: unavailable_softwarewebgl2: unavailable_softwarewebgpu: disabled_off

Load (avg)
1, 1, 0

Memory (System)
31.13GB (25.80GB free)

Process Argv
--no-sandbox --force-user-env --unity-launch

Screen Reader
no

VM
0%

DESKTOP_SESSION
ubuntu

XDG_CURRENT_DESKTOP
Unity

XDG_SESSION_DESKTOP
ubuntu

XDG_SESSION_TYPE
x11

Extensions (24)

Extension
Author (truncated)
Version

vscode-intelephense-client
bme
1.9.5

better-toml
bun
0.3.2

path-intellisense
chr
2.8.4

gc-excelviewer
Gra
4.2.56

vscode-graphviz
joa
0.0.6

rainbow-csv
mec
3.5.0

php-namespace-resolver
Meh
1.1.9

vscode-docker
ms-
1.23.3

isort
ms-
2022.8.0

python
ms-
2023.2.0

jupyter
ms-
2023.1.2010391206

jupyter-keymap
ms-
1.0.0

jupyter-renderers
ms-
1.0.14

vscode-jupyter-cell-tags
ms-
0.1.6

vscode-jupyter-slideshow
ms-
0.1.5

remote-containers
ms-
0.275.1

remote-ssh
ms-
0.96.0

remote-ssh-edit
ms-
0.84.0

remote-explorer
ms-
0.2.0

php-docblocker
nei
2.7.0

vscode-yaml
red
1.11.0

bash-debug
rog
0.3.9

code-spell-checker
str
2.16.0

php-debug
xde
1.31.1

Since this is intermittent, I assume you will need logs of some sort. I am happy to support.

Thanks, yeah will be hard to reproduce.
The logs are found in the Pty Host output channel with log level set to trace

Can you be a little more specific?  I don't know what the PTY Host output channel is and how to set it to trace?

In the Output view, select Pty Host

In the command palette, run this command and select Trace

Thank you for the detailed instruction. I was able to set my developer log level to trace, but was not able to see the pty host output channel. I just have normal vs code. Is it an extension? because on my sidebar, I have explorer and search but no pty host. I also did some googling but could not find anything.

This command and then there will be a dropdown

Got it. So do you want for me to wait for it to happen again and then send you the logs? ptyhost.log file only has a day or so of logs. I have not had the issue in that time. It usually happens once a week or so.

Yes, when it next happens, the logs would be great

Keeping this issue open would be great. It happen across to 2 OS's ubuntu 20.02 and 22.04. I have been watching but it has not happened yet/

So it finally happened again. Here is the last bit of the ptyhost.log

2023-03-19 14:41:21.348 [warning] Couldn't get layout info, a terminal was probably disconnected Could not find pty on pty host
2023-03-19 14:41:21.349 [warning] Couldn't get layout info, a terminal was probably disconnected Could not find pty on pty host
2023-03-19 14:41:21.349 [warning] Couldn't get layout info, a terminal was probably disconnected Could not find pty on pty host
2023-03-19 14:41:26.045 [warning] Persistent process "1": Process had no disconnect runners but was an orphan
2023-03-19 14:41:26.045 [info] Persistent process reconnection "1"
2023-03-19 14:41:26.048 [warning] Persistent process "2": Process had no disconnect runners but was an orphan
2023-03-19 14:41:26.048 [info] Persistent process reconnection "2"
2023-03-19 14:41:26.130 [info] Persistent process "1": Replaying 2672 chars and 1 size events
2023-03-19 14:41:26.188 [info] Persistent process "2": Replaying 5847 chars and 1 size events

Happened again today:
2023-03-21 14:11:58.789 [warning] Persistent process "1": Process had no disconnect runners but was an orphan
2023-03-21 14:11:58.789 [info] Persistent process reconnection "1"
2023-03-21 14:11:58.808 [info] Persistent process "1": Replaying 2621 chars and 1 size events

Happened again. Appeared to recover 4 terminals but 1 was frozen.
2023-03-22 14:19:28.599 [warning] Couldn't get layout info, a terminal was probably disconnected Could not find pty on pty host
2023-03-22 14:19:32.719 [warning] Persistent process "3": Process had no disconnect runners but was an orphan
2023-03-22 14:19:32.720 [info] Persistent process reconnection "3"
2023-03-22 14:19:32.748 [info] Persistent process "3": Replaying 5243 chars and 1 size events
2023-03-22 14:19:32.777 [warning] Persistent process "3": Process had no disconnect runners but was an orphan
2023-03-22 14:19:32.777 [info] Persistent process reconnection "3"
2023-03-22 14:19:32.783 [info] Persistent process "3": Replaying 5243 chars and 1 size events
2023-03-22 14:19:36.718 [warning] Persistent process "2": Process had no disconnect runners but was an orphan
2023-03-22 14:19:36.719 [info] Persistent process reconnection "2"
2023-03-22 14:19:36.719 [warning] Persistent process "1": Process had no disconnect runners but was an orphan
2023-03-22 14:19:36.720 [info] Persistent process reconnection "1"
2023-03-22 14:19:36.738 [info] Persistent process "2": Replaying 2689 chars and 1 size events
2023-03-22 14:19:36.742 [info] Persistent process "1": Replaying 2878 chars and 1 size events

I have not encountered the problem in several months.

Let us know if you see it again.