Vagrant: error in task [init-examples | Deploy the ace job]

Unable to run successfully run Scale within Vagrant within CentOS 7 with Ansible 1.9.6 (does 1.9.x version still matter?) and Vagrant 1.8.1 and VirtualBox 5.0.20. I hit a few issues prior to the one listed below. I hit them on both the '3.0.0' branch/tag and master, now attempting solely from master.

After hitting out of storage issues in VMs, ran vagrant up --no-provision to bring up VMs, then ssh to each (scale-master, scale-slave1, and scale-slave2) and ran btrfs fi show to verify re-balancing of filesystem chunks was needed, then btrfs fi balance start -dusage=5 / to relocate chunks with less than 5% of usage, reference btrfs troubleshooting here
After troubleshooting btrfs ran vagrant up provision for second attempt, got a failure in task [migrate | migrate db changes] due to docker image '<IP_ADDRESS>:5000/geoint/scale:3.0.0-snapshot' not being found. After ssh into scale-master was able to manually fix with docker tag <scale id> <IP_ADDRESS>:5000/geoint/scale:3.0.0-snapshot. Another vagrant up provision made it past with something like the following: TASK: [migrate | migrate db changes] ******************************************
<<IP_ADDRESS>> REMOTE_MODULE docker command=migrate docker_api_version=1.20 state=started image=<IP_ADDRESS>:5000/geoint/scale:3.0.0-snapshot
changed: [scale-master] => {"ansible_facts": {"docker_containers": [{"Id": "6c76d8f37440123fdd57b3672e36e975db69418cc52c1219a26aa61c3de0b337", "Warnings": null}]}, "changed": true, "containers": [{"Id": "6c76d8f37440123fdd57b3672e36e975db69418cc52c1219a26aa61c3de0b337", "Warnings": null}], "msg": "started 1 container, created 1 container.", "reload_reasons": null, "summary": {"created": 1, "killed": 0, "pulled": 0, "removed": 0, "restarted": 0, "started": 1, "stopped": 0}}
However, vagrant up provision now is stuck on task [init-examples | Deploy the ace job] with the following issue:
TASK: [init-examples | Deploy the ace job] ************************************
<<IP_ADDRESS>> REMOTE_MODULE command /scale/bin/goscale jobs deploy chdir="/scale/dockerfiles/examples/ace"
failed: [scale-master] => {"changed": true, "cmd": ["/scale/bin/goscale", "jobs", "deploy"], "delta": "0:00:00.067436", "end": "2016-06-15 15:57:10.853385", "rc": 2, "start": "2016-06-15 15:57:10.785949", "warnings": []}
stderr: panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x10 pc=0x46dbbf]

goroutine 1 [running]:
panic(0x8d23e0, 0xc82000e0e0)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli.HandleAction.func1(0xc820121e08)
/scale/src/github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli/app.go:474 +0x417
panic(0x8d23e0, 0xc82000e0e0)
/usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/ngageoint/scale/scale-cli/vendor/github.com/ngageoint/scale/scale-cli.GetJobTypes(0xc820144a60, 0x17, 0xc8201a65d8, 0x3, 0x0, 0x0, 0x0, 0x7f3c50cd0e50, 0xc820112d50)
/scale/src/github.com/ngageoint/scale/scale-cli/vendor/github.com/ngageoint/scale/scale-cli/jobs.go:103 +0x51f
main.jobs_deploy(0xc82000d2c0, 0x0, 0x0)
/scale/src/github.com/ngageoint/scale/scale-cli/cmd/goscale/jobs.go:453 +0x1047
reflect.Value.call(0x828640, 0xa41788, 0x13, 0x963f00, 0x4, 0xc820121d68, 0x1, 0x1, 0x0, 0x0, ...)
/usr/local/go/src/reflect/value.go:435 +0x120d
reflect.Value.Call(0x828640, 0xa41788, 0x13, 0xc820121d68, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/reflect/value.go:303 +0xb1
github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli.HandleAction(0x828640, 0xa41788, 0xc82000d2c0, 0x0, 0x0)
/scale/src/github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli/app.go:483 +0x2ee
github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli.Command.Run(0x9666b8, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0xa1b8c0, 0x59, 0x0, ...)
/scale/src/github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli/command.go:186 +0x1301
github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli.(_App).RunAsSubcommand(0xc820001800, 0xc82000d040, 0x0, 0x0)
/scale/src/github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli/app.go:357 +0x10d5
github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli.Command.startApp(0x967420, 0x4, 0x0, 0x0, 0xc82010e8b0, 0x1, 0x1, 0x9c82c0, 0x19, 0x0, ...)
/scale/src/github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli/command.go:273 +0xa30
github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli.Command.Run(0x967420, 0x4, 0x0, 0x0, 0xc82010e8b0, 0x1, 0x1, 0x9c82c0, 0x19, 0x0, ...)
/scale/src/github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli/command.go:74 +0x7b
github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli.(_App).Run(0xc820001680, 0xc82000a240, 0x3, 0x3, 0x0, 0x0)
/scale/src/github.com/ngageoint/scale/scale-cli/vendor/github.com/codegangsta/cli/app.go:237 +0xa9c
main.main()
/scale/src/github.com/ngageoint/scale/scale-cli/cmd/goscale/main.go:110 +0x10fd
FATAL: all hosts have already failed -- aborting

Do we have a blurb in the docs stating Mesosphere DCOS CE is the preferred install?  The vagrant scripts seem crusty.

We are no longer supporting the vagrant/ansible configuration, but will be shortly updating our quick start documentation to encompass installation to both a local environment (DCOS-Vagrant) in addition to the currently documented AWS environment. This is being done under #620 and #766.


[SPARK-24601] Bump Jackson version

Hi all,
Jackson is incompatible with upstream versions, therefore bump the Jackson version to a more recent one. I bumped into some issues with Azure CosmosDB that is using a more recent version of Jackson. This can be fixed by adding exclusions and then it works without any issues. So no breaking changes in the API's.
I would also consider bumping the version of Jackson in Spark. I would suggest to keep up to date with the dependencies, since in the future this issue will pop up more frequently.
What changes were proposed in this pull request?
Bump Jackson to 2.9.6
How was this patch tested?
Compiled and tested it locally to see if anything broke.
Please review http://spark.apache.org/contributing.html before opening a pull request.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

The tests will be already failed given my tries before and there are many concerns about this. Also see and address the concerns in https://github.com/apache/spark/pull/9759.

I agree we should upgrade, but just changing the pom version will not work. I have an internal patch for this and I needed small changes in a few other places...

Which is, btw, a way of saying you should run dev/run-tests locally, at least, when changing a dependency.

I could not get the tests working locally. :-) Let me give it another try.

ok to test

Test build #92166 has started for PR 21596 at commit ba454b0.

Test build #92166 has finished for PR 21596 at commit ba454b0.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92166/
Test FAILed.

Please, make sure that performance doesn't degrade after upgrading Jackson. You can check that by JsonBenchmarks.

https://github.com/apache/spark/pull/20738
Bump jackson from 2.6.7&<IP_ADDRESS> to 2.7.7
Jackson(>=2.7.7) fixes the possibility of missing tail data when the length of the value is in a range
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.7.7
https://github.com/FasterXML/jackson-core/issues/30
spark-shell:
val value = "x" * 3000
val json = s"""{"big": "$value"}"""
spark.sql("select length(get_json_object(\'"+json+"\','$.big'))" ).collect
res0: Array[org.apache.spark.sql.Row] = Array([2991])

expect result : 3000
actual  result : 2991

@cxzl25, 2.7.x has a regression so we had to revert it back. Please see https://github.com/apache/spark/pull/9759.

@Fokko, mind adding the test described in #20738 here too?

Yes, let me add it.

I've pushed the changes, the tests are still running locally.

Test build #92209 has started for PR 21596 at commit dfcc039.

Test build #92209 has finished for PR 21596 at commit dfcc039.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92209/
Test FAILed.

Hmm, I see a relevant failing test:
org.scalatest.exceptions.TestFailedException: "...430","name":"scope1"[,"parent":null]}" did not equal "...430","name":"scope1"[]}"

Test build #92228 has started for PR 21596 at commit 0d2e186.

Ok, I've found the issue:
https://github.com/FasterXML/jackson-module-scala/issues/325
Changing this to NON_ABSENT should fix this.

Test build #92229 has started for PR 21596 at commit 9107328.

Test build #92229 has finished for PR 21596 at commit 9107328.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92229/
Test FAILed.

Oops, a dangling file. Sorry for triggering so many builds guys.

Test build #92230 has started for PR 21596 at commit 0cb0ea5.

Test build #92231 has started for PR 21596 at commit 1fd1d10.

Test build #92228 has finished for PR 21596 at commit 0d2e186.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92228/
Test FAILed.

Merged build finished. Test FAILed.

Test build #92231 has finished for PR 21596 at commit 1fd1d10.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92231/
Test FAILed.

Test build #92230 has finished for PR 21596 at commit 0cb0ea5.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92230/
Test FAILed.

When looking at the history server, we have a similar issue. From at the list command https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92231/testReport/org.apache.spark.deploy.history/HistoryServerSuite/stage_list_json/ :
I've formatted the json, expected:
[{
	"status": "COMPLETE",
	"stageId": 3,
	"attemptId": 0,
	"numTasks": 8,
	"numActiveTasks": 0,
	"numCompleteTasks": 8,
	"numFailedTasks": 0,
	"numKilledTasks": 0,
	"numCompletedIndices": 8,
	"executorRunTime": 162,
	"executorCpuTime": 0,
	"submissionTime": "2015-02-03T16:43:07.191GMT",
	"firstTaskLaunchedTime": "2015-02-03T16:43:07.191GMT",
	"completionTime": "2015-02-03T16:43:07.226GMT",
	"inputBytes": 160,
	"inputRecords": 0,
	"outputBytes": 0,
	"outputRecords": 0,
	"shuffleReadBytes": 0,
	"shuffleReadRecords": 0,
	"shuffleWriteBytes": 0,
	"shuffleWriteRecords": 0,
	"memoryBytesSpilled": 0,
	"diskBytesSpilled": 0,
	"name": "count at <console>:17",
	"details": "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line19.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)\n$line19.$read$$iwC$$iwC$$iwC.<init>(<console>:22)\n$line19.$read$$iwC$$iwC.<init>(<console>:24)\n$line19.$read$$iwC.<init>(<console>:26)\n$line19.$read.<init>(<console>:28)\n$line19.$read$.<init>(<console>:32)\n$line19.$read$.<clinit>(<console>)\n$line19.$eval$.<init>(<console>:7)\n$line19.$eval$.<clinit>(<console>)\n$line19.$eval.$print(<console>)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:606)\norg.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:852)\norg.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1125)\norg.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:674)\norg.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:705)\norg.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:669)",
	"schedulingPool": "default",
	"rddIds": [6, 5],
	"accumulatorUpdates": [],
	"killedTasksSummary": {}
}, ... ]

found:
[{
	"status": "COMPLETE",
	"stageId": 3,
	"attemptId": 0,
	"numTasks": 8,
	"numActiveTasks": 0,
	"numCompleteTasks": 8,
	"numFailedTasks": 0,
	"numKilledTasks": 0,
	"numCompletedIndices": 8,
	"executorRunTime": 162,
	"executorCpuTime": 0,
	"submissionTime": "2015-02-03T16:43:07.191GMT",
	"firstTaskLaunchedTime": "2015-02-03T16:43:07.191GMT",
	"completionTime": "2015-02-03T16:43:07.226GMT",
	"failureReason": null,
	"inputBytes": 160,
	"inputRecords": 0,
	"outputBytes": 0,
	"outputRecords": 0,
	"shuffleReadBytes": 0,
	"shuffleReadRecords": 0,
	"shuffleWriteBytes": 0,
	"shuffleWriteRecords": 0,
	"memoryBytesSpilled": 0,
	"diskBytesSpilled": 0,
	"name": "count at <console>:17",
	"description": null,
	"details": "org.apache.spark.rdd.RDD.count(RDD.scala:910)\n$line19.$read$$iwC$$iwC$$iwC$$iwC.<init>(<console>:17)\n$line19.$read$$iwC$$iwC$$iwC.<init>(<console>:22)\n$line19.$read$$iwC$$iwC.<init>(<console>:24)\n$line19.$read$$iwC.<init>(<console>:26)\n$line19.$read.<init>(<console>:28)\n$line19.$read$.<init>(<console>:32)\n$line19.$read$.<clinit>(<console>)\n$line19.$eval$.<init>(<console>:7)\n$line19.$eval$.<clinit>(<console>)\n$line19.$eval.$print(<console>)\nsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\nsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)\nsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\njava.lang.reflect.Method.invoke(Method.java:606)\norg.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:852)\norg.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1125)\norg.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:674)\norg.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:705)\norg.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:669)",
	"schedulingPool": "default",
	"rddIds": [6, 5],
	"accumulatorUpdates": [],
	"tasks": null,
	"executorSummary": null,
	"killedTasksSummary": {}
}, ... ]

We observe that the "failureReason": null, is listed, while it should have been omitted.

@Fokko, btw, I believe https://github.com/apache/spark/pull/21596#issuecomment-399160931 is a valid comment. I think it would be nicer if we execute this benchmark and update the results in the file, and push it into here.

BTW, I believe it's okay to use Jenkins resource if that's a faster way to resolve this issue in any event.

re: https://github.com/apache/spark/pull/21596#issuecomment-399639768
can we keep the previous behaviour although it's by kind of a bug in Jackson? BTW, I thought we should use NON_EMPTY instead. Might be worth to double check using NON_ABSENT.

I will first fix the tests, and then assert the performance if you agree. Changing the annotations might impact the performance.
Regarding the difference between NON_NULL, NON_ABSENT, NON_EMPTY, please refer to Jackson:  https://fasterxml.github.io/jackson-annotations/javadoc/2.7/com/fasterxml/jackson/annotation/JsonInclude.Include.html
The NON_EMPTY makes the most sense to me. I agree that we should keep the behaviour like it is right now.
NON_EMPTY:  Value that indicates that only properties with null value, or what is considered empty, are not to be included.

I will first fix the tests, and then assert the performance if you agree. Changing the annotations might impact the performance.

Ah, yea sure.

Test build #92246 has started for PR 21596 at commit 86fdd87.

Test build #92246 has finished for PR 21596 at commit 86fdd87.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92246/
Test FAILed.

retest this please

Test build #92249 has finished for PR 21596 at commit 86fdd87.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92249/
Test FAILed.

Merged build finished. Test FAILed.

So now it trims too much, we get:
[ {
  "id" : "local-1422981759269",
  "name" : "Spark shell",
  "attempts" : [ {
    "startTime" : "2015-02-03T16:42:38.277GMT",
    "endTime" : "2015-02-03T16:42:46.912GMT",
    "lastUpdated" : "",
    "duration" : 8635,
    "sparkUser" : "irashid",
    "completed" : true,
    "appSparkVersion" : "",
    "lastUpdatedEpoch" : 0,
    "startTimeEpoch" :<PHONE_NUMBER>277,
    "endTimeEpoch" :<PHONE_NUMBER>912
  } ]
} ]

instead of:
[ {
  "id" : "local-1422981759269",
  "name" : "Spark shell",
  "attempts" : [ {
    "startTime" : "2015-02-03T16:42:38.277GMT",
    "endTime" : "2015-02-03T16:42:46.912GMT",
    "lastUpdated" : "",
    "duration" : 8635,
    "sparkUser" : "irashid",
    "completed" : true,
    "lastUpdatedEpoch" : 0,
    "endTimeEpoch" :<PHONE_NUMBER>912,
    "startTimeEpoch" :<PHONE_NUMBER>277
  } ]
} ]

Look at appSparkVersion

Test build #92254 has started for PR 21596 at commit 1c5ea8a.

Test build #92254 has finished for PR 21596 at commit 1c5ea8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92254/
Test PASSed.

@MaxGekk Any pointers on how to build this test-jar containing spark sql test-classes?

mvn package -pl sql/core -DskipTests should build such jar

Test build #92268 has started for PR 21596 at commit 4dd812a.

Test build #92270 has started for PR 21596 at commit 95840a5.

Test build #92270 has finished for PR 21596 at commit 95840a5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92270/
Test FAILed.

Merged build finished. Test FAILed.

Thanks @MaxGekk for pointing out. I've also added the command to the comments in the test for future reference.
I've ran the benchmark. The inferring became a bit slower (0.85x relative), but the parsing much faster, 2.59x relative for short lines, and 2.50x relative for wide lines. Keep in mind that the old benchmark ran with a slightly faster cpu.

Test build #92271 has started for PR 21596 at commit 92a78aa.

Test build #92268 has finished for PR 21596 at commit 4dd812a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92268/
Test PASSed.

Test build #92271 has finished for PR 21596 at commit 92a78aa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Can we target this to Spark 3.0, which should be the next release after Spark 2.4 release?

Test build #92296 has started for PR 21596 at commit d4a4326.

Test build #92296 has finished for PR 21596 at commit d4a4326.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92296/
Test PASSed.

I would preferably get this in Spark 2.4 :-)

Test build #92337 has started for PR 21596 at commit 40c29cb.

Test build #92337 has finished for PR 21596 at commit 40c29cb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92337/
Test PASSed.

Could you write a more descriptive summary? At least including the version.
Your benchmark number updates don't really make a lot of sense. I think the idea of those is to post a comment - or put that in the PR summary - with the benchmark run on your machine before and after the change. Otherwise the numbers aren't really comparable, since your machine is different from the one where the results that are checked in were generated.

Also please use the PR summary to describe the change, not for asking questions or other things that are more suited for comments / Jira.

Test build #92419 has started for PR 21596 at commit fd45cb4.

Test build #92419 has finished for PR 21596 at commit fd45cb4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

It took some time, but here it is, current master:
MacBook-Pro-van-Fokko:dist fokkodriesprong$ cat /tmp/spark-master.txt
Preparing data for benchmarking ...
Running benchmark: JSON schema inferring
  Running case: No encoding
  Stopped after 3 iterations, 153857 ms
  Running case: UTF-8 is set
  Stopped after 3 iterations, 210849 ms

Java HotSpot(TM) 64-Bit Server VM 1.8.0_172-b11 on Mac OS X 10.13.5
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz

JSON schema inferring:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
No encoding                                 43607 / 51286          2,3         436,1       1,0X
UTF-8 is set                                69107 / 70283          1,4         691,1       0,6X

Preparing data for benchmarking ...
Running benchmark: JSON per-line parsing
  Running case: No encoding
  Stopped after 3 iterations, 32557 ms
  Running case: UTF-8 is set
  Stopped after 3 iterations, 52323 ms

Java HotSpot(TM) 64-Bit Server VM 1.8.0_172-b11 on Mac OS X 10.13.5
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz

JSON per-line parsing:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
No encoding                                 10157 / 10853          9,8         101,6       1,0X
UTF-8 is set                                17310 / 17441          5,8         173,1       0,6X

Preparing data for benchmarking ...
Running benchmark: JSON parsing of wide lines
  Running case: No encoding
  Stopped after 3 iterations, 74416 ms
  Running case: UTF-8 is set
  Stopped after 3 iterations, 88192 ms

Java HotSpot(TM) 64-Bit Server VM 1.8.0_172-b11 on Mac OS X 10.13.5
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz

JSON parsing of wide lines:              Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
No encoding                                 24311 / 24806          0,4        2431,1       1,0X
UTF-8 is set                                27971 / 29398          0,4        2797,1       0,9X

New Jackson:
Java HotSpot(TM) 64-Bit Server VM 1.8.0_172-b11 on Mac OS X 10.13.5
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz

JSON schema inferring:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
No encoding                                 45615 / 46109          2,2         456,2       1,0X
UTF-8 is set                                67516 / 69291          1,5         675,2       0,7X

Preparing data for benchmarking ...
Running benchmark: JSON per-line parsing
  Running case: No encoding
  Stopped after 3 iterations, 36206 ms
  Running case: UTF-8 is set
  Stopped after 3 iterations, 63400 ms

Java HotSpot(TM) 64-Bit Server VM 1.8.0_172-b11 on Mac OS X 10.13.5
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz

JSON per-line parsing:                   Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
No encoding                                 11803 / 12069          9,2         118,0       1,0X
UTF-8 is set                                19500 / 21134          5,1         195,0       0,6X

Preparing data for benchmarking ...
Running benchmark: JSON parsing of wide lines
  Running case: No encoding
  Stopped after 3 iterations, 69479 ms
  Running case: UTF-8 is set
  Stopped after 3 iterations, 82703 ms

Java HotSpot(TM) 64-Bit Server VM 1.8.0_172-b11 on Mac OS X 10.13.5
Intel(R) Core(TM) i7-7820HQ CPU @ 2.90GHz

JSON parsing of wide lines:              Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------
No encoding                                 23044 / 23160          0,4        2304,4       1,0X
UTF-8 is set                                27194 / 27568          0,4        2719,4       0,8X

Test build #92478 has started for PR 21596 at commit 82e1fce.

Test build #92478 has finished for PR 21596 at commit 82e1fce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92478/
Test PASSed.

What is the conclusion of this perf comparison? A regression?
If we do not have a strong reason, we should avoid the upgrade. Jackson is pretty important for our JSON sources.

@gatorsmile The obvious regression is in schema inferring benchmarks but in other cases there is significant performance boost even on slower hardware.  We will see the difference on the same hardware.

@MaxGekk I've reran the benchmarks as you asked. The first run to set the baseline based on master: https://github.com/apache/spark/pull/21596/commits/32ec12bdfe1fd9377c0f4037942da9e2a49e8f9a
And after switching to Jackson 2.9.6:
https://github.com/Fokko/spark/blob/8dbc310ac6a8bcd2eb9f583b0a9990f9bed2aea0/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonBenchmarks.scala
@gatorsmile One of the reasons is mentioned in the starting post. I have to add exclusions to the Cosmos Spark connector because the Spark Jackson version is lagging behind. Would be nice to keep a bit up.

Test build #92537 has started for PR 21596 at commit 8dbc310.

Personally I'm not sure how representative the tests are. I have the feeling there is a lot of variation between the runs. Also it is not just parsing the plain JSON, but everything around it :)

Test build #92537 has finished for PR 21596 at commit 8dbc310.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92537/
Test PASSed.

Test build #92554 has started for PR 21596 at commit 5006467.

I can do a 5 runs before and after the patch, and come up with some statistics such as average and stddev, to get rid of the uncertainty a bit.

Test build #92554 has finished for PR 21596 at commit 5006467.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92554/
Test FAILed.

Test build #92558 has started for PR 21596 at commit 4b78651.

LGTM

Test build #92607 has started for PR 21596 at commit 61c6dd3.

@srowen I understand the situation, looking at the fasterxml dependency:
Not used by Hadoop 2.6: https://hadoop.apache.org/docs/r2.6.5/hadoop-project-dist/hadoop-hdfs/dependency-analysis.html
Not used by Hadoop 2.7: https://hadoop.apache.org/docs/r2.7.6/hadoop-project-dist/hadoop-hdfs/dependency-analysis.html
Not used by Hadoop 2.8: https://hadoop.apache.org/docs/r2.8.4/hadoop-project-dist/hadoop-hdfs/dependency-analysis.html
Version 2.7.8 used by Hadoop 2.9: https://hadoop.apache.org/docs/r2.9.1/hadoop-project-dist/hadoop-hdfs/dependency-analysis.html
Version 2.7.8 used by Hadoop 3.0: https://hadoop.apache.org/docs/r3.0.3/hadoop-project-dist/hadoop-hdfs/dependency-analysis.html
Version 2.7.8 used by Hadoop 3.1: https://hadoop.apache.org/docs/r3.1.0/hadoop-project-dist/hadoop-hdfs/dependency-analysis.html
NON_ABSENT has been introduced in Version 2.6 of fasterxml, so that should not be a problem: https://fasterxml.github.io/jackson-annotations/javadoc/2.6/com/fasterxml/jackson/annotation/JsonInclude.Include.html
It could be that different versions are pulled in transitively, but this is hard to tell without diving in into the dependency trees. But when Hadoop itself is set a version, I would not expect it to be overridden with a lower version.

Test build #92607 has finished for PR 21596 at commit 61c6dd3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92607/
Test PASSed.

Users can (perhaps should) be shading Jackson but I bet most won't.

Would it be better to shade Jackson on Spark side? In that case, we will have more space for manoeuvres in the future. For now Spark even cannot switch to a version with critical/security fixes if we don't want to break user's apps (the apps can use non public jackson's api).

Yes, long term that is the right thing to do. It still introduces a user-visible classpath change, which is why I'm on the fence about doing so in Spark 2.4. I feel like several people out there use Jackson in their app and probably end up inheriting the Spark version and then it disappears for them. Not impossible, but needs to be weighed against the upside: is it some medium-sized bug fixes and performance gain? tough call.
I'd be all for resolving this in Spark 3.0 though.

Rebased onto master

Test build #92847 has started for PR 21596 at commit 4fcf9b7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92847/
Test PASSed.

Rebased onto master

Test build #93232 has started for PR 21596 at commit 7d4ac0b.

Test build #93232 has finished for PR 21596 at commit 7d4ac0b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93232/
Test PASSed.

Test build #93544 has started for PR 21596 at commit 5742678.

@Fokko Let us wait for the code freeze of Spark 2.4 release?

@gatorsmile Sure, just checking if it still works against recent master :)

Test build #93544 has finished for PR 21596 at commit 5742678.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93544/
Test FAILed.

Feels like a flaky test in the KafkaContinuousSinkSuite. Lets rerun :-)

Test build #93569 has started for PR 21596 at commit e16f7a1.

Test build #93569 has finished for PR 21596 at commit e16f7a1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93569/
Test PASSed.

Are we still waiting for the 2.4 code freeze @gatorsmile @Fokko ?

@jerryshao I believe the freeze is still pending because some Scala 2.12 compatibility issues: http://apache-spark-developers-list.1001551.n3.nabble.com/code-freeze-and-branch-cut-for-Apache-Spark-2-4-td24365i20.html
Also, no branch has been cut yet, so I think we have to wait a bit longer :-)

I've rebased, just to check if all the tests are still ok against latest master.

Test build #94347 has started for PR 21596 at commit c1f96d9.

Test build #94347 has finished for PR 21596 at commit c1f96d9.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94347/
Test FAILed.

Jenkins, retest this please.

Test build #94350 has started for PR 21596 at commit c1f96d9.

Test build #94350 has finished for PR 21596 at commit c1f96d9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94350/
Test FAILed.

@jerryshao This is for 3.0

Test build #94410 has started for PR 21596 at commit f43de17.

Test build #94410 has finished for PR 21596 at commit f43de17.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94410/
Test PASSed.

Test build #95161 has started for PR 21596 at commit 5f887e1.

I've rebased onto master 👍

Test build #95161 has finished for PR 21596 at commit 5f887e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95161/
Test PASSed.

Test build #95866 has started for PR 21596 at commit 4f9a9c5.

@HyukjinKwon I've rebased onto master since the Spark 2.4 branch has been cut.

FYI - I have found nondeterministic flakes with RDDOperationScope in newer jackson, you can see fix at https://github.com/palantir/spark/pull/379. What happens is that jackson object mapper doesn't apply option serializer to parent field resulting in wrong json representation. Unfortunately I haven't been able to isolate the issue.

Thanks @robert3005 But also API calls are failing, so it is not isolated to the RDDOperationScope. I'm compiling Spark locally to check what the API calls are returning.

Test build #95894 has started for PR 21596 at commit bb8b661.

Test build #95894 has finished for PR 21596 at commit bb8b661.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95894/
Test FAILed.

retest this please

Test build #95919 has started for PR 21596 at commit bb8b661.

Test build #95919 has finished for PR 21596 at commit bb8b661.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95919/
Test FAILed.

Test build #95929 has started for PR 21596 at commit 98d459f.

Some of the JSON endpoints are invalid, for example:
{
	"info": {
		"id": "driver",
		"hostPort": "localhost:61103",
		"isActive": true,
		"rddBlocks": 0,
		"memoryUsed": 0,
		"diskUsed": 0,
		"totalCores": 0,
		"maxTasks": 0,
		"activeTasks": 0,
		"failedTasks": 0,
		"completedTasks": 0,
		"totalTasks": 0,
		"totalDuration": 0,
		"totalGCTime": 0,
		"totalInputBytes": 0,
		"totalShuffleRead": 0,
		"totalShuffleWrite": 0,
		"isBlacklisted": false,
		"maxMemory": 278019440,
		"addTime":<PHONE_NUMBER>880,
		"executorLogs": {},
		"blacklistedInStages": [],
		"peakMemoryMetrics"
	}
}

Some of the code has been introduced here: https://github.com/apache/spark/commit/9241e1e7e66574cfafa68791771959dfc39c9684

Test build #95932 has started for PR 21596 at commit 627f611.

Test build #95932 has finished for PR 21596 at commit 627f611.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95932/
Test FAILed.

Test build #95929 has finished for PR 21596 at commit 98d459f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95929/
Test FAILed.

Test build #95940 has started for PR 21596 at commit f444a32.

Test build #95942 has started for PR 21596 at commit 322ef7f.

Test build #95943 has started for PR 21596 at commit 278ff11.

Test build #95946 has started for PR 21596 at commit 97192d2.

I've rebase onto master, and fixed the test. Lets wait for the CI's opinion of the fix.
The problem was the following, introduced in https://github.com/apache/spark/pull/21221:
``
scala> None == null
res0: Boolean = false

We need to override the isEmpty of the serializer to verify if the `Option[_]` is Empty or not.

Test build #95940 has finished for PR 21596 at commit f444a32.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95940/
Test FAILed.

Test build #95942 has finished for PR 21596 at commit 322ef7f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95942/
Test FAILed.

Test build #95943 has finished for PR 21596 at commit 278ff11.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95943/
Test FAILed.

Test build #95960 has started for PR 21596 at commit f093c39.

Test build #95946 has finished for PR 21596 at commit 97192d2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95946/
Test FAILed.

Test build #95960 has finished for PR 21596 at commit f093c39.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95960/
Test PASSed.

@HyukjinKwon All green again 👍

Test build #96142 has started for PR 21596 at commit b889a9d.

Test build #96142 has finished for PR 21596 at commit b889a9d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96142/
Test PASSed.

Test build #96330 has started for PR 21596 at commit 44b8d1b.

@HyukjinKwon Any idea if this has any change of getting merged? The 2.4 branch has been cut a while ago.
The number of merge conflicts are minimal in all the times I've rebased onto master. I think this should be done some time to bump the Jackson version, since old version of Jackson aren't compatible with the newer ones. Please let me know.

Test build #96330 has finished for PR 21596 at commit 44b8d1b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96330/
Test FAILed.

Test build #96335 has started for PR 21596 at commit 0b0b8fc.

I think we're (at least i am) not clear on 2.5 vs 3.0 now (correct me if im mistaken). The change itself looks okay but I think it's targeted to 3.0.0 as discussed. I can take an action here when I'm clear on this.

Test build #96335 has finished for PR 21596 at commit 0b0b8fc.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96335/
Test FAILed.

Test build #96413 has started for PR 21596 at commit 2bab06f.

Test build #96413 has finished for PR 21596 at commit 2bab06f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Rebased onto master. @HyukjinKwon can we target this for 2.5?

Test build #96926 has started for PR 21596 at commit b53c8f2.

Test build #96926 has finished for PR 21596 at commit b53c8f2.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96926/
Test FAILed.

retest this please

Test build #96928 has started for PR 21596 at commit b53c8f2.

Test build #96928 has finished for PR 21596 at commit b53c8f2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96928/
Test PASSed.

Merged to master.

It sounds like Jackson have an important behavior change that might impact us.
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper

val mapper = new ObjectMapper with ScalaObjectMapper
mapper.setSerializationInclusion(Include.NON_NULL)
mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
mapper.registerModule(DefaultScalaModule)

println(mapper.writeValueAsString(Map("abc" -> null)))

Previously, it outputs {"abc":null}, but now it is {}, unless we explicitly set it to Include.ALWAYS. The upgrade looks risky to me.

@gatorsmile Are you completely sure about that? The Spark REST API null values are always omitted. Where is this behaviour different?
mapper.setSerializationInclusion(Include.NON_ABSENT):
scala> println(mapper.writeValueAsString(Map("abc" -> null)))
{}

mapper.setSerializationInclusion(Include.ALWAYS):
scala> println(mapper.writeValueAsString(Map("abc" -> null)))
{"abc":null}

If you're doing proper Scala and try to avoid the null's in general, with None in combination with mapper.setSerializationInclusion(Include.NON_ABSENT) you'll get:
scala> println(mapper.writeValueAsString(Map("abc" -> None)))
{"abc":null}


feat(Card): productive card stacked body tiles

Contributes to #3766
The requirement is that there would be multiple clickable sections within the tile.
What did you change?

Added a new story for Stacked type productive card
Updated Card component to accept stacked-type body component

How did you test and verify your work?
Storybook

@Ratheeshrajan any update here?

Closing this for now and will move this issue back to backlog/triage for now. These files also have significant changes since we’ve migrated to Typescript and introduced additional card variants.


Push notifications deeplink not wrapped

Hi,
I'm currently trying to track push notification opens by following the documentation and integrating the Parcelvoy library on iOS. However, I noticed that the URL passed to the native side is not wrapped, which prevents the native library from sending the HTTP request to the Parcelvoy endpoint to track the push notification opening.
After checking the platform code, I found that the paramsToEncodedLink function (https://github.com/parcelvoy/platform/blob/a180ec4ced2ef20409088d272402fad380cacd6e/apps/platform/src/render/LinkService.ts#L22C14-L22C33) is only used in the subscription service to handle the unsubscribe email link (https://github.com/parcelvoy/platform/blob/a180ec4ced2ef20409088d272402fad380cacd6e/apps/platform/src/subscriptions/SubscriptionService.ts#L146).
I plan to create a pull request to wrap push notification deeplinks before sending them to the services. However, I wanted to check first: is there a specific reason this hasn’t been done yet, or is there another recommended approach?
Thank you!

Overall definitely an addition that could be added! Had mostly been intending that field to be used for deeplinks but you're right, you can open it up for anything if you wanted and wrapping would allow for tracking.
This would be the way I currently think about link wrapping:
Email: Zero problems wrapping, though some people opt out because they are wrapping themselves or are having issues with mobile links
SMS: Never should be wrapped, takes up too many characters and looks super spammy
Push Notification: 50/50 on if it should be wrapped. If you do wrap, it becomes a universal link which then requires the person implementing the app to have set that up correctly (including an assetlinks.json file for Android and a apple-app-site-association for iOS). This can be more work than some people want to do, especially if they are just using deeplinks for navigation and are tracking events at the destination of the navigation.
Webhook: Never, mostly a raw data source
Currently the boolean toggle for link wrapping is universal, it might be that we should switch that to allow other integer values and make it be a segmented control to allow turning on for one or both of those sources (0 = off completely, 2 = both, 3 = email, 4 = notification or something)
The thing I would want to triple check in any implementation is that a link wrapped deep link still works properly. It should, but just something to check!


http links in the signature get hijacked with a nylas.com redirect url

I have just been informed by my employer that the http link in the signature I've added is now a cloaked url that redirects through nylas.com
This is very sneaky looking behaviour (phishing?) and my employer has made it clear that I either need to turn this behaviour off or move to another email client.
How to I disable this http url redirection? I see no benefit to it whatsoever.
What operating system are you using?
... linux
What version of Nylas Mail are you using?
... 2.0.32
Is the issue reproducible with a particular attachment, message, signature, etc?

...
Create a signature using a http:// link for example http://test.com
When I check the raw html in the signature editor of the url I entered, the nylas link is not there - it just says http://test.com. When I use the nylas client, the nylas link is not there either.
But, if I send an email to my gmail account and log in to gmail to view it, the http url in the signature redirects through nylas.com when I hover over the link.
Feature Request?
Does this feature exist in another mail client or tool you use?
...
no

Same things here.
Please, solve this!


Fix bug in line 226

how did i missed that :v
BTW Plan9 support is possible, I am testing right now.


Add a restart button while running elevator

If I start an elevator, I think it needs to run the whole way through before I can reset it. It would be nice to have a button to restart it.

You can use the Apply Button below the code field to reset the current simulation.

@apricote is right. The Apply button restarts the challenge.


[markdown doc page][m] guide for writing markdown

We added a new docs topic, called  misc
There are three new articles in it:

Awesome pages guide - original file is in 'Awesome repo' and will be written due to Awesome Epic
Markdown 5 minutes guide
Making docs for datahub guide

Issue: https://github.com/datahq/pm/issues/79

I just have read this whole 'making-docs-guide' page again - and I don't like it at all. Reasons:

the article is very short - less then one page
info for 'making docs' and 'making blogs' are almost the same
info about FrontMatter are doubled here and on MD-guide page
no links to our repo so user doesn't know where to contribute.

I think this whole page could be replaced by one button like we have on Awesome pages - "edit on github"
I'm removing this page from PR.
@Mikanebu if you want to recreate this page later - we could open an issue for that.


load-balancing and session persistence

What is the recommended way to manage session persistence while running multiple instances of excalidraw as docker containers in a Kubernetes setup.

Also wondering if Redis support is provided in the application

Hey @thecloudgarage, which container are you referring to? The frontend client or the socket server?
The frontend client is basically a frontend app so there is no state/session server-side.
The socket server, well even excalidraw.com runs on a single instance and is able to handle a vast amount of requests. I would be surprised if you needed more than that but you know best of course. I would stick with a single instance of the socket server.
Re: Redis, we run a basic implementation of socket.io which supports persistent sessions using Redis out of the box but we would have to implement it.

I am basically running the docker image, which I believe has both the frontend app and the socket server. Please correct me if I am wrong. In this case, if I want to run multiple instances of the docker image (docker hub: excalidraw/excalidraw), both the frontend app and the socket server., both get scaled out. Is there a way to run the front-end app separately as a docker container and the socket server as a separate docker container. If that's possible, then I can independently scale the frontend app by running multiple containers and as per your suggestion run a single instance of the socket server.
On point 2, i.e. redis, do you have plans to implement that within the application as an option

There are two images atm:

The socket server: https://hub.docker.com/r/excalidraw/excalidraw-room
The frontend client: https://hub.docker.com/r/excalidraw/excalidraw

The latter can be configured through an env variable to point to your self-hosted socket server. Docs are not really up to date, something to fix.
Try it and let us know how it went. :)

On point 2, i.e. redis, do you have plans to implement that within the application as an option

No AFAIC but we could implement it as optional if someone would like to scale up the socket server.

Got it.. thanks a bunch... will try leveraging redis for the socket server (I did not see .env file earlier to the heroku url for socket server.. my bad)... One other question...in the .env file I see references to json server... is that a separate docker image that I can host and any guidance there will be helpful

Will mark this one as done.


Support for packer -var-file

Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: Feature
Enter Task Name: PackerBuild
list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
Environment

Server - Azure Pipelines or TFS on-premises? Azure Pipelines

Agent - Hosted or Private:  Hosted

Issue Description
It would be nice if we could pass a variables file (e.g. vars.json) to packer that lives in the main source so as to keep fewer modifications to the pipeline code. If we are setting component versions in these variables, that means we need to edit the pipeline code.

No replies since 19 february 👍


sdk methods generated as virtual methods

@RicoSuter  This updates the Client.Class.liquid template to make SDK methods generated as virtual methods.
How it could be used from class that inherits generated sdk class:
        public override Task<SwaggerResponse<CompanyFeature>> GetCompanyFeatureByCompanyIdAsync(int companyId, CancellationToken cancellationToken = default)
        {
          //some code needed to extend goes here (ex. maybe some small cache code)
        }

Original output:
        public async System.Threading.Tasks.Task<SwaggerResponse<CompanyFeature>> GetCompanyFeatureByCompanyIdAsync(int companyId, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken))

New output:
        public virtual async System.Threading.Tasks.Task<SwaggerResponse<CompanyFeature>> GetCompanyFeatureByCompanyIdAsync(int companyId, System.Threading.CancellationToken cancellationToken = default(System.Threading.CancellationToken))

@jonlin noticed issue 2704 you previously posted

@RicoSuter added VirtualMethods parameter options in C# generator and made it optional + default to false

@RicoSuter
I noticed your comments on issue #2704 "Currently there is no option for that specifically.. you need to override the template and add the virtual modifier. The question is whether we should add this by default or if we should add an option. But maybe you can change the access modifier from “public” to “public virtual”?"
I was hoping to see if there were any concerns with this feature addition in this pull request. It defaults to existing functionality and has an option to enable the access modifier to be "public virtual" instead of "public".
Again appreciate your time.

@RicoSuter Would you kindly approve the workflow? I'm a first-time contributor.

I wonder whether we really need to add a new setting for that (more complexity) or just add virtual to the template - performance wise these methods are "slow" anyway as they are I/O based and adding virtual is not a breaking change (right?). Opinions? /cc @lahma @jeremyVignelles

I don't see any harm for them being virtual by default. If the API would change it would anyway break the caller and also then the inheritor/override.

OK for virtual by default. You know my opinion about adding an option for each use case :)

@rw-ben ok I agree, let's remove the setting and just add virtual to the templates - then we have a quite simple PR.

@RicoSuter @jeremyVignelles @lahma thank you all for taking a look at this. I will make it default and push the changes up.

@RicoSuter @jeremyVignelles @lahma removed the virtual method C# generation param and it always generate virtual methods

@RicoSuter thanks

Thanks for the PR.

@RicoSuter How do I get a new release built? Is this something you have to kick off?

what about C# properties? they are not virtual is it possible to ovverride them too? (e.g. to give defaults to null/empty)


Update Pods [--no-repo-update]

pod update needs to access the network. When the network is not good, need to wait for a long time.  And with --no-repo-update, it can save the network requests, in most cases we do not need to update the warehouse, we can quickly complete!

Looks good! Just the one change and I'll bring it in.

Merged! Thank you!


cli: popper run fails in Git repo without commits

Steps to reproduce
git init myrepo
cd myrepo
popper scaffold
popper run -f wf.yml

Error message:
Traceback (most recent call last):
  File "/home/living/.local/bin/popper", line 8, in <module>
    sys.exit(cli())
  File "/home/living/.local/popper/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/living/.local/popper/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/living/.local/popper/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/living/.local/popper/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/living/.local/popper/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/living/.local/popper/lib/python3.8/site-packages/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/living/.local/popper/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/living/.local/popper/lib/python3.8/site-packages/popper/commands/cmd_run.py", line 169, in cli
    config = ConfigLoader.load(
  File "/home/living/.local/popper/lib/python3.8/site-packages/popper/config.py", line 52, in load
    "git_commit": scm.get_sha(repo),
  File "/home/living/.local/popper/lib/python3.8/site-packages/popper/scm.py", line 173, in get_sha
    return repo.git.rev_parse(repo.head.object.hexsha)
  File "/home/living/.local/popper/lib/python3.8/site-packages/git/refs/symbolic.py", line 190, in _get_object
    return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
  File "/home/living/.local/popper/lib/python3.8/site-packages/git/refs/symbolic.py", line 132, in dereference_recursive
    hexsha, ref_path = cls._get_ref_info(repo, ref_path)
  File "/home/living/.local/popper/lib/python3.8/site-packages/git/refs/symbolic.py", line 181, in _get_ref_info
    return cls._get_ref_info_helper(repo, ref_path)
  File "/home/living/.local/popper/lib/python3.8/site-packages/git/refs/symbolic.py", line 164, in _get_ref_info_helper
    raise ValueError("Reference at %r does not exist" % ref_path)
ValueError: Reference at 'refs/heads/master' does not exist

Popper version: Current master, in virtual environment with Python 3.8.3
Git version: 2.27.0
Operating system: Arch Linux

Expected behavior
An error message that Popper needs an initial commit, or ignore that we are in a Git repository.

Thanks for reporting @wtraylor . We will push a fix soon.

thanks a lot for the detailed report @wtraylor ! For this we need to modify the cli/popper/scm.py

I would like to work in this @ivotron .


[feature] Send Sol component

Implementation: Send Solana tokens from one wallet to other

https://user-images.githubusercontent.com/77020164/182446594-50ea7022-9e29-480a-a09f-c6f0c0b7eb41.mp4

Failed to compile.
--
13:38:25.115 |
13:38:25.116 | ./src/components/ExplorerCard.tsx:97:22
13:38:25.116 | Type error: Type '{ children: string; tabindex: string; className: string; }' is not assignable to type 'DetailedHTMLProps<LabelHTMLAttributes<HTMLLabelElement>, HTMLLabelElement>'.
13:38:25.116 | Property 'tabindex' does not exist on type 'DetailedHTMLProps<LabelHTMLAttributes<HTMLLabelElement>, HTMLLabelElement>'. Did you mean 'tabIndex'?


Cassandra backend

Keyspace
CREATE KEYSPACE example WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'}  AND durable_writes = true;
Table
CREATE TABLE test (key varchar PRIMARY KEY, value varchar);
Example config where Backend:1 is Cassandra
{
	"Backend": "cassandra",
	"Cassandra": {
		"ProtoVersion": 3,
		"Keyspace": "example",
		"Table": "test",
		"Hosts": ["<IP_ADDRESS>"],
		"Consistency": 4
	}
}

@FlxJaron can you also add something in the readme? Basically what you put in the description of this PR is a good start.

Looks good 👍


Quick Fingers Keyboard

PR Description 📜
Added Quick fingers Keyboard game.
Fixes #1968

Mark the task you have completed ✅

[x] I follow CONTRIBUTING GUIDELINE & CODE OF CONDUCT of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I have followed proper naming convention showed in CONTRIBUTING GUIDELINE
[x] I have added screenshot for website preview in assets/images
[x] I have added entries for my game in main README.md
[x] I have added README.md in my folder
[ ] I have added working video of the game in README.md (optional)
[x] I have specified the respective issue number for which I have requested the new game.

Thank you soo much for contributing to our repository 💗

Thank you @Bhuvanchand26 ,for creating the PR and contributing to our GameZone 💗
Review team will review the PR and will reach out to you soon! 😇
Make sure that you have marked all the tasks that you are done with ✅.
Thank you for your patience! 😀

the game is already there on GameZone !!! 👀

Thank you  @Bhuvanchand26 , for your valuable time and contribution in our GameZone 💗.
It’s our GameZone, so Let’s build this GameZone altogether !!🤝
Hoping to see you soon with another PR again 😇
Wishing you all the best for your journey into Open Source🚀


[FLOC-2235] Add access to Drive requirement

Fixes https://clusterhq.atlassian.net/browse/FLOC-2235

Please address comment and merge when green.

We discussed only needing the docs builder to pass for .rst changes so I won't retry  functional/aws/ubuntu-14.04/storage-driver


🖋 Fix warning message formatting in KTOTrainer

What does this PR do?
Currently you've a indentation when warning is raised
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you read the contributor guideline,
Pull Request section?
[ ] Was this discussed/approved via a GitHub issue? Please add a link
to it if that's the case.
[ ] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines.
[ ] Did you write any new necessary tests?

Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


avoid race condition  or multiple download of same file at same time

Ui already prevent User from clicking download button/ calling download function over and over.
It's an extra layer of protection from mistakes that we as maintainer might make.

is now in separate PR..


Refactor code

Refactor all code

Closing this one as it's solved in up coming release, planning to release it within two days.


DeviceConnectionStateEvents  is missing from IOT management API

Bug Report
DeviceConnectionStateEvents routing source is missing:

Enumeration: https://github.com/Azure/azure-sdk-for-go/blob/862673ef08ed0598660f54db809772c921455c23/services/iothub/mgmt/2020-03-01/devices/enums.go#L326
Possible routing-source values: https://github.com/Azure/azure-sdk-for-go/blob/862673ef08ed0598660f54db809772c921455c23/services/iothub/mgmt/2020-03-01/devices/enums.go#L341

This enumeration was added on May 13 to the Azure SDK (see https://github.com/Azure/azure-sdk-for-net/blob/978837dcd7f8459f4e54ea11fd3934d787b8490b/sdk/iothub/Microsoft.Azure.Management.IotHub/src/Generated/Models/RoutingSource.cs#L24).

Need to refresh the GO SDK to catch the latest change in Swagger.

Hi @ramondeklein thanks for this issue!
Releasing a new package will require the service to submit a new SDK release request, I have added the label on this issue so that the corresponding service team could take this a look.
I also sent some internal emails to the contact of the corresponding service team about this.

iothub stable latest version 2021-03-01 released in: https://github.com/Azure/azure-sdk-for-go/tree/v57.4.0/services/iothub/mgmt/2021-03-31/devices


Add length, truncate, whitespace handling methods for ystring

 Here is an overview of what got changed by this pull request:

Issues
======
- Added 1

Coverage increased per file
===========================
+ ystring/whitespace.go  100
+ ystring/length.go  100

Complexity increasing per file
==============================
- ystring/whitespace_test.go  3
- ystring/whitespace.go  4
- ystring/length_test.go  4
- ystring/length.go  6

See the complete overview on Codacy

 Here is an overview of what got changed by this pull request:

Coverage increased per file
===========================
+ ystring/length.go  100
+ ystring/whitespace.go  100

Complexity increasing per file
==============================
- ystring/whitespace_test.go  3
- ystring/length.go  6
- ystring/length_test.go  4
- ystring/whitespace.go  4

See the complete overview on Codacy


Allow Regular List Args as well as Keyword List Args

What
The builder did not allow the user to pass lists of any type in as arguments. For example:
%Query{
  ...
  variables: [
    ids: [1, 2]
  ]
}

What Changed?
Separated out the encoding of list lists from keyword lists in the builder
Note
Thanks for building this! I would have liked to use Jason.encode! here, but I'm assuming you didn't want to add any dependencies.

Thanks for the contribution. I'll be looking at this over the weekend and hopefully getting a release with this fix!


Feature Request: Levels and Difficulty Modes for Alien Invasion Defense Game.

Levels:

Adding multiple levels with increasing difficulty would create a more engaging experience. Each level could introduce new alien types, challenges, and unique environments.

Difficulty Modes:

Implementing "Easy" and "Hard" modes could cater to a wider audience.

Easy Mode: Suitable for newcomers, with slower enemy spawn rates and more resources.

Hard Mode: For experienced players, featuring faster enemy waves, limited resources, and additional challenges.

I'm expecting a gssoc-ext and a level3 label,  hacktoberfest-accepted.
Assign me!

Can I get your discord?! Or can social to contact you??

https://www.linkedin.com/in/yakhilesh/

https://www.linkedin.com/in/yakhilesh/

the feature is added by another contributer! i'm closing the issue!


File not found dist/index.js error when trying to run action

I'm trying to run the action from the current main branch, rather than the v2 release, since #648 should be fixed in main.
I'm getting the following error:
Run katyo/publish-crates@9766ec7ed8ebbb0d34168814ecacf51833b6f97d
Error: File not found: '/home/runner/work/_actions/katyo/publish-crates/9766ec7ed8ebbb0d34168814ecacf51833b6f97d/dist/index.js'

See the job run.

@daniestevez Because I don't like keep derived products with source code, you should not use such branches in CI. Instead use v* tags or dist/v* branches. Keep in mind this branches and tags updates automatically when CI run complete successfully.

Does the dist/v2 branch contain the fix for #648?

@daniestevez The dist/v2 actually derived from main with by adding build products.
You can see commit log to be sure.


Qualification tool Update and Run integration tests on new version

Is your feature request related to a problem? Please describe.
With the changes to qualification tool output we need to update our integration tests

Updated the expected_results for integration tests on YARN cluster.


Decouple from Scarb

Introduced changes

Checklist

[x] Linked relevant issue
[ ] Updated relevant documentation
[ ] Added relevant tests
[ ] Performed self-review of the code
[ ] Added changes to CHANGELOG.md

Tests won't pass bcs they require Scarb with snforge-test-collector xD


Updated README to use GenericImageView trait

Updated the README to use GenericImageView trait. I'm just trying the examples, and this took a bit to figure out. I'll submit more PR's as I find more issues. :)

Thanks!


fix: fixed chapi components layout

Fixes #753
Fixed chapi components layout. Since https://chapi-demo-issuer.digitalbazaar.com/ updated their layout, we had to adjust a few things.

Signed-off-by: Anton Biriukov<EMAIL_ADDRESS>

@birtony can you please add the screenshot  for the other screens like didconnect etc as well

Are you covering remaining screens in follow up task?

Addressed the rest of them in the following commit and opened a related issue.


⚠ Samsung has degraded performance

In d20e218, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2254 ms

Resolved: Samsung performance has improved in 8da12a2.


Add possibility to choose filter method

Add possibility to choose the filter method among equalTo and notEqualTo when filtering by metadata key. In this way in the future we can easily introduce additional methods.

LGTM


fix: high severity vulnerability fixed

Before:

After:

Some workflows are failing due to this warning.
Source: https://github.com/catosaurusrex2003/website/actions/runs/12120101309
I am not quite sure how to fix this.
@AceTheCreator do you have any suggestions ?


Show non-ready custom domains

🔧 Changes
I was recently working with custom domains, specifically adding and verifying it. I was able to add my custom domain with auth0 domains create. However, when I went to list with auth0 domains list, it stated that there were no custom domains.    This is because the Auth0 CLI hides non-ready custom domains. Considering that most (all?) tenants have a limit of one custom domain, there is very little reason to ever hide it if one exists. Especially because it interferes when trying to add and verify a custom domain.
🔬 Testing
Updated unit tests. Manual verification.
📝 Checklist

[x] All new/changed/fixed functionality is covered by tests (or N/A)
[x] I have added documentation for all new/changed functionality (or N/A)

Codecov Report
Patch coverage has no change and project coverage change: -0.01% :warning:

Comparison is base (2f4e29e) 72.07% compared to head (563d318) 72.07%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #781      +/-   ##
==========================================
- Coverage   72.07%   72.07%   -0.01%
==========================================
  Files          89       89
  Lines       11156    11155       -1
==========================================
- Hits         8041     8040       -1
  Misses       2613     2613
  Partials      502      502

Files Changed
Coverage Δ

internal/cli/custom_domains.go
78.83% <ø> (-0.07%)
:arrow_down:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add tests for AccumulateNode that insert the same facts and u…

…se the same rules, but vary the insert, retract, and fire patterns
I changed the default insertion duplication level from 1 to 0 here.  This is functionality that would, for example, take the pattern

Insert A

and change it to

Insert A
Insert A
Retract A

and the various order permutations of the latter, as well as keeping the original single-insertion scenario.  When I tried using the generator with the previous default I found that it made tests take far too long to run in some scenarios I wanted, presumably due to the combinatorial explosion of permutations.
Overall, none of the tests in this PR are going to get into complex scenarios in the compiler, rules network, etc.  My target here is more of a sanity check to improve confidence that the insert and retract algorithms on the AccumulateNode work correctly; this is very much a test of the AccumulateNode in isolation.   The tests are still written in terms of rules and sessions rather than direct calls to the node of course, with the attending advantages in reducing test brittleness.  However, given that we've had bugs that were purely internal to the accumulator nodes, I think such testing could be helpful.

Merged. Love the improved test coverage. Thanks, Will!


Bug fix

Noticed a bug when using your application. After usinga a command for example: help or add the application executes the command and exits making further commands being impossible. I would like to fix this bug and update your application if you consent.

ok


Make it easier to run tests and code checks

What does this Pull Request do?
This PR makes it easier to run unit tests, style tests, etc with a single command: composer test
What's new?
This does not change any code. It only changes the way in which code can be tested and checked, making use of Composer scripts.
How should this be tested?
From within the Hypercube, Gemini or Houdini directories, execute composer test, and you will see the output of phpunit and the other various code checks that are performed.
Additional Notes:
See here for more information: https://getcomposer.org/doc/articles/scripts.md#writing-custom-commands
Interested parties
@Islandora-CLAW/committers

If you'd like, I can send a similar PR to the other PHP-based repositories.


Multiple Images From ArrayBuffers

Are you using the latest version of jsPDF?
I'm using master.
Have you tried using jspdf.debug.js?
Yes
Steps to reproduce
Create a document, then add two different images to it, using ArrayBuffer sources for both.
What I saw
Both images are the first one in the PDF, but the frame is correct.
What I expected
Two different images.
Analysis
The reason is that jspdf tries to generate a hash for the image in addImage -> generateAliasFromData:
return typeof data === 'string' && jsPDFAPI.sHashCode(data);

which always returns false for ArrayBuffers due to the type check. Then it adds the image with the key false to the cache, irrespective of the content. After that, all ArrayBuffers have the same hash false and thus return the same image data when they're looked up in that cache.

Thank you for this bug  report and the good analysis.


Week icon

What is your question?
Where is the information to show week icon (from base weeks in the downloaded build)?

What

When you go in story mode, there are week icons. I would like to know where can you see the system call of the game (in a file) to show the week icon in because when i create a week, and change week icon access path, the game crashes.
I hope to be understood

Look my dude if u want to change the week icon:
1- Make sure ur icon is png
2- put it on path Ur fnf folder/mods/weeks/put ur image here
3- open the game
4- click 7 on main screen
5- go on week editor
6- there is the week icon place in there put ur png name
7- Done!

Precisely, when I change the access path of the image of the week (in custom week), the game crashes. That's why I prefer to copy a base week and that's why I want to know in a base week the line of code (I don't know which file) which appeals to the image.

So... are you going to show where you put the files and how you've set it in the editor?


[CLI] Support to pass the system variable name as org/image/version when building and running

Description:
Currently in cellery build and run, we can pass the image names. And during this we should be able to use the system variable to prepare the image name. For example, I should be able to build the image as below, and here org, and image are the two system variables.
cellery build ${org}/${image}:1.0.0

This is some what required for katacoda scenarios where users will need to build the images, and push it to their docker hubs.

as we discussed offline, Cellery CLI has the functionality to accept env variables as a parameter. We can close this issue. @sinthuja


WcfFacility's WcfClientExtension.AddChannelBuilder doesn't appear to work

Given the following:
private void RegisterService<T>(IWindsorContainer container, string serviceName)
        where T : class
    {
        var serviceSettings = new ServiceSettings(serviceName);
        var binding = new WS2007FederationHttpBinding(WSFederationHttpSecurityMode.TransportWithMessageCredential);
        binding.Security.Message.EstablishSecurityContext = false;
        binding.Security.Message.IssuedKeyType = System.IdentityModel.Tokens.SecurityKeyType.BearerKey;
        var endpoint = WcfEndpoint.BoundTo(binding).At(serviceSettings.Endpoint).AddExtensions(new WcfClientExtension().AddChannelBuilder<BearerTokenChannelBuilder>());
        var clientModel = new SecureWcfClientModel(endpoint);
        container.Register(Component.For<T>().AsWcfClient(clientModel));
    }

    public class BearerTokenChannelBuilder : AbstractChannelBuilder<SecureWcfClientModel>
    {
        private ITokenGetter tokenGetter;

        public BearerTokenChannelBuilder(IKernel kernel, IChannelFactoryBuilder<SecureWcfClientModel> channelFactoryBuilder, ITokenGetter tokenGetter)
            : base(kernel, channelFactoryBuilder)
        {
            this.tokenGetter = tokenGetter;
        }

        protected override ChannelCreator CreateChannelCreator(Type contract, SecureWcfClientModel clientModel, IChannelBuilderScope scope, params object[] channelFactoryArgs)
        {
            var type = typeof(ChannelFactory<>).MakeGenericType(new[] { contract });
            var channelFactory = this.ChannelFactoryBuilder.CreateChannelFactory(type, clientModel, channelFactoryArgs);
            scope.ConfigureChannelFactory(channelFactory);

            var methodInfo = type.GetMethod("CreateChannelWithIssuedToken", new Type[] { typeof(SecurityToken) });
            ChannelCreator creator = new ChannelCreator(() =>
            {
                return methodInfo.Invoke(channelFactory, new[] { this.tokenGetter.GetToken() });
            });
            return creator;
        }
    }

    public class SecureWcfClientModel : WcfClientModel<SecureWcfClientModel>
    {
        public SecureWcfClientModel()
        {
        }

        public SecureWcfClientModel(IWcfEndpoint endpoint)
            : base(endpoint)
        {
        }
    }

    public interface ITokenGetter
    {
        SecurityToken GetToken();
    }

    public class TokenGetter : ITokenGetter
    {
        public SecurityToken GetToken()
        {
            throw new NotImplementedException();
        }
    }

When I try use a WCF service with my configuration like this, I get
No component for supporting the service Castle.Facilities.WcfIntegration.IChannelBuilder`1[[SecureWcfClientModel, AssemblyNameHere, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=null]] was found

However, if I manually register the builder like so:
container.Register(Component.For<IChannelBuilder<SecureWcfClientModel>>().ImplementedBy<BearerTokenChannelBuilder>());

Then it works and I get my NotImplementedException as expected.
Am I missing something?

Fixed it with this:
container.AddFacility<WcfFacility>(c => c.Clients.AddChannelBuilder<BearerTokenChannelBuilder>());

@biltongza
Welcome(I am a fellow country man). Closing as you appear to have fixed the problem.


command to run baselines

Hi, I am really amazed by the number of options that are being provided in the code to run the experiments. I was wondering how to run the baselines, that is, Joint Training and None [simple sequential learning], as mentioned in the article. What is the code and the parameters to run these baseline algorithms seperately, like we can run algorithms like GR, LWF, etc?

Hi, I'm sorry for the late reply, your message got buried somewhere.
To run the baselines you mention, you can use the following code:

Joint training: ./main_cl.py --replay=offline
None (simple sequential learnign): ./main_cl.py --replay=none

Note that you can also leave out --replay=none, as this is the default.

Hey,
Thank you for the reply.
I went through the compare_[dataset].py files and figured out the parameters for none and joint baselines.
Just wanted to make sure that these were the only parameters which are passed in the command line.
Once again, thank you very much @GMvandeVen .


Fixing a goroutine leak in the 0.12 release candidate when using KILL QUERY

This is being fixed in master with a complete refactor of the
QueryExecutor, but that refactor won't cleanly apply to 0.12. This fix
is pretty important for ensuring that the goroutine used to monitor a
query finishes when a query is killed through KILL QUERY.

Accidentally opened this against master instead of 0.12. Closing.


Unhelpful error message when compiling with default features off

If v4l crate expects user to enable either libv4l or v4l2 features then there should probably be some #[cfg(...)] comile_error!() stating this if none or both of the features are enabled.
Now I get something like:
error[E0432]: unresolved import `crate::v4l_sys`
 --> /home/vi/.cargo/registry/src/-3d9d141e372ea94e/v4l-0.14.0/src/v4l2/videodev.rs:1:12
  |
1 | use crate::v4l_sys::v4l2_ext_control;
  |            ^^^^^^^ could not find `v4l_sys` in the crate root

(many similar compiler errors)

Yes, that sounds like a great idea. Could you open a pull request with the change?


Start new frame seamlessly to old frame

Sometimes I want to log time for several frames already finished. An exemplaric use case is given in
https://github.com/TailorDev/Watson/issues/233.
It would be great if there was a way to start a new frame seamlessly to the old one. Then, at the end of several short meetings, one could issue
watson start Project_1 +Meetings
#Go chatting to several colleagues
watson stop --at 13:37
watson start Project_2 +Meetings --seamless
watson stop --at 13:51

This PR is closely related to https://github.com/TailorDev/Watson/issues/75, https://github.com/TailorDev/Watson/issues/233 and https://github.com/TailorDev/Watson/issues/28.
Thank you very much.

I have nothing prepared, but I will give it a try.

As you can see in https://github.com/TailorDev/Watson/pull/235, I managed to dive into the codebase. It will take me a week or two. I hope that this is ok.

Of course it's ok! You owe us nothing. Take your time.

Shouldn't this issue be closed now?


Compile error in newest Xcode/Swift

Emoji.swift:2815:9: The compiler is unable to check that this switch is exhaustive in reasonable time
Should add default clauses to the switch statements

Me too, same problem here.

Same problem, any updates here?


Add IP address(host) to secret for Instance.redis

Description of your changes
This PR adds IP address to secret for Instance.redis resource.
Fixes: https://github.com/upbound/provider-gcp/issues/387
I have:

[ ] Run make reviewable test to ensure this PR is ready for review.

How has this code been tested
Tested manually:
> k describe secret example-redis-instance-secret -n upbound-system
Name:         example-redis-instance-secret
Namespace:    upbound-system
Labels:       <none>
Annotations:  <none>

Type:  connection.crossplane.io/v1alpha1

Data
====
host:                   13 bytes
attribute.auth_string:  36 bytes

/test-examples="examples/redis/instance.yaml"


Advanced items should be hidden behind Option key

The advanced items should only be visible when option+clicking the BitBar menu item.
Everything under the system separator should otherwise be invisible:

Fix is here: #227

I'm not sure about this one. Preferences are already hard to find (#225). Why make it harder? Instead I suggest a defaults flag that can be set from the command line for deployment.

I'm in agreement with @iosdeveloper. I think a defaults flag to the bitbar plist would make more sense for this. As was already mentioned in #180 by @morgrowe its trivial to discover the Option key trick. I'd rather be able to hide the advanced items menu with a defaults setting myself. This would allow regular users of BitBar to be able to set this themselves if they'd rather not see the advanced menu items, and allow Mac administrators to set this programmatically at deployment time.
This could potentially even be "managed" with a custom Configuration Profile so it can't be easily overridden.


The result of  mod expr is not correct

Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)

create table t2(a double ,b float);
insert into t2 values(-1,1);
alter table t2 set tiflash replica 2;
select mod(a,b)  in (0) from t2; -- result is 1
select mod(a,b)  in (0,1) from t2; -- expect 1 but got 0

2. What did you expect to see? (Required)
3. What did you see instead (Required)
4. What is your TiFlash version? (Required)
master

Checked that TiDB implement in func with hash map, key type float64 which ignore -0.0 and 0.0; MySQL implement with cmp, and -0.0 == 0.0, so they treat -0.0 and 0.0 the same.
Maybe we can change our in func implementation as TiDB does, don't use float64's int as hash key, use float64 type directly.


Release v0.1.0

Tag an initial release of the library.

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@312af7c). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff            @@
##             main        #1   +/-   ##
========================================
  Coverage        ?   100.00%
========================================
  Files           ?         2
  Lines           ?        51
  Branches        ?         0
========================================
  Hits            ?        51
  Misses          ?         0
  Partials        ?         0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 312af7c...f9243af. Read the comment docs.


can only certain element of component be made drag source and drop target.

Suppose I have a component with

             some texts

           how can I make only this div drop target or drag source??

You could make a new React component which only contains that div.

The link in the previous comment leads to a 404, but I think the same example now lives here: https://react-dnd.github.io/react-dnd/examples/customize/handles-and-previews
Excellent example by the way.


Helm dependency update not working anymore on 6.2.4 and 6.2.5

What happened?
Our helm dependencies are no longer getting updated appropriately on version 6.2.4 and 6.2.5. 6.2.3 works fine.
We think this is related to the change introduced here.
For some context, our directory structure looks something like this:
./devspace.yaml
./helm/
./helm/<chart-name>/
./helm/<chart-name>/Chart.yaml
./helm/<chart-name>/<all other helm related files are in this folder>

On version 6.2.3, when we run devspace dev, it appropriately runs the helm dependency update command, and we see a charts folder with our dependencies show up in helm/<chart-name>/charts.
But on version 6.2.4 and 6.2.5, we see the following error:
deploy:test Deploying chart  (test) with helm...
create_deployments: error deploying test: unable to deploy helm chart: Release "<redacted>" does not exist. Installing it now.
 error executing 'helm upgrade <redacted> --values /var/folders/jy/fpvqp56d3_s3v83dzs6vhxp80000gp/T/3112120045 --install --namespace <redacted> /Users/<redacted>/<redacted>/helm/<chart-name> --wait --timeout=300s --kube-context <redacted>: Error: An error occurred while checking for chart dependencies. You may need to run `helm dependency build` to fetch missing dependencies: found in Chart.yaml, but missing in charts/ directory: <name of chart dependency>

fatal exit status 1

If we manually run helm dependency build inside the helm/<chart-name> folder, we see all the appropriate dependencies show up, and subsequent runs of devspace dev work great. So we have a workaround, but something definitely broke here at least for us and our folder structure.

Thanks @tobalsgithub , we will take a look at it :+1:

@tobalsgithub Could you try something for me?

Add the following upgradeArgs to your deployment config for your deployments:deployments:
  test:
    helm:
      # other config same as before
      upgradeArgs:
        --dependency-update

Clear the charts directory and maybe also Chart.lock
Run devspace dev again

If that clears up the issue we can refactor DevSpace to add that flag.

@lizardruss this worked beautifully, and is very easy for us to implement.
Are you expecting to make a change that would remove the need for this flag? It's really no pain at all for us to have it. And maybe some use cases don't want the flag, so perhaps this is actually the way we want it?

@tobalsgithub Great! Thanks for trying that out. I've created PR #2545 to use that flag instead.
I have doubts that it would work for your particular case, since it seems that the helm dependency update command was skipped which led to your original error. With the directory structure mentioned above, how is your devspace.yaml's deployments defined? Hopefully I can improve the PR to automatically add the flag for your case as well. If possible, it'd be great if you could try out that PR on your project too.

@lizardruss I was able to give your PR a try, and added some debugging to see what might be going on.
It looks to me like right here, the chartPath is set incorrectly. I'm seeing something that looks like
/Users/<username>/<path>/<to>/<where>/<I>/<ran>/<devspace>/Users/<username>/<path>/<to>/<where>/<I>/<ran>/<devspace>/helm/<chart-directory>

You can see that /Users is in there twice. And I think what's going on is that the chartName variable is actually set to the path of the chart, not the name of the chart. So when you do filepath.Join(ctx.WorkingDir(), chartName) to get chartPath, it's not giving us what we want.

If I just take out the filepath.Join(ctx.WorkingDir(), chartName) piece completely and just use chartName instead, everything works as I expect it to. I think this all may come down to the fact that chartName is actually the path to the chart, not the name of the chart.

There's a separate question here around why we're putting the --dependency-update code behind an if check on whether the directory exists. Seems like if the chartName is not a valid path to a chart that we have bigger problems than whether we should run a dependency update 🤷
But there may be a good reason for that that I'm not aware of.

Is the name of the chart a full path? I think the intent was for it to be a relative path from devspace.yaml

Our chart names are relative, not absolute. I think that particular variable is getting resolved to the full path way up here inside that function. So when you're resolving it again down here it ends up double resolved.

@tobalsgithub I see, my tests weren't hitting that since I was only using name, but I see it gets doubled when using path. I've update the PR and added some tests, so hopefully it works for you again.


Auto PEP8 your source code

From<EMAIL_ADDRESS>on 2012-06-19T02:44:39Z
There is an utility to auto pep8 your source code. I guess this would go hand in hand with the current "remove trailing spaces". It could be nice if that could be done directly in spyder. http://pypi.python.org/pypi/autopep8
Original issue: http://code.google.com/p/spyderlib/issues/detail?id=1096

From ccordoba12 on 2012-07-04T07:31:28Z
Sounds interesting but someone will have to stand up to implement it. It shouldn't be too hard to do it though :)
Status: HelpNeeded
Labels: -Priority-Medium Priority-Low Cat-Editor

From<EMAIL_ADDRESS>on 2013-03-08T07:14:56Z
What's the purpose of https://code.google.com/p/spyderlib/source/browse/pep8.py ?

From contrebasse on 2013-08-29T15:35:24Z
I made a plugin for autopep8 (see attached file). This is my first experience with spyder code and it's mainly copy-paste. It was tested on python 2.7 and python 3.3. It runs with the default options.
I still have a few problems:

I did not find how to enable undoing the changes (with ctrl+Z)
on Arch Linux the bundled pep8 version was too old to be compatible with arch autopep8. I guess compatible versions should be put in external-py2 and external-py3 (autopep8 is compatible with both python versions without modifications) to avoid this kind of problem.

Attachment: p_autopep8.py

From contrebasse on 2013-08-30T01:33:45Z
I forgot a third problem:

how to stop loading the plugin if no autopep8 module is found ? Note that this won't be a problem if autopep8 is embedded like pep8.

From contrebasse on 2013-09-07T05:42:53Z
No answers ? Is my code that bad ? ^^

From ccordoba12 on 2013-09-07T07:19:27Z
Sorry, I overlooked it. Thanks for pinging us again. I'll take a look at it later today.
Giving the increasing interest in adding Spyder plugins, one solution would be to add them to PyPi until they are mature enough.

From pierre.raybaut on 2013-08-30T05:22:39Z
issue #1504 has been merged into this issue.

From contrebasse on 2013-09-16T02:30:13Z
I created a github repo for the plugin : https://github.com/Nodd/spyder_autopep8 .
Is there another way to install plugins other than putting the files in /usr/lib/python2.7/dist-packages/spyderplugins/ (or equivalent) ? Something like ~/.spyder2/plugins would be useful.

From contrebasse on 2013-09-07T11:20:44Z
I can put it on pypi, but it won't work if installed with pip, the file has to be put in the right spyder subfolder. github may be a better compromise ?

From ccordoba12 on 2015-01-02T18:17:41Z
Labels: MS-v2.4

From contrebasse on 2013-10-14T15:34:02Z
I found solutions to my basic problems. It should be usable now.

@Nodd do you want to make your plugin more like a package? I had already started to do it here
@ccordoba12, should we make an additional repo for this plugin inside spyder-ide repo? It could also be useful as an example of how to make such plugins.

I depends on @Nodd :-)

Yes, it should be made as a package, but only when spyder will be able to load plugins from a directory in spyderplugins.

@ccordoba12 have you made any progress in this direction ?

For information I just started working on spyderplugins. I'll soon create a PR to discuss about it.

@goanpeca, nop, I'm in review mode these days :-)

Then you can start reviewing #2362 !

There is a working external plugin for spyder 3.0 (incoming!) at https://github.com/spyder-ide/spyder.autopep8