NullPointerException when Running CaffeOnSpark on EC2

I am running CaffeOnSpark on EC2 following the instructions https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_EC2. I got the following errors:
16/05/06 00:10:35 INFO TaskSetManager: Starting task 1.1 in stage 1.0 (TID 5, ip-10-30-15-17.us-west-2.compute.internal, partition 1,PROCESS_LOCAL, 2197 bytes)
16/05/06 00:10:35 WARN TaskSetManager: Lost task 0.1 in stage 1.0 (TID 4, ip-10-30-15-17.us-west-2.compute.internal): java.lang.NullPointerException
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply$mcVI$sp(CaffeOnSpark.scala:153)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply(CaffeOnSpark.scala:149)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply(CaffeOnSpark.scala:149)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

more logs:
16/05/06 00:10:35 ERROR TaskSetManager: Task 0 in stage 1.0 failed 4 times; aborting job
16/05/06 00:10:35 INFO TaskSchedulerImpl: Cancelling stage 1
16/05/06 00:10:35 INFO TaskSchedulerImpl: Stage 1 was cancelled
16/05/06 00:10:35 INFO DAGScheduler: ResultStage 1 (collect at CaffeOnSpark.scala:155) failed in 112.354 s
16/05/06 00:10:35 INFO DAGScheduler: Job 1 failed: collect at CaffeOnSpark.scala:155, took 112.367186 s
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 8, ip-10-30-15-17.us-west-2.compute.internal): java.lang.NullPointerException
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply$mcVI$sp(CaffeOnSpark.scala:153)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply(CaffeOnSpark.scala:149)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply(CaffeOnSpark.scala:149)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:927)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.collect(RDD.scala:926)
at com.yahoo.ml.caffe.CaffeOnSpark.train(CaffeOnSpark.scala:155)
at com.yahoo.ml.caffe.CaffeOnSpark$.main(CaffeOnSpark.scala:40)
at com.yahoo.ml.caffe.CaffeOnSpark.main(CaffeOnSpark.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NullPointerException
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply$mcVI$sp(CaffeOnSpark.scala:153)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply(CaffeOnSpark.scala:149)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply(CaffeOnSpark.scala:149)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16/05/06 00:10:35 INFO SparkContext: Invoking stop() from shutdown hook
16/05/06 00:10:35 INFO TaskSetManager: Lost task 1.3 in stage 1.0 (TID 9) on executor ip-10-30-15-17.us-west-2.compute.internal: org.apache.spark.TaskKilledException (null) [duplicate 5]
16/05/06 00:10:35 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
16/05/06 00:10:35 INFO SparkUI: Stopped Spark web UI at http://ec2-52-24-22-149.us-west-2.compute.amazonaws.com:4040
16/05/06 00:10:35 INFO SparkDeploySchedulerBackend: Shutting down all executors
16/05/06 00:10:35 INFO SparkDeploySchedulerBackend: Asking each executor to shut down
16/05/06 00:10:35 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/05/06 00:10:35 INFO MemoryStore: MemoryStore cleared
16/05/06 00:10:35 INFO BlockManager: BlockManager stopped
16/05/06 00:10:35 INFO BlockManagerMaster: BlockManagerMaster stopped
16/05/06 00:10:35 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/05/06 00:10:35 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/05/06 00:10:35 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/05/06 00:10:35 INFO SparkContext: Successfully stopped SparkContext
16/05/06 00:10:35 INFO ShutdownHookManager: Shutdown hook called
16/05/06 00:10:35 INFO ShutdownHookManager: Deleting directory /mnt/spark/spark-23e9b9f9-8b51-4b03-bd3d-6fdb15bb9fe6/httpd-8c4d711a-1bf6-4917-911c-161187fb2b61
16/05/06 00:10:35 INFO ShutdownHookManager: Deleting directory /mnt/spark/spark-23e9b9f9-8b51-4b03-bd3d-6fdb15bb9fe6

Here is my parameters:
export CORES_PER_WORKER=8
export DEVICES=1
export SPARK_WORKER_INSTANCES=2
export TOTAL_CORES=$((${CORES_PER_WORKER}*${SPARK_WORKER_INSTANCES}))
export MASTER_URL=spark://$(hostname):7077

Here is my command:
spark-submit --master ${MASTER_URL}
--files lenet_memory_train_test.prototxt,lenet_memory_solver.prototxt
--conf spark.cores.max=${TOTAL_CORES}
--conf spark.task.cpus=${CORES_PER_WORKER}
--conf spark.driver.extraLibraryPath="${LD_LIBRARY_PATH}"
--conf spark.executorEnv.LD_LIBRARY_PATH="${LD_LIBRARY_PATH}"
--class com.yahoo.ml.caffe.CaffeOnSpark
${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar
-train
-features accuracy,loss -label label
-conf lenet_memory_solver.prototxt
-clusterSize ${SPARK_WORKER_INSTANCES}
-devices ${DEVICES}
-connection ethernet
-model /mnist.model
-output /mnist_features_result

Please paste the content of this file:  lenet_memory_train_test.prototxt
Alternative check your dataset path is correct
On Wed, May 18, 2016 at 4:16 PM, syuquad<EMAIL_ADDRESS>wrote:

Here is my command:
spark-submit --master ${MASTER_URL}
--files lenet_memory_train_test.prototxt,lenet_memory_solver.prototxt
--conf spark.cores.max=${TOTAL_CORES}
--conf spark.task.cpus=${CORES_PER_WORKER}
--conf spark.driver.extraLibraryPath="${LD_LIBRARY_PATH}"
--conf spark.executorEnv.LD_LIBRARY_PATH="${LD_LIBRARY_PATH}"
--class com.yahoo.ml.caffe.CaffeOnSpark
${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar

-train
-features accuracy,loss -label label
-conf lenet_memory_solver.prototxt
-clusterSize ${SPARK_WORKER_INSTANCES}
-devices ${DEVICES}
-connection ethernet
-model /mnist.model
-output /mnist_features_result
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/yahoo/CaffeOnSpark/issues/67#issuecomment-220185925

I see the following in the log and I believe that's the reason for the failure. But I don't know how to fix it.
16/05/20 01:23:58 ERROR TaskSchedulerImpl: Lost executor 1 on ip-10-30-23-120.us-west-2.compute.internal: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
16/05/20 01:23:58 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, ip-10-30-23-120.us-west-2.compute.internal): ExecutorLostFailure (executor 1 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues.
Andy
On Mon, May 23, 2016 at 1:24 PM, syuquad<EMAIL_ADDRESS>wrote:

I see the following in the log and I believe that's the reason for the
failure. But I don't know how to fix it.
16/05/20 01:23:58 ERROR TaskSchedulerImpl: Lost executor 1 on
ip-10-30-23-120.us-west-2.compute.internal: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
16/05/20 01:23:58 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2,
ip-10-30-23-120.us-west-2.compute.internal): ExecutorLostFailure (executor
1 exited caused by one of the running tasks) Reason: Remote RPC client
disassociated. Likely due to containers exceeding thresholds, or network
issues. Check driver logs for WARN messages.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/yahoo/CaffeOnSpark/issues/67#issuecomment-221085210

Not sure which log you are looking for. Here is the log file from the master node under /spark/logs:
root@ip-10-30-10-204:~/spark/logs# ls -lt
total 12
-rw-r--r-- 1 root root 9240 May 23 22:52 spark-root-org.apache.spark.deploy.master.Master-1-ip-10-30-10-204.us-west-2.compute.internal.out
root@ip-10-30-10-204:~/spark/logs# more spark-root-org.apache.spark.deploy.master.Master-1-ip-10-30-10-204.us-west-2.compute.internal.out
Spark Command: /usr/lib/jvm/java-1.7.0-openjdk-amd64/bin/java -cp /root/s
park/conf/:/root/spark/lib/spark-assembly-1.6.0-hadoop2.4.0.jar:/root/spa
rk/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.
2.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/ephemeral-hdfs/
conf/ -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.master.M
aster --ip <IP_ADDRESS> --port 7077 --webui-port 8080
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.prop
erties
16/05/23 21:31:00 INFO Master: Registered signal handlers for [TERM, HUP,
INT]
16/05/23 21:31:00 WARN NativeCodeLoader: Unable to load native-hadoop lib
rary for your platform... using builtin-java classes where applicable
16/05/23 21:31:00 INFO SecurityManager: Changing view acls to: root
16/05/23 21:31:00 INFO SecurityManager: Changing modify acls to: root
16/05/23 21:31:00 INFO SecurityManager: SecurityManager: authentication d
isabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
16/05/23 21:31:01 INFO Utils: Successfully started service 'sparkMaster'
on port 7077.
16/05/23 21:31:01 INFO Master: Starting Spark master at spark://10.30.10.
204:7077
16/05/23 21:31:01 INFO Master: Running Spark version 1.6.0
16/05/23 21:31:01 INFO Utils: Successfully started service 'MasterUI' on
port 8080.
16/05/23 21:31:01 INFO MasterWebUI: Started MasterWebUI at http://ec2-54-
148-250-247.us-west-2.compute.amazonaws.com:8080
16/05/23 21:31:01 INFO Utils: Successfully started service on port 6066.
16/05/23 21:31:01 INFO StandaloneRestServer: Started REST server for subm
itting applications on port 6066
16/05/23 21:31:02 INFO Master: I have been elected leader! New state: ALI
VE
16/05/23 21:31:24 INFO Master: Registering worker <IP_ADDRESS>:51910 with 8
cores, 13.7 GB RAM
16/05/23 21:31:24 INFO Master: Registering worker <IP_ADDRESS>:41021 with 8
cores, 13.7 GB RAM
16/05/23 21:55:18 INFO Master: Registering app com.yahoo.ml.caffe.CaffeOn
Spark
16/05/23 21:55:18 INFO Master: Registered app com.yahoo.ml.caffe.CaffeOnS
park with ID app-20160523215518-0000
16/05/23 21:55:18 INFO Master: Launching executor app-20160523215518-0000
/0 on worker worker-20160523213123-<IP_ADDRESS>-51910
16/05/23 21:55:18 INFO Master: Launching executor app-20160523215518-0000
/1 on worker worker-20160523213123-<IP_ADDRESS>-41021
16/05/23 21:55:21 INFO Master: Received unregister request from applicati
on app-20160523215518-0000
16/05/23 21:55:21 INFO Master: Removing app app-20160523215518-0000
16/05/23 21:55:21 INFO Master: ip-10-30-10-204.us-west-2.compute.internal
:50082 got disassociated, removing it.
16/05/23 21:55:21 INFO Master: <IP_ADDRESS>:44056 got disassociated, remo
ving it.
16/05/23 21:55:21 WARN Master: Got status update for unknown executor app
-20160523215518-0000/1
16/05/23 21:55:21 WARN Master: Got status update for unknown executor app
-20160523215518-0000/0
16/05/23 21:58:40 INFO Master: Registering app com.yahoo.ml.caffe.CaffeOn
Spark
16/05/23 21:58:40 INFO Master: Registered app com.yahoo.ml.caffe.CaffeOnS
park with ID app-20160523215840-0001
16/05/23 21:58:40 INFO Master: Launching executor app-20160523215840-0001
/0 on worker worker-20160523213123-<IP_ADDRESS>-51910
16/05/23 21:58:40 INFO Master: Launching executor app-20160523215840-0001
/1 on worker worker-20160523213123-<IP_ADDRESS>-41021
16/05/23 21:58:42 INFO Master: Received unregister request from applicati
on app-20160523215840-0001
16/05/23 21:58:42 INFO Master: Removing app app-20160523215840-0001
16/05/23 21:58:42 INFO Master: ip-10-30-10-204.us-west-2.compute.internal
:50091 got disassociated, removing it.
16/05/23 21:58:42 INFO Master: <IP_ADDRESS>:52644 got disassociated, remo
ving it.
16/05/23 21:58:43 WARN Master: Got status update for unknown executor app
-20160523215840-0001/0
16/05/23 21:58:43 WARN Master: Got status update for unknown executor app
-20160523215840-0001/1
16/05/23 22:00:08 INFO Master: Registering app com.yahoo.ml.caffe.CaffeOn
Spark
16/05/23 22:00:08 INFO Master: Registered app com.yahoo.ml.caffe.CaffeOnS
park with ID app-20160523220008-0002
16/05/23 22:00:08 INFO Master: Launching executor app-20160523220008-0002
/0 on worker worker-20160523213123-<IP_ADDRESS>-51910
16/05/23 22:00:08 INFO Master: Launching executor app-20160523220008-0002
/1 on worker worker-20160523213123-<IP_ADDRESS>-41021
16/05/23 22:00:11 INFO Master: Received unregister request from applicati
on app-20160523220008-0002
16/05/23 22:00:11 INFO Master: Removing app app-20160523220008-0002
16/05/23 22:00:11 INFO Master: ip-10-30-10-204.us-west-2.compute.internal
:50094 got disassociated, removing it.
16/05/23 22:00:11 INFO Master: <IP_ADDRESS>:58235 got disassociated, remo
ving it.
16/05/23 22:00:11 WARN Master: Got status update for unknown executor app
-20160523220008-0002/1
16/05/23 22:00:11 WARN Master: Got status update for unknown executor app
-20160523220008-0002/0
16/05/23 22:00:28 INFO Master: Registering app com.yahoo.ml.caffe.CaffeOn
Spark
16/05/23 22:00:28 INFO Master: Registered app com.yahoo.ml.caffe.CaffeOnS
park with ID app-20160523220028-0003
16/05/23 22:00:28 INFO Master: Launching executor app-20160523220028-0003
/0 on worker worker-20160523213123-<IP_ADDRESS>-51910
16/05/23 22:00:28 INFO Master: Launching executor app-20160523220028-0003
/1 on worker worker-20160523213123-<IP_ADDRESS>-41021
16/05/23 22:02:58 INFO Master: Removing executor app-20160523220028-0003/
1 because it is EXITED
16/05/23 22:02:58 INFO Master: Launching executor app-20160523220028-0003
/2 on worker worker-20160523213123-<IP_ADDRESS>-41021
16/05/23 22:02:58 INFO Master: Removing executor app-20160523220028-0003/
0 because it is EXITED
16/05/23 22:02:58 INFO Master: Launching executor app-20160523220028-0003
/3 on worker worker-20160523213123-<IP_ADDRESS>-51910
16/05/23 22:03:01 INFO Master: Received unregister request from applicati
on app-20160523220028-0003
16/05/23 22:03:01 INFO Master: Removing app app-20160523220028-0003
16/05/23 22:03:01 INFO Master: ip-10-30-10-204.us-west-2.compute.internal
:50096 got disassociated, removing it.
16/05/23 22:03:01 INFO Master: <IP_ADDRESS>:44475 got disassociated, remo
ving it.
16/05/23 22:03:01 WARN Master: Got status update for unknown executor app
-20160523220028-0003/2
16/05/23 22:03:01 WARN Master: Got status update for unknown executor app
-20160523220028-0003/3
16/05/23 22:14:23 INFO Master: Registering app com.yahoo.ml.caffe.CaffeOn
Spark
16/05/23 22:14:23 INFO Master: Registered app com.yahoo.ml.caffe.CaffeOnS
park with ID app-20160523221423-0004
16/05/23 22:14:23 INFO Master: Launching executor app-20160523221423-0004
/0 on worker worker-20160523213123-<IP_ADDRESS>-51910
16/05/23 22:14:23 INFO Master: Launching executor app-20160523221423-0004
/1 on worker worker-20160523213123-<IP_ADDRESS>-41021
16/05/23 22:14:39 INFO Master: Removing executor app-20160523221423-0004/
0 because it is EXITED
16/05/23 22:14:39 INFO Master: Launching executor app-20160523221423-0004
/2 on worker worker-20160523213123-<IP_ADDRESS>-51910
16/05/23 22:14:39 INFO Master: Removing executor app-20160523221423-0004/
1 because it is EXITED
16/05/23 22:14:39 INFO Master: Launching executor app-20160523221423-0004
/3 on worker worker-20160523213123-<IP_ADDRESS>-41021
16/05/23 22:14:42 INFO Master: Received unregister request from applicati
on app-20160523221423-0004
16/05/23 22:14:42 INFO Master: Removing app app-20160523221423-0004
16/05/23 22:14:42 INFO Master: ip-10-30-10-204.us-west-2.compute.internal
:50113 got disassociated, removing it.
16/05/23 22:14:42 INFO Master: <IP_ADDRESS>:44328 got disassociated, remo
ving it.
16/05/23 22:14:43 WARN Master: Got status update for unknown executor app
-20160523221423-0004/3
16/05/23 22:14:43 WARN Master: Got status update for unknown executor app
-20160523221423-0004/2
16/05/23 22:52:31 INFO Master: Registering app com.yahoo.ml.caffe.CaffeOn
Spark
16/05/23 22:52:31 INFO Master: Registered app com.yahoo.ml.caffe.CaffeOnS
park with ID app-20160523225231-0005
16/05/23 22:52:31 INFO Master: Launching executor app-20160523225231-0005
/0 on worker worker-20160523213123-<IP_ADDRESS>-51910
16/05/23 22:52:31 INFO Master: Launching executor app-20160523225231-0005
/1 on worker worker-20160523213123-<IP_ADDRESS>-41021
16/05/23 22:52:47 INFO Master: Removing executor app-20160523225231-0005/
0 because it is EXITED
16/05/23 22:52:47 INFO Master: Launching executor app-20160523225231-0005
/2 on worker worker-20160523213123-<IP_ADDRESS>-51910
16/05/23 22:52:47 INFO Master: Removing executor app-20160523225231-0005/
1 because it is EXITED
16/05/23 22:52:47 INFO Master: Launching executor app-20160523225231-0005
/3 on worker worker-20160523213123-<IP_ADDRESS>-41021
16/05/23 22:52:50 INFO Master: Received unregister request from applicati
on app-20160523225231-0005
16/05/23 22:52:50 INFO Master: Removing app app-20160523225231-0005
16/05/23 22:52:50 INFO Master: ip-10-30-10-204.us-west-2.compute.internal
:50159 got disassociated, removing it.
16/05/23 22:52:50 INFO Master: <IP_ADDRESS>:45211 got disassociated, remo
ving it.
16/05/23 22:52:51 WARN Master: Got status update for unknown executor app
-20160523225231-0005/3
16/05/23 22:52:51 WARN Master: Got status update for unknown executor app
-20160523225231-0005/2

Here is the log file from one of the slave node.
root@ip-10-30-3-42:~/spark/logs# ls
spark-root-org.apache.spark.deploy.worker.Worker-1-ip-10-30-3-42.out
root@ip-10-30-3-42:~/spark/logs# more spark-root-org.apache.spark.deploy.worker.Worker-1-ip-10-30-3-42.out
Spark Command: /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java -cp /root/s
park/conf/:/root/spark/lib/spark-assembly-1.6.0-hadoop2.4.0.jar:/root/spa
rk/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.
2.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/ephemeral-hdfs/
conf/ -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.worker.W
orker --webui-port 8081 spark://<IP_ADDRESS>:7077
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.prop
erties
16/05/23 21:31:22 INFO Worker: Registered signal handlers for [TERM, HUP,
INT]
16/05/23 21:31:22 WARN NativeCodeLoader: Unable to load native-hadoop lib
rary for your platform... using builtin-java classes where applicable
16/05/23 21:31:23 INFO SecurityManager: Changing view acls to: root
16/05/23 21:31:23 INFO SecurityManager: Changing modify acls to: root
16/05/23 21:31:23 INFO SecurityManager: SecurityManager: authentication d
isabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
16/05/23 21:31:23 INFO Utils: Successfully started service 'sparkWorker'
on port 51910.
16/05/23 21:31:23 INFO Worker: Starting Spark worker <IP_ADDRESS>:51910 wit
h 8 cores, 13.7 GB RAM
16/05/23 21:31:23 INFO Worker: Running Spark version 1.6.0
16/05/23 21:31:23 INFO Worker: Spark home: /root/spark
16/05/23 21:31:24 INFO Utils: Successfully started service 'WorkerUI' on
port 8081.
16/05/23 21:31:24 INFO WorkerWebUI: Started WorkerWebUI at http://ec2-54-
149-117-170.us-west-2.compute.amazonaws.com:8081
16/05/23 21:31:24 INFO Worker: Connecting to master <IP_ADDRESS>:7077...
16/05/23 21:31:24 INFO Worker: Successfully registered with master spark:
//<IP_ADDRESS>:7077
16/05/23 21:55:18 INFO Worker: Asked to launch executor app-2016052321551
8-0000/0 for com.yahoo.ml.caffe.CaffeOnSpark
16/05/23 21:55:18 INFO SecurityManager: Changing view acls to: root
16/05/23 21:55:18 INFO SecurityManager: Changing modify acls to: root
16/05/23 21:55:18 INFO SecurityManager: SecurityManager: authentication d
isabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
16/05/23 21:55:18 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java
-7-openjdk-amd64/jre/bin/java" "-cp" "/root/ephemeral-hdfs/conf/:/root/sp
ark/conf/:/root/spark/lib/spark-assembly-1.6.0-hadoop2.4.0.jar:/root/spar
k/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.2
.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/ephemeral-hdfs/c
onf/" "-Xms12991M" "-Xmx12991M" "-Dspark.driver.port=44056" "-XX:MaxPermS
ize=256m" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--dri
ver-url" "spark://CoarseGrainedScheduler@<IP_ADDRESS>:44056" "--executor-
id" "0" "--hostname" "<IP_ADDRESS>" "--cores" "8" "--app-id" "app-201605232
15518-0000" "--worker-url" "spark://Worker@<IP_ADDRESS>:51910"
16/05/23 21:55:21 INFO Worker: Asked to kill executor app-20160523215518-
0000/0
16/05/23 21:55:21 INFO ExecutorRunner: Runner thread for executor app-201
60523215518-0000/0 interrupted
16/05/23 21:55:21 INFO ExecutorRunner: Killing process!
16/05/23 21:55:21 ERROR FileAppender: Error writing stream to file /root/
spark/work/app-20160523215518-0000/0/stderr
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.j
ava:162)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:272
)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.spark.util.logging.FileAppender.appendStreamToFile(
FileAppender.scala:70)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply$mcV$sp(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala
:1741)
at org.apache.spark.util.logging.FileAppender$$anon$1.run(FileApp
ender.scala:38)
16/05/23 21:55:21 INFO Worker: Executor app-20160523215518-0000/0 finishe
d with state KILLED exitStatus 143
16/05/23 21:55:21 INFO Worker: Cleaning up local directories for applicat
ion app-20160523215518-0000
16/05/23 21:55:21 INFO ExternalShuffleBlockResolver: Application app-2016
0523215518-0000 removed, cleanupLocalDirs = true
16/05/23 21:58:40 INFO Worker: Asked to launch executor app-2016052321584
0-0001/0 for com.yahoo.ml.caffe.CaffeOnSpark
16/05/23 21:58:40 INFO SecurityManager: Changing view acls to: root
16/05/23 21:58:40 INFO SecurityManager: Changing modify acls to: root
16/05/23 21:58:40 INFO SecurityManager: SecurityManager: authentication d
isabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
16/05/23 21:58:40 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java
-7-openjdk-amd64/jre/bin/java" "-cp" "/root/ephemeral-hdfs/conf/:/root/sp
ark/conf/:/root/spark/lib/spark-assembly-1.6.0-hadoop2.4.0.jar:/root/spar
k/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.2
.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/ephemeral-hdfs/c
onf/" "-Xms12991M" "-Xmx12991M" "-Dspark.driver.port=52644" "-XX:MaxPermS
ize=256m" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--dri
ver-url" "spark://CoarseGrainedScheduler@<IP_ADDRESS>:52644" "--executor-
id" "0" "--hostname" "<IP_ADDRESS>" "--cores" "8" "--app-id" "app-201605232
15840-0001" "--worker-url" "spark://Worker@<IP_ADDRESS>:51910"
16/05/23 21:58:42 INFO Worker: Asked to kill executor app-20160523215840-
0001/0
16/05/23 21:58:42 INFO ExecutorRunner: Runner thread for executor app-201
60523215840-0001/0 interrupted
16/05/23 21:58:42 INFO ExecutorRunner: Killing process!
16/05/23 21:58:42 ERROR FileAppender: Error writing stream to file /root/
spark/work/app-20160523215840-0001/0/stderr
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.j
ava:162)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:272
)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.spark.util.logging.FileAppender.appendStreamToFile(
FileAppender.scala:70)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply$mcV$sp(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala
:1741)
at org.apache.spark.util.logging.FileAppender$$anon$1.run(FileApp
ender.scala:38)
16/05/23 21:58:42 INFO Worker: Executor app-20160523215840-0001/0 finishe
d with state KILLED exitStatus 143
16/05/23 21:58:42 INFO Worker: Cleaning up local directories for applicat
ion app-20160523215840-0001
16/05/23 21:58:42 INFO ExternalShuffleBlockResolver: Application app-2016
0523215840-0001 removed, cleanupLocalDirs = true
16/05/23 22:00:08 INFO Worker: Asked to launch executor app-2016052322000
8-0002/0 for com.yahoo.ml.caffe.CaffeOnSpark
16/05/23 22:00:08 INFO SecurityManager: Changing view acls to: root
16/05/23 22:00:08 INFO SecurityManager: Changing modify acls to: root
16/05/23 22:00:08 INFO SecurityManager: SecurityManager: authentication d
isabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
16/05/23 22:00:08 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java
-7-openjdk-amd64/jre/bin/java" "-cp" "/root/ephemeral-hdfs/conf/:/root/sp
ark/conf/:/root/spark/lib/spark-assembly-1.6.0-hadoop2.4.0.jar:/root/spar
k/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.2
.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/ephemeral-hdfs/c
onf/" "-Xms12991M" "-Xmx12991M" "-Dspark.driver.port=58235" "-XX:MaxPermS
ize=256m" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--dri
ver-url" "spark://CoarseGrainedScheduler@<IP_ADDRESS>:58235" "--executor-
id" "0" "--hostname" "<IP_ADDRESS>" "--cores" "8" "--app-id" "app-201605232
20008-0002" "--worker-url" "spark://Worker@<IP_ADDRESS>:51910"
16/05/23 22:00:11 INFO Worker: Asked to kill executor app-20160523220008-
0002/0
16/05/23 22:00:11 INFO ExecutorRunner: Runner thread for executor app-201
60523220008-0002/0 interrupted
16/05/23 22:00:11 INFO ExecutorRunner: Killing process!
16/05/23 22:00:11 ERROR FileAppender: Error writing stream to file /root/
spark/work/app-20160523220008-0002/0/stderr
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.j
ava:162)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:272
)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.spark.util.logging.FileAppender.appendStreamToFile(
FileAppender.scala:70)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply$mcV$sp(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala
:1741)
at org.apache.spark.util.logging.FileAppender$$anon$1.run(FileApp
ender.scala:38)
16/05/23 22:00:11 INFO Worker: Executor app-20160523220008-0002/0 finishe
d with state KILLED exitStatus 143
16/05/23 22:00:11 INFO Worker: Cleaning up local directories for applicat
ion app-20160523220008-0002
16/05/23 22:00:11 INFO ExternalShuffleBlockResolver: Application app-2016
0523220008-0002 removed, cleanupLocalDirs = true
16/05/23 22:00:28 INFO Worker: Asked to launch executor app-2016052322002
8-0003/0 for com.yahoo.ml.caffe.CaffeOnSpark
16/05/23 22:00:28 INFO SecurityManager: Changing view acls to: root
16/05/23 22:00:28 INFO SecurityManager: Changing modify acls to: root
16/05/23 22:00:28 INFO SecurityManager: SecurityManager: authentication d
isabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
16/05/23 22:00:28 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java
-7-openjdk-amd64/jre/bin/java" "-cp" "/root/ephemeral-hdfs/conf/:/root/sp
ark/conf/:/root/spark/lib/spark-assembly-1.6.0-hadoop2.4.0.jar:/root/spar
k/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.2
.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/ephemeral-hdfs/c
onf/" "-Xms12991M" "-Xmx12991M" "-Dspark.driver.port=44475" "-XX:MaxPermS
ize=256m" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--dri
ver-url" "spark://CoarseGrainedScheduler@<IP_ADDRESS>:44475" "--executor-
id" "0" "--hostname" "<IP_ADDRESS>" "--cores" "8" "--app-id" "app-201605232
20028-0003" "--worker-url" "spark://Worker@<IP_ADDRESS>:51910"
16/05/23 22:02:58 INFO Worker: Executor app-20160523220028-0003/0 finishe
d with state EXITED message Command exited with code 134 exitStatus 134
16/05/23 22:02:58 INFO Worker: Asked to launch executor app-2016052322002
8-0003/3 for com.yahoo.ml.caffe.CaffeOnSpark
16/05/23 22:02:58 INFO SecurityManager: Changing view acls to: root
16/05/23 22:02:58 INFO SecurityManager: Changing modify acls to: root
16/05/23 22:02:58 INFO SecurityManager: SecurityManager: authentication d
isabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
16/05/23 22:02:58 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java
-7-openjdk-amd64/jre/bin/java" "-cp" "/root/ephemeral-hdfs/conf/:/root/sp
ark/conf/:/root/spark/lib/spark-assembly-1.6.0-hadoop2.4.0.jar:/root/spar
k/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.2
.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/ephemeral-hdfs/c
onf/" "-Xms12991M" "-Xmx12991M" "-Dspark.driver.port=44475" "-XX:MaxPermS
ize=256m" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--dri
ver-url" "spark://CoarseGrainedScheduler@<IP_ADDRESS>:44475" "--executor-
id" "3" "--hostname" "<IP_ADDRESS>" "--cores" "8" "--app-id" "app-201605232
20028-0003" "--worker-url" "spark://Worker@<IP_ADDRESS>:51910"
16/05/23 22:03:01 INFO Worker: Asked to kill executor app-20160523220028-
0003/3
16/05/23 22:03:01 INFO ExecutorRunner: Runner thread for executor app-201
60523220028-0003/3 interrupted
16/05/23 22:03:01 INFO ExecutorRunner: Killing process!
16/05/23 22:03:01 ERROR FileAppender: Error writing stream to file /root/
spark/work/app-20160523220028-0003/3/stderr
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.j
ava:162)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:272
)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.spark.util.logging.FileAppender.appendStreamToFile(
FileAppender.scala:70)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply$mcV$sp(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala
:1741)
at org.apache.spark.util.logging.FileAppender$$anon$1.run(FileApp
ender.scala:38)
16/05/23 22:03:01 INFO Worker: Executor app-20160523220028-0003/3 finishe
d with state KILLED exitStatus 143
16/05/23 22:03:01 INFO Worker: Cleaning up local directories for applicat
ion app-20160523220028-0003
16/05/23 22:03:01 INFO ExternalShuffleBlockResolver: Application app-2016
0523220028-0003 removed, cleanupLocalDirs = true
16/05/23 22:14:23 INFO Worker: Asked to launch executor app-2016052322142
3-0004/0 for com.yahoo.ml.caffe.CaffeOnSpark
16/05/23 22:14:23 INFO SecurityManager: Changing view acls to: root
16/05/23 22:14:23 INFO SecurityManager: Changing modify acls to: root
16/05/23 22:14:23 INFO SecurityManager: SecurityManager: authentication d
isabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
16/05/23 22:14:23 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java
-7-openjdk-amd64/jre/bin/java" "-cp" "/root/ephemeral-hdfs/conf/:/root/sp
ark/conf/:/root/spark/lib/spark-assembly-1.6.0-hadoop2.4.0.jar:/root/spar
k/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.2
.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/ephemeral-hdfs/c
onf/" "-Xms5120M" "-Xmx5120M" "-Dspark.driver.port=44328" "-XX:MaxPermSiz
e=512M" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--drive
r-url" "spark://CoarseGrainedScheduler@<IP_ADDRESS>:44328" "--executor-id
" "0" "--hostname" "<IP_ADDRESS>" "--cores" "8" "--app-id" "app-20160523221
423-0004" "--worker-url" "spark://Worker@<IP_ADDRESS>:51910"
16/05/23 22:14:39 INFO Worker: Executor app-20160523221423-0004/0 finishe
d with state EXITED message Command exited with code 134 exitStatus 134
16/05/23 22:14:39 INFO Worker: Asked to launch executor app-2016052322142
3-0004/2 for com.yahoo.ml.caffe.CaffeOnSpark
16/05/23 22:14:39 INFO SecurityManager: Changing view acls to: root
16/05/23 22:14:39 INFO SecurityManager: Changing modify acls to: root
16/05/23 22:14:39 INFO SecurityManager: SecurityManager: authentication d
isabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
16/05/23 22:14:39 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java
-7-openjdk-amd64/jre/bin/java" "-cp" "/root/ephemeral-hdfs/conf/:/root/sp
ark/conf/:/root/spark/lib/spark-assembly-1.6.0-hadoop2.4.0.jar:/root/spar
k/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.2
.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/ephemeral-hdfs/c
onf/" "-Xms5120M" "-Xmx5120M" "-Dspark.driver.port=44328" "-XX:MaxPermSiz
e=512M" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--drive
r-url" "spark://CoarseGrainedScheduler@<IP_ADDRESS>:44328" "--executor-id
" "2" "--hostname" "<IP_ADDRESS>" "--cores" "8" "--app-id" "app-20160523221
423-0004" "--worker-url" "spark://Worker@<IP_ADDRESS>:51910"
16/05/23 22:14:42 INFO Worker: Asked to kill executor app-20160523221423-
0004/2
16/05/23 22:14:42 INFO ExecutorRunner: Runner thread for executor app-201
60523221423-0004/2 interrupted
16/05/23 22:14:42 INFO ExecutorRunner: Killing process!
16/05/23 22:14:42 ERROR FileAppender: Error writing stream to file /root/
spark/work/app-20160523221423-0004/2/stderr
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.j
ava:162)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:272
)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.spark.util.logging.FileAppender.appendStreamToFile(
FileAppender.scala:70)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply$mcV$sp(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala
:1741)
at org.apache.spark.util.logging.FileAppender$$anon$1.run(FileApp
ender.scala:38)
16/05/23 22:14:42 INFO Worker: Executor app-20160523221423-0004/2 finishe
d with state KILLED exitStatus 143
16/05/23 22:14:42 INFO Worker: Cleaning up local directories for applicat
ion app-20160523221423-0004
16/05/23 22:14:42 INFO ExternalShuffleBlockResolver: Application app-2016
0523221423-0004 removed, cleanupLocalDirs = true
16/05/23 22:52:31 INFO Worker: Asked to launch executor app-2016052322523
1-0005/0 for com.yahoo.ml.caffe.CaffeOnSpark
16/05/23 22:52:31 INFO SecurityManager: Changing view acls to: root
16/05/23 22:52:31 INFO SecurityManager: Changing modify acls to: root
16/05/23 22:52:31 INFO SecurityManager: SecurityManager: authentication d
isabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
16/05/23 22:52:31 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java
-7-openjdk-amd64/jre/bin/java" "-cp" "/root/ephemeral-hdfs/conf/:/root/sp
ark/conf/:/root/spark/lib/spark-assembly-1.6.0-hadoop2.4.0.jar:/root/spar
k/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.2
.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/ephemeral-hdfs/c
onf/" "-Xms12991M" "-Xmx12991M" "-Dspark.driver.port=45211" "-XX:MaxPermS
ize=256m" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--dri
ver-url" "spark://CoarseGrainedScheduler@<IP_ADDRESS>:45211" "--executor-
id" "0" "--hostname" "<IP_ADDRESS>" "--cores" "8" "--app-id" "app-201605232
25231-0005" "--worker-url" "spark://Worker@<IP_ADDRESS>:51910"
16/05/23 22:52:47 INFO Worker: Executor app-20160523225231-0005/0 finishe
d with state EXITED message Command exited with code 134 exitStatus 134
16/05/23 22:52:47 INFO Worker: Asked to launch executor app-2016052322523
1-0005/2 for com.yahoo.ml.caffe.CaffeOnSpark
16/05/23 22:52:47 INFO SecurityManager: Changing view acls to: root
16/05/23 22:52:47 INFO SecurityManager: Changing modify acls to: root
16/05/23 22:52:47 INFO SecurityManager: SecurityManager: authentication d
isabled; ui acls disabled; users with view permissions: Set(root); users
with modify permissions: Set(root)
16/05/23 22:52:47 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java
-7-openjdk-amd64/jre/bin/java" "-cp" "/root/ephemeral-hdfs/conf/:/root/sp
ark/conf/:/root/spark/lib/spark-assembly-1.6.0-hadoop2.4.0.jar:/root/spar
k/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.2
.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar:/root/ephemeral-hdfs/c
onf/" "-Xms12991M" "-Xmx12991M" "-Dspark.driver.port=45211" "-XX:MaxPermS
ize=256m" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--dri
ver-url" "spark://CoarseGrainedScheduler@<IP_ADDRESS>:45211" "--executor-
id" "2" "--hostname" "<IP_ADDRESS>" "--cores" "8" "--app-id" "app-201605232
25231-0005" "--worker-url" "spark://Worker@<IP_ADDRESS>:51910"
16/05/23 22:52:50 INFO Worker: Asked to kill executor app-20160523225231-
0005/2
16/05/23 22:52:50 INFO ExecutorRunner: Runner thread for executor app-201
60523225231-0005/2 interrupted
16/05/23 22:52:50 INFO ExecutorRunner: Killing process!
16/05/23 22:52:50 ERROR FileAppender: Error writing stream to file /root/
spark/work/app-20160523225231-0005/2/stderr
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.j
ava:162)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:272
)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.spark.util.logging.FileAppender.appendStreamToFile(
FileAppender.scala:70)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply$mcV$sp(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.logging.FileAppender$$anon$1$$anonfun$ru
n$1.apply(FileAppender.scala:39)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala
:1741)
at org.apache.spark.util.logging.FileAppender$$anon$1.run(FileApp
ender.scala:38)
16/05/23 22:52:50 INFO Worker: Executor app-20160523225231-0005/2 finishe
d with state KILLED exitStatus 143
16/05/23 22:52:50 INFO Worker: Cleaning up local directories for applicat
ion app-20160523225231-0005
16/05/23 22:52:50 INFO ExternalShuffleBlockResolver: Application app-2016
0523225231-0005 removed, cleanupLocalDirs = true

I use the following command to start the cluster:
${SPARK_HOME}/ec2/spark-ec2 --key-pair=${EC2_KEY} --identity-file=${EC2_PEM_FILE} --region=${EC2_REGION} --zone=${EC2_ZONE} --ebs-vol-size=50 --instance-type=${EC2_INSTANCE_TYPE} --master-instance-type=m4.xlarge --ami=${AMI_IMAGE} -s ${SPARK_WORKER_INSTANCES} --copy-aws-credentials --vpc-id=vpc-fe813496 --subnet-id=subnet-2d833645 --private-ips --hadoop-major-version=yarn --spark-version=1.6.0 --no-ganglia --user-data=/Users/myusername/work/CaffeOnSpark-master/scripts/ec2-cloud-config.txt launch CaffeOnSparkDemoVPC

I also see the following log from the work nodes:
root@ip-10-30-9-14:~/spark/work/app-20160525000805-0000/0# ls
caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar
hs_err_pid3378.log
lenet_memory_solver.prototxt
lenet_memory_train_test.prototxt
stderr
stdout
root@ip-10-30-9-14:~/spark/work/app-20160525000805-0000/0# more hs_err_pid3378.log

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f3cdfd26ff9, pid=3378, tid=139898812618496

JRE version: OpenJDK Runtime Environment (7.0_101) (build 1.7.0_101-b00
)
Java VM: OpenJDK 64-Bit Server VM (24.95-b01 mixed mode linux-amd64 com
pressed oops)
Derivative: IcedTea 2.6.6
Distribution: Ubuntu 14.04 LTS, package 7u101-2.6.6-0ubuntu<IP_ADDRESS>
Problematic frame:
C  [libc.so.6+0x97ff9]

Failed to write core dump. Core dumps have been disabled. To enable cor
e dumping, try "ulimit -c unlimited" before starting Java again

If you would like to submit a bug report, please include
instructions on how to reproduce the bug and visit:
http://icedtea.classpath.org/bugzilla
The crash happened outside the Java Virtual Machine in native code.
See problematic frame for where to report the bug.

Will check tomorrow but looks like permission errors on your filesystems
On Wednesday, May 25, 2016, syuquad<EMAIL_ADDRESS>wrote:

I also see the following log from the work nodes:
root@ip-10-30-9-14:~/spark/work/app-20160525000805-0000/0# ls
caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar
hs_err_pid3378.log
lenet_memory_solver.prototxt
lenet_memory_train_test.prototxt
stderr
stdout
root@ip-10-30-9-14:~/spark/work/app-20160525000805-0000/0# more
hs_err_pid3378.log

A fatal error has been detected by the Java Runtime Environment:

SIGSEGV (0xb) at pc=0x00007f3cdfd26ff9, pid=3378, tid=139898812618496

JRE version: OpenJDK Runtime Environment (7.0_101) (build 1.7.0_101-b00
)
Java VM: OpenJDK 64-Bit Server VM (24.95-b01 mixed mode linux-amd64 com
pressed oops)
Derivative: IcedTea 2.6.6 Distribution: Ubuntu 14.04 LTS, package
7u101-2.6.6-0ubuntu<IP_ADDRESS> Problematic frame: C [libc.so.6+0x97ff9]

Failed to write core dump. Core dumps have been disabled. To enable cor
e dumping, try "ulimit -c unlimited" before starting Java again

If you would like to submit a bug report, please include instructions on
how to reproduce the bug and visit: http://icedtea.classpath.org/bugzilla The
crash happened outside the Java Virtual Machine in native code. See
problematic frame for where to report the bug.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/yahoo/CaffeOnSpark/issues/67#issuecomment-221703902

If you want to specify/increase executor memory you can use these in spark submit:
--executor-memory 38g --conf spark.yarn.executor.memoryOverhead=16384

it doesn't help with "--executor-memory 38g --conf spark.yarn.executor.memoryOverhead=16384" or other adjusted values (the max is 13g on the g2.2xlarge box). Same errors.

I have a similar issue on my local deploy following https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_yarn
Master(no nvidia and cuda setup and caffeOnSpark build,caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar is copied from other slave node)
Two slave:
Slave1(one GPU)
Slave2(two GPU)
the model file:
lenet_memory_solver.prototxt.txt
lenet_memory_train_test.prototxt.txt
hadoop config file:
etc-hadoop.zip
After I use
export SPARK_WORKER_INSTANCES=2
export DEVICES=1
hadoop fs -rm -f hdfs:///mnist.model
hadoop fs -rm -r -f hdfs:///mnist_features_result
spark-submit --master yarn --deploy-mode cluster
--num-executors ${SPARK_WORKER_INSTANCES}
--files ${CAFFE_ON_SPARK}/data/lenet_memory_solver.prototxt,${CAFFE_ON_SPARK}/data/lenet_memory_train_test.prototxt
--conf spark.driver.extraLibraryPath="${LD_LIBRARY_PATH}"
--conf spark.executorEnv.LD_LIBRARY_PATH="${LD_LIBRARY_PATH}"
--class com.yahoo.ml.caffe.CaffeOnSpark
${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar
-train
-features accuracy,loss -label label
-conf lenet_memory_solver.prototxt
-devices ${DEVICES}
-connection ethernet
-model hdfs:///mnist.model
-output hdfs:///mnist_features_result
I got the error below:
User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID 7, heracles): java.lang.NullPointerException
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply$mcVI$sp(CaffeOnSpark.scala:153)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply(CaffeOnSpark.scala:149)
at com.yahoo.ml.caffe.CaffeOnSpark$$anonfun$train$1.apply(CaffeOnSpark.scala:149)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:927)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1858)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
But you can notice when I submit it using  spark mode(it's also one master and two slaves)
export MASTER_URL=spark://$(hostname):7077
export SPARK_WORKER_INSTANCES=1
export CORES_PER_WORKER=1
export TOTAL_CORES=$((${CORES_PER_WORKER}*${SPARK_WORKER_INSTANCES}))
pushd ${CAFFE_ON_SPARK}/data
rm -rf ${CAFFE_ON_SPARK}/mnist_lenet.model
rm -rf ${CAFFE_ON_SPARK}/lenet_features_result
spark-submit --master ${MASTER_URL}
--files ${CAFFE_ON_SPARK}/data/lenet_memory_solver.prototxt,${CAFFE_ON_SPARK}/data/lenet_memory_train_test.prototxt
--conf spark.cores.max=${TOTAL_CORES}
--conf spark.task.cpus=${CORES_PER_WORKER}
--conf spark.driver.extraLibraryPath="${LD_LIBRARY_PATH}"
--conf spark.executorEnv.LD_LIBRARY_PATH="${LD_LIBRARY_PATH}"
--class com.yahoo.ml.caffe.CaffeOnSpark
${CAFFE_ON_SPARK}/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar
-train
-features accuracy,loss -label label
-conf lenet_memory_solver.prototxt
-clusterSize ${SPARK_WORKER_INSTANCES}
-devices 1
-connection ethernet
-model hdfs:///mnist.model
-output hdfs:///mnist_features_result
Everything is OK.

This is usually caused by incorrect settings.
I believe syuquad has solved this issue. @syuquad, do you want to share your solution?

I didn’t try to run it with yarn. I only run it with the local mode.
From: Jun Shi
Reply-To: yahoo/CaffeOnSpark
Date: Wednesday, July 6, 2016 at 1:57 AM
To: yahoo/CaffeOnSpark
Cc: Shenxi Yu, Mention
Subject: Re: [yahoo/CaffeOnSpark] NullPointerException when Running CaffeOnSpark on EC2 (#67)
This is usually caused by incorrect settings.
I believe syuquad has solved this issue. @syuquadhttps://github.com/syuquad, do you want to share your solution?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/yahoo/CaffeOnSpark/issues/67#issuecomment-230716465, or mute the threadhttps://github.com/notifications/unsubscribe/ASidFCQaIEeVKzjP8bXEF-aQbZW6-_I_ks5qS24XgaJpZM4IhyMG.

Hello @syuquad , could you share your solution for this error?
The root cause maybe the same for the similar error.
Thank you!

Hi,
You got the exception with
spark-submit --master yarn --deploy-mode cluster….
That’s different from my case. I got the exception with
spark-submit --master ${MASTER_URL} ….
After I use Jun’s ami, that exception is gone. I never use
spark-submit --master yarn --deploy-mode cluster….
From: guchensmile
Reply-To: yahoo/CaffeOnSpark
Date: Thursday, July 7, 2016 at 1:02 AM
To: yahoo/CaffeOnSpark
Cc: Shenxi Yu, Mention
Subject: Re: [yahoo/CaffeOnSpark] NullPointerException when Running CaffeOnSpark on EC2 (#67)
Hello @syuquadhttps://github.com/syuquad , could you share your solution for this error?
The root cause maybe the same for the similar error.
Thank you!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/yahoo/CaffeOnSpark/issues/67#issuecomment-231010010, or mute the threadhttps://github.com/notifications/unsubscribe/ASidFIO-g1ADQ8Jt3U6YqPnE12IArx-Cks5qTLKJgaJpZM4IhyMG.

OK, I will try reconfigure all for yarn mode.

No update. Closing this. Reopen if required


Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 31 - Encryption(ssl/tls) handshake failed)

Docker container response : Getting issue while api call in .Net Core 6
Microsoft.Data.SqlClient.SqlException (0x80131904): A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 31 - Encryption(ssl/tls) handshake failed)\n ---> System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.\n   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](TIOAdapter adapter)\n   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)\n   at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)\n   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)\n   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost)\n   at Microsoft.Data.SqlClient.SNI.SNITCPHandle.EnableSsl(UInt32 options)\n   at Microsoft.Data.SqlClient.SNI.SNIProxy.EnableSsl(SNIHandle handle, UInt32 options)\n   at Microsoft.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action1 wrapCloseInAction)\n   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\n   at Microsoft.Data.SqlClient.TdsParser.ConsumePreLoginHandshake(Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean& marsCapable, Boolean& fedAuthRequired)\n   at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, Boolean encrypt, Boolean trustServerCert, Boolean integratedSecurity, Boolean withFailover, SqlAuthenticationMethod authType)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean ignoreSniOpenTimeout, TimeoutTimer timeout, Boolean withFailover)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)\n   at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.CreatePooledConnection(DbConnectionPool pool, DbConnection owningObject, DbConnectionOptions options, DbConnectionPoolKey poolKey, DbConnectionOptions userOptions)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.ProviderBase.DbConnectionClosed.TryOpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource1 retry, SqlConnectionOverrides overrides)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Microsoft.Data.SqlClient.SqlConnection.Open()\n   at WebApiDotNetCoreNew.DataAccessLayer.Emp.AddEmployee(EmpDetails empDetail) in /src/DataAccessLayer/Emp.cs:line 22\nClientConnectionId:876225fe-0bf3-4c15-a6f3-7ab762aa9d74
Docker file code :
FROM mcr.microsoft.com/dotnet/aspnet:6.0-alpine3.14 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
Install cultures (same approach as Alpine SDK image)
RUN apk add --no-cache icu-libs
Disable the invariant mode (set in base image)
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=false
FROM mcr.microsoft.com/dotnet/sdk:6.0-alpine3.14 AS build
WORKDIR /src
COPY ["WebApiDotNetCoreNew.csproj", "./"]
RUN dotnet restore "WebApiDotNetCoreNew.csproj"
COPY . .
WORKDIR "/src/"
RUN dotnet build "WebApiDotNetCoreNew.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "WebApiDotNetCoreNew.csproj" -c Release -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "WebApiDotNetCoreNew.dll"]
"ConnectionStrings": {
"DefaultConnection": "Data Source=xxx.xxx.xx.xx; Persist Security Info=True; Initial Catalog=EmployeeDB; User ID=user;Password=Abc@123"
},
Note : Also added TrustServerCertificate=True;MultiSubnetFailover=True in connection but getting same response
Operating System : Windows 10
Software Used :
Visual studio 2022
Sql server :
Product : Microsoft sql server express(64 bit)
Version : 12.0.2000.8
I have tried with .Net core 3.1 it's giving response successfully on same system.

Add this to your connection string TrustServerCertificate=True;Encrypt=false
Fix your database encryption if you need to have an encrypted connection between server and database.

Issue is resolved when I updated the Version of Sql Server 15.0.4236.7
Thank you!


Validation Edits (CAN BE MERGED NOW)

Requirements

[x] You have completed your website.
[x] The website is reachable.
[x] The CNAME record doesn't contain https:// or /.
[x] There is sufficient information at the owner field.
[x] There is no NS Records (Enforced as of Sepetember 4th, 2024)

Description

Link to Website

This can be merged when ever, as this should be good


Create a macro that injects compute_note_hash_and_nullifier

compute_note_hash_and_nullifier is a boilerplate and we should inject it with macros. We want to give devs the ability to define the function on their own. This is straightforward to do because we can inject it only if it doesn't already exist.
To generate compute_note_hash_and_nullifier we need:

names of the NoteInterface (e.g. AddressNoteMethods) and its import path (it might not be imported in the contract because the interface could be used in a custom type and not directly passed to e.g. Set in Storage struct),
we need to somehow extract the storage slot of the notes defined in the Storage struct.

To make it more complicated we need to extract the info above only when the Notes are used in a private context and the information about the NoteInterface might be nested deep in custom type definitions like is done for example in our Token contract:

This is the function definition in the Token contract:

Highlighted is the info we need to obtain.
@Maddiaa0 pointed out that we might need to annotate each the storage struct with state visibility of each type (e.g. with #[storage(private)]) but I am not sure if it helps us with extracting the NoteInterface name.
@sirasistant @Maddiaa0 any ideas how to solve this?
Side note: With Sean's help today I've already implemented the compute_note_hash_and_nullifier presence check in this PR. The issue with that PR is that it forces users to define the function even when they work with public state only. Since solving that issue is also quite complicated I decided to bring up this one because that is the end-game.

FWIW this relates to having a more declarative view of the storage layout, which also popped up in https://github.com/AztecProtocol/aztec-packages/issues/2806, since that'd allow type-safe access to storage via get(Public|Private)StorageAt calls.

With https://github.com/AztecProtocol/aztec-packages/issues/2918 this is now relatively simple to do. We'd find all implementations of NoteInterface, and then for each of them we produce a new case in the switch statement, checking against its note type id and calling the deserialize function on a match. This would also be a good opportunity to implement https://github.com/AztecProtocol/aztec-packages/issues/4520.
Note that to find these impls we'll need to traverse the typed AST, which is passed to transform_hir and not transform. Since we're first checking for the existence of compute_note_hash_and_nullifier on the untyped AST, we'd need to move this check until later on the compilation pipeline (i.e. inside transform_hir), potentially making it so that we only provide the function if it is not already user-defined.

We also need to pick an array size that is large enough to fit all of the different types of notes in a contract. This can be done by looking at the global len values (e.g. VALUE_NOTE_LEN), or ideally by finding the generic value over which the NoteInterface trait is implemented.

https://github.com/AztecProtocol/aztec-packages/blob/827afd10edfca8b2c8273742717f039981543194/noir-projects/noir-contracts/contracts/delegated_on_contract/src/main.nr#L47
This should now be able to be resolved.


Return lab status in ClientLibrary

What does this PR do?
When using the virl2_client.virl2_client.ClientLibrary you are able to return all labs in a list and the labs as a list of lab objects. The issue is that when you return all labs it would great to also have the status.
Recommendations for how to test this, or anything you are worried about?
I thought about implementing this in the get_lab_list() since it already returns a list. But it only returns a list of lab IDs.

Thanks, @labeveryday !  Just curious why you want to push this into the library vs. having your application support grabbing status.  This is what I do in cmlutils:
    lab_ids = client.get_lab_list(all_users)
    for id in lab_ids:
        labs.append(client.join_existing_lab(id))
    ...
    # print(labs[0].state())

This then gives me all the lab properties.  If I were going to push some of this back to the library, I might have the method return the whole lab object as I might want something other than title or state (and I do in cmlutils), and this would save me a call to join_existing_lab().

lab_ids = client.get_lab_list(all_users)
    for id in lab_ids:
        labs.append(client.join_existing_lab(id))
    ...
    # print(labs[0].state())

As I was writing code to implement this I found that I had I had to do more work to get the status of all labs. So I created a function that tells me the details like id, lab title, and then the status of the lab. From there I can now start or stop my lab based off of id. Rather than going through all of the iterations.

Thanks for your interest and contribution here, @labeveryday. I do agree with @xorrkaz here...  this is code that should go into the application side of things, not into the library itself.  it really depends on your specific use case what attributes of a lab is returned as part of your implementation here.  and i think it's easy enough to have this implemented on the application side of things, not inside of the library.


Big data analytics with enterprise-grade security using Azure Synapse

Hi, can you send me the visio file of this architecture?

@zeevtur Can you share the document that you are referring to?

Here is the link:
https://arch-center.azureedge.net/big-data-analytics-enterprise-grade-security.vsdx


has no exported member

Hi, after I install the npm package and build my project, I get many errors about has no exported member.
Those are the errors
ModelSignal
InputSignal
OutputEmitterSignalRef

Hi @yusufskebbe, if you are cloning the project and running 'npm install' please remove '^' from  "ngx-countries-dropdown": "^1.0.4", package. After that it should work fine.
though i will push the latest changes after upgrading to Angular 17 and using latest version of 'ngx-countries-dropdown'
Thanks,
Kapil


Update AIBrainDebuggable.cs

Call StoreLastKnownPosition to match superclass.
This works in my Top Down Engine environment - I don't know if AIBrain is the same in Corgi?

@bitindicium the AIBrain is the same, unless Corgi/TopDown versions are a bit out of sync, but at the moment this should not be the case.


titleCase doesn't accept options

Unlike capitalCase, titleCase doesn't accept second options argument. Is that possible to extend it?
Also, the documentation doesn't make it clear what's the difference between the two. It only refers to some 'English rules' but example output doesn't demo the usage of these rules.

What options do you want to add? Short answer: no it's not possible, it's a much different package to the rest and not for converting between "cases".

What options do you want to add?

These options: https://github.com/blakeembrey/change-case#options, for example splitRegexp.

Short answer: no it's not possible, it's a much different package to the rest and not for converting between "cases".

As I told, the documentation doesn't explain what exactly is "much different" in this package. See:
capitalCase("test string");
//=> "Test String"

titleCase("a simple test");
//=> "A Simple Test"

From these examples, one can't really infer why the first one can accept options and the second one can't.

Actually, the original issue I had is that titleCase kind of "broke" after upgrading from 2.1.1 to 3.0.2. titleCase("some_text") used to return Some Text and after upgrading it started to return Some_text. That's why I started to look for options, and discovered that titleCase was ignoring them. Only then I discovered capitalCase (which apparently works as titleCase@2 used to work), and reported the confusion and inconsistencies.
So what I think worth doing is:

Clearly state that titleCase is the advanced version of capitalCase or something along the lines. Come up with better example for titleCase which would show the difference between the two.
Clearly outline that unlike most of the function, titleCase doesn't accept options.

You’re welcome to submit a PR. Does this explain it enough for you? https://en.wikipedia.org/wiki/Letter_case#Title_case
That is linked in the README of the package. Those two packages are unrelated to each other. Also, there’s not a single suggestion in the library that talks about options for title case anymore. If you’d like a better description, feel free to submit a PR.


Use lewton instead of libvorbis and some little improvements

Use lewton instead of libvorbis.  I need to come up with a better concept for this upstream.

Out of similar reasons, I didn't do a PR for the gif crate yet.

This looks good to me, but I'd like sign-off from a Vorbis expert.
@metajack Do you have thoughts on lewton?

A few points about lewton:

Floor0 is probably not relevant anymore, but its still supported, just like libvorbis does. stb_vorbis doesn't do it for example.
On rust nightly-2016-10-19, it is about 20% slower than libvorbis for floor 1, but for floor 0 its faster (not that floor 0 is relevant, but hey, its faster xD). Performance in both scenarios should be fast enough for all purposes.
all uses of unsafe in the crate are two one liner functions doing something terribly trivial.
For all files I've tested it produces almost exactly the same results as libvorbis. This means that in all the floor 1 files I've tested, I haven't found a single sample where the two decoders would be farther away than 1 (for 16 bit signed samples). For a floor 0 file I found one sample where the two implementations differed by 1.
You can play around yourself by cd'ing to the dev/cmp directory and running cargo run --release vals <file name here> to compare the output or cargo run --release perf <file name here> to compare the speed with libvorbis. cargo test --release and cargo run --release bench will run benchmarks and value comparisons for a set of test files.

This looks a lot simpler, and I'm a big fan of the removal of most unsafe code. We should make sure we are opting into the ieee754 feature when used.
Also, we should probably have some unit tests that decode simple files just as a sanity check.
I'd love to see benchmarks of this vs. libvorbis so we know what optimization work needs to be done down the road.

@metajack if you run the benchmark tool I described in the post above you'll get (these files are automatically downloaded from fairly stable locations):
Comparing speed for bwv_1043_vivace.ogg : libvorbis=0.8301s we=1.0253s difference=1.24x
Comparing speed for bwv_543_fuge.ogg    : libvorbis=1.3267s we=1.6383s difference=1.23x
Comparing speed for maple_leaf_rag.ogg  : libvorbis=0.3890s we=0.4762s difference=1.22x
Comparing speed for hoelle_rache.ogg    : libvorbis=0.6277s we=0.7707s difference=1.23x
Comparing speed for thingy-floor0.ogg   : libvorbis=0.3503s we=0.2717s difference=0.78x

Overall time spent for decoding by libvorbis: 3.5238s
Overall time spent for decoding by us: 4.1823s
Overall ratio of difference: 1.19x

@metajack I've enabled the ieee754 feature as requested.
About the sanity test unit tests: Where do you want them, here or in the lewton repo, because there are already such test files inside the lewton repo (you cd into dev/cmp then run cargo test --release)?

I think here is a good place for the tests.

I'm totally fine with 20% perf regression in exchange for safety. We can optimize lewton if needed.
I would like some sanity tests here, but they can be cursory since lewton is already also testing. Mostly I want to protect against the error case where rust-media still builds but doesn't actual work; that situation is common on Servo for android where we don't have any smoke tests.

Added an unit test. I couldn't use the ogg container decoder from rust-media as it isn't exposed via the high level API yet.

Awesome, thanks!


add option to get report images with screenshot

@troibe what do you mean about integrating tcl with python?

@troibe what do you mean about integrating tcl with python?

As in adding a unit test that checks if this feature calls this tcl function.

Added SC-611


check: lint: ineffassign in qemu.go

We were not checking the return err from addDeviceToBridge, which
can validly error out. lint/ineffassign picked this up.
Let's check the error return then...
Fixes: #650
Signed-off-by: Graham whaley<EMAIL_ADDRESS>

lgtm

LGTM


[multistage][agg] support agg with filter

TODOs
Major

IS NULL and IS NOT NULL as agg filter is not supported
SQL basic call-binding and Aggregate call-binding has mismatched info regarding filterArg and thus some time create unsupported nullable/non-nullable type during inferReturnType method.
mixed/conflict filter is not supported, for example SELECT COUNT(*) FILTER (WHERE a = 1 OR a = 2) FROM tbl WHERE a = 2 will result in a compilation error (occurred on v1 FilterAggregateOperator, can be individually fixed)
Pinot behavior is to not return any rows if all of the aggregate filter function has no matches --> this conflicts with standard behavior, which should return a null.

Minor

ServerPlanRequestVisitor: agg with filter and group-by causes conversion issue on v1 if the group-by field is not in the select list -- work around: selecting out all the group-by fields
IN clause not supported: FILTER WHERE clause with IN hard-wired to translate into subquery in calcite for some reason -- work around: use OR and equal

Codecov Report

Merging #11144 (0f55841) into master (cfc3abc) will increase coverage by 0.00%.
The diff coverage is 0.00%.

@@            Coverage Diff            @@
##           master   #11144     +/-   ##
=========================================
  Coverage    0.11%    0.11%
=========================================
  Files        2204     2150     -54
  Lines      118324   115988   -2336
  Branches    17912    17648    -264
=========================================
  Hits          137      137
+ Misses     118167   115831   -2336
  Partials       20       20

Flag
Coverage Δ

integration1temurin11
?

integration1temurin17
?

integration1temurin20
?

integration2temurin20
?

unittests1temurin11
?

unittests1temurin17
?

unittests1temurin20
?

unittests2temurin11
?

unittests2temurin17
0.11% <0.00%> (-0.01%)
:arrow_down:

unittests2temurin20
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

.../apache/pinot/common/datablock/DataBlockUtils.java
0.00% <0.00%> (ø)

...inot/core/common/IntermediateStageBlockValSet.java
0.00% <0.00%> (ø)

...pinot/query/parser/CalciteRexExpressionParser.java
0.00% <0.00%> (ø)

...he/pinot/query/planner/plannode/AggregateNode.java
0.00% <0.00%> (ø)

...inot/query/runtime/operator/AggregateOperator.java
0.00% <0.00%> (ø)

...untime/operator/MultistageAggregationExecutor.java
0.00% <0.00%> (ø)

...ry/runtime/operator/MultistageGroupByExecutor.java
0.00% <0.00%> (ø)

.../pinot/query/runtime/plan/PhysicalPlanVisitor.java
0.00% <0.00%> (ø)

.../runtime/plan/server/ServerPlanRequestVisitor.java
0.00% <0.00%> (ø)

... and 65 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Question: Where are logs saved?

Hi,
Sorry for noob question. Am runnin RTSPtoWeb docker and am desperately looking for the logs ^^
Should i ssh into the docker and go to /var/log or is there some hidden config I did not take into consideration?

docker logs --tail 100 --follow containerName-or-id
https://docs.docker.com/engine/reference/commandline/logs/


Application fails installation on Nexus 6

Nexus 6 Build number LVY48E
Android 5.1.1
When installing the apk found at this link https://github.com/nowsecure/android-vts/tree/master/bin
I receive the following message in a pop up dialog:
Parse error
There was a problem parsing the package.
ADB logcat output:
D/        (27741): Zip: EOCD not found, /storage/emulated/0/Download/VulnChecker.apk is not zip
W/zipro   (27741): Error opening archive /storage/emulated/0/Download/VulnChecker.apk: Invalid file
D/asset   (27741): failed to open Zip archive '/storage/emulated/0/Download/VulnChecker.apk'
W/PackageInstaller(27741): Parse error when parsing manifest. Discontinuing installation

It appears that downloading it via mobile causes the problem.  I used ES file explorer to download the file and installation failed.  I was able to download and "adb install" without an issue though.


Increase mouse wheel scroll

I propose to increase the factor of the mouse wheel scroll in BlInfiniteMouseScrollListener>>mouseWheelEvent:
Actually this is 10, when I'm using 100 this is fine for me, as you see :

The demo is here : https://github.com/plantec/Toplo/blob/master/src/Toplo/ToMiniBrowser.class.st
But I am on Windows, before to fix that I need to know what are the behavior in other OS.
Do you have same behavior on Mac and Linux ?
My configuration :
Windows 11 Pro
Pharo-11.0.0+build.421 64bits
Logitech G502 mouse

I think we can add a system setting to customize the wheel multiplier, and maybe use an initial value based on OSPlatform current.

A complete text about smooth scrolling: https://pavelfatin.com/scrolling-with-pleasure/
An SDL2 based app that offers wheel and mouse sensitivity multiplier as user setting: https://guavaman.com/projects/rewired/docs/RewiredEditor.html
A related issue: Mouse scroll delta is not scaled properly
A related question: https://stackoverflow.com/questions/63465961/smooth-scrolling-in-sdl
Another: https://gamedev.stackexchange.com/questions/109182/sdl2-event-mouse-wheel-trouble

(For the record, I link: https://github.com/pharo-graphics/bloc-pac/commit/cd594eaa9db9aa514bdf0ac5372ffb2f07e57dbe)

fixed by https://github.com/pharo-graphics/Bloc/commit/36414289fe89269bef6167dcab930b3150087219


signinSilent not working

Hi,
I have the oidc implemented following the Angular2 example.
It works to signinRedirect and to signoutRedirect , also automaticSilentRenew is set to true and works, when the token expires I receive a new valid one.
The issue is that the signinSilent is not working, when I call the method I'm still logged in with a valid token and I receive :
user not found in storage
signinRedirectCallback successful, signed in sub:  8608ac66dd2d4cfb122e40d0c6050e43
Error: Frame window timed out

Even the frame error I receive a new valid token, so it almost works fine.
I also managed to debug the silent iframe link, trying in a new tab it says :
No matching state found in storage
user not found in storage

Any ideas ?
PS : The oidc config is set based on Angular2 example, redirect_uri and silent_redirect_uri are simple valid HTML files, are not part of the SPA (same as in the example)

I have redirect uri as just component in my angular spa an had same issue
Error: Frame window timed out
core.es5.js:1084 ERROR Error: Uncaught (in promise): Error: Frame window timed out
Error: Frame window timed out
    at t._error (oidc-client.min.js:74) [angular]
    at t._timeout (oidc-client.min.js:74) [angular]
    at Object.onInvokeTask (core.es5.js:4140) [angular]
    at timer (zone.js:1655) [<root>]
    at t._error (oidc-client.min.js:74) [angular]
    at t._timeout (oidc-client.min.js:74) [angular]
    at Object.onInvokeTask (core.es5.js:4140) [angular]
    at timer (zone.js:1655) [<root>]
    at resolvePromise (zone.js:710) [angular]
    at resolvePromise (zone.js:681) [angular]
    at polyfills.bundle.js:6308:17 [angular]
    at Object.onInvokeTask (core.es5.js:4140) [angular]
    at drainMicroTaskQueue (zone.js:591) [<root>]
    at ZoneTask.invoke (zone.js:464) [<root>]
    at timer (zone.js:1655) [<root>]

Just figure out that my frame time out was due the fact that I was making router.navigateinside promise:
silentRenew(route: string) {
    this.userManager.signinSilent().then((user: User) => {
      this.router.navigate([route]);
    });
  }

After I took it out I've got this error:

Just was looking trough the issues may be this one is an answer why the error is taking place https://github.com/IdentityModel/oidc-client-js/issues/172#issuecomment-257851885

The iframe will time out if the authorization endpoint does not honor prompt=none, or if it's showing an error of some sort (that the user can't see).

I just looked at the sample and  configured that with my identity server https://github.com/IdentityServer/IdentityServer4.Samples/tree/release/Clients/src/JsOidc/wwwroot to make sure that it is working. Then in my angular SPA I setup separate components to handle signinSilentCallback and signinRedirectCallback but got in to the same Frame window timeout issue
The after that the only difference between my SPA and a sample was that the sample was using html files for call backs so I just replicate that in my SPA and it is works now.

I'm already using separate files for redirects, not sure what's wrong with mine

@tibbus try to setup https://github.com/IdentityServer/IdentityServer4.Samples/tree/release/Clients/src/JsOidc/wwwroot against your idserv and see if it is working.

@kuncevic Thanks, that's a good idea, I tried before with the angular2 sample and didn't worked, but this one is more simple and clear

All set on this issue -- can we close?

At this moment I use the auto silent renew instead of this and don't have the time to test it to see where the issue was, so I will close it.

@tibbus Do you get this issue while open the application in multiple tabs ?

@kuncevic Thank you, really appreciate your answer above. I was getting the timeout error because I was referencing the oidc client through a CDN and I had caching disabled locally, so after various intervals there would be an error retrieving the client library resulting in a frame timeout. Having a local copy in my assets folder resolved the issue.


Move algo finding to cudnnFind*Ex

Move from cudnnFind* to cudnnFind*Ex for algorithm choosing.
Properly gate finding dgrad algorithm to when dgrad is actually needed to be computed.
This is a) better and b) fixes some potential deadlock issues in multi-GPU runs relating to freeing of device memory.

Thanks Simon :)


"Country" is misspelled in Holiday Dates error message

In line 271, "contry" should be spelled as "country"

Fixed in upcoming 1.15 release.


bug(ObservableMap): Avoid no-op MapChangeRecord

i.e., notifyChange(new MapChangeRecord(this, 'key', 'someValue', 'someValue')).
Was needed to land the previous version internally.

@matanlurey ya'll gotta dartfmt! 😁

Finally got around to this. Will merge after travis.


A few small tweaks to support Arrays of const T objects [array_const_int_fix]

This resolves #1026.

@mlstowell, can you take a quick look?


This service isn't available in your region or app version.

{'error': {'code': 403, 'message': "This service isn't available in your region or app version.", 'metadata': None, '_data': {1: 403, 3: "This service isn't available in your region or app version."}}}
[2023-04-27 19:54:57] Not support Square: This service isn't available in your region or app version.
When I use login_web, I encounter the error mentioned above. Has anyone experienced this issue? The number is from Thailand.

It's not CHRLINE problem, please make sure you use the service correctly


Errors when running after first time in cross-dep bins workspaces

Title
Description

Version
v0.23.2

Type
Issue

node
v10.15.0

Operating System
MacOS Mojave (10.14.4)

Short Description
bolt errors when running after the first time in cross-dep bins workspaces.

Detailed description
The original issue was that bolt didn't manage cross dependencies binary files (#126) and was fixed in #191. However, even bolt now creates the symlink, a second run will fail with the error below this table.

NOTE: Keep in mind that I'm also replicating this error via running bolt internal tests.
Error: EEXIST: file already exists, symlink '../../../bar/bin.js' -> '/private/var/folders/m3/hdz1bgz94vz5rc3x16gm7jr80000gn/T/0fd44005f5f3b1950271e2eff9b65abc/workspace-with-bin/packages/foo/node_modules/.bin/bar'

We were able to work around this by declaring the bin as an object instead of a string.


⚠ Bofrost.de has degraded performance

In 99f0875, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 938 ms

Resolved: Bofrost.de performance has improved in 07f5bc1 after 6 minutes.


Request: Search for text in strings/literals

Search only in strings or literals
A great feature that could be added to Visual Studio Code is the ability to look for text only within literals (or the reverse, i.e. all matches outside literals).
For instance, if I want to look for the literal string "button" in an HTML document, chances are, I will have to go over tens or even thousands of results because I have a lot of <button> tags. On the other hand, if such a feature could be implemented, I can easily narrow down the matches to get exactly what I want.
Implementation
Personally, I think this part would not be too difficult. If you can syntax-highlight strings, it means that you can identify strings and this feature would not be too far away.

Previous response from @roblourens https://github.com/microsoft/vscode/issues/72912#issuecomment-487075936
You can probably use a regex search to narrow down your matches.

Language-specific search features are not practical for us to implement right now

Just wanted to +1 this. It's one of the few features of the Webstorm/PHPStorm ecosystem that I miss now that I've been working in VSCode for the past year. Totally get that it would not be an easy feature to build, maybe best left for a plugin. But would be very useful.


DX: updagte @PhpCsFixer set config for phpdoc_order rule

follows #6466

use same config for PhpCsFixer ruleset
make memo to update default config for next major

cc @kusab85 @julienfalque

Coverage remained the same at 92.938% when pulling dec32476f92d2048916c70c7ce9e88e630870bda on keradus:6466_b into 26204b5564993a6ddfcecfcbf55e0787cbbc3937 on FriendsOfPHP:master.

as you see almost no changes in codebase in the end. we are more and more strictly typed and don't rely on those annotations ;d


Dependency "JTAppleCalendar" has no shared framework schemes for any of the platforms: iOS

Version Number: 8.0.2
Description
While trying to use the latest version of JTAppleCalendar via Carthage the build fails with the following message: *** Skipped building JTAppleCalendar due to the error: Dependency "JTAppleCalendar" has no shared framework schemes for any of the platforms: iOS
Steps To Reproduce
Simply add github "patchthecode/JTAppleCalendar" to cartfile and run carthage
Expected Behavior
The build should succeed
Additional Context

Use master branch for now until i release 8.0.3
github "patchthecode/JTAppleCalendar" "master"
closing this issue.
let meknow if youre still stuck.


Pass JAVA15_HOME in build

With this commit we pass the environment variable JAVA15_HOME in the
build if it is defined.

@elasticmachine test this please


Allow passing of ax to Subarray.peek, do not always call tight_layout

What is the use case here?  Really, you should just use an ArrayDisplay directly to do any more fancy plotting, the peek() function is for debugging or a quick look at what is there, not to make useful figures.  One of the design considerations early on in ctapipe was to avoid having plotting functions inside classes (after lessons learned about deep circular dependencies), which I agree is sort of violated already with this and the atmosphere module...

The main thing I wanted to address was that I got a large warning from matplotlib from the .peek() methd in a notebook where I had constrained layout enabled by default for all figures.
Then, I am of the strong opinion that any matplotlib related function should allow overriding in which axes the thing is plotted, so I added that.


Linked to packages docs instead of duplicating.

Fixes #1848

@jasonpet please review, now the links should take user to individual readme files.

ah missing websocket

websocket added ae73044c02b81b1daa408cec26cd16fe63dcf9dc

@jasonpet fix link and repo periods here 5cdb972f96a89132ad58f6e00bf965b4c9fb79be

Updated outdated link to catalog.md 62f5ee04fcccd5c482dfb23a8deb7b34b210a8ea

merged bb5b7cb1d0dbd69355d6b2508ce006db22ae2584


php83.extensions.soap: Re-enable tests

Description of changes
Only the online ones were failing.
Let’s disable those using environment variable.
Also, on PHP 8.4 the tests are completely broken
PHP Warning:  PHP Startup: Unable to load dynamic library 'soap.so' (tried: /build/php-8.4.0beta3/ext/soap/modules/soap.so (/build/php-8.4.0beta3/ext/soap/modules/soap.so: undefined symbol: php_set_session_var)

presumably because of https://github.com/php/php-src/commit/c7797fc8c0503c18791fd651f5ebdd7164b4bc48
so let’s disable them altogether there.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Looks like the Darwin fails on 8.4 as well, although not fatally:
Bug #49278 (SoapClient::__getLastResponseHeaders returns NULL if wsdl operation !has output) [tests/bugs/bug49278.phpt]
Bug #55639 (Digest authentication dont work) [tests/bugs/bug55639.phpt]
Bug #76232 (SoapClient Cookie Header Semicolon) [tests/bugs/bug76232.phpt]
SOAP customized Content-Type, eg. SwA use case [tests/custom_content_type.phpt]

And on 8.1:
SOAP customized Content-Type, eg. SwA use case [tests/custom_content_type.phpt]

Superseded by https://github.com/NixOS/nixpkgs/pull/345893


JCDP vs ESAS

I received an email from Tim Dunn from JNCC... This regarded the possibilities on cetacean data from ESAS flowing into JCDP... My preferred option would be that cetacean data can easily be downloaded from ESAS, and in turn submitted to JCDP. The only feature needed in that case would be the possibility to query cetaceans on the download page. So it would be nice to have such a query with with for example the following options:

birds
cetaceans
all observations

Not urgent, to be discussed after actual data migration!

Thanks @nicolasvanermen
the dataflow would be the other way (Cetaceans interested in ESAS data, not the way you describe). We will take this up on the JCDP gihub

@neil-ices-dk any progress on this? Can the issue be closed here?

@peterdesmet and @nicolasvanermen there is an item on the WGJCDP meeting coming up on the 15th Sept. https://www.ices.dk/news-and-events/meeting-calendar/Pages/ICES-CalendarDisp.aspx?sd=106fc5ba-b653-ec11-9296-00155d2c0c14
Nicolas - not sure if Nikki Taylor (JNCC) has reached out to you, but it could be an idea if you speak with her and/or attend the online WGJCDP meeting.


Outdated examples or outdated README

https://github.com/admin-shell-io/aas-specs/blob/21e68502e367b72fd82cfa29488a686cbd3892a5/schemas/json/examples/README.md?plain=1#L3
According to this line, the examples are from V3.0RC02
though the latest commit message on the examples implies V3.0

Thanks, @jh-isw ! I'll try to fix it tomorrow.

You're welcome :)


[kubernetes-dashboard] Adds docs about fullnameOverride for kubectl proxy

If you want to use 'localhost:8001/ui' as a shorthand to get
to the kubernetes-dashboard, the service name must be
'kubernetes-dashboard', but this is not typically what
Helm generates. This commit adds a short note
about how to override that.
I initially looked into changing the value which is used to set service.name, but it looks like that's been revised a few times already. That might be a preferable change (to only override one value instead of several), but I didn't want go backwards on previous work.

What this PR does / why we need it:
Adds extra documentation for people wanting to use the kubernetes-dashboard chart in combination with kubectl proxy and the 'localhost:8001/ui' shorthand.
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Happy to find a different technique if this isn't the best solution, but I think it'd be good to have some documentation for how people can use the Helm chart in combination with the kubectl proxy shorthand address.

/assign @scottrigby
as suggested by the bot

Please increment the version in chart.yaml

/lgtm

/lgtm


Add support for Licenses API

Announcement: https://developer.github.com/changes/2015-03-09-licenses-api/
Docs: https://developer.github.com/v3/licenses/

Working on this now.