at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py", line 111, in main
process()
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/worker.py", line 106, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 2346, in pipeline_func
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 317, in func
File "/usr/local/spark-1.6-hadoop-2.4/bin/tfspark.zip/com/yahoo/ml/tf/TFSparkNode.py", line 452, in _inference
File "./tfspark.zip/com/yahoo/ml/tf/TFSparkNode.py", line 64, in _get_manager
logging.info("Connected to TFSparkNode.mgr on {0}, ppid={1}, state={2}".format(host, ppid, str(TFSparkNode.mgr.get('state'))))
AttributeError: 'NoneType' object has no attribute 'get'
at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:166)
at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:207)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:125)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
... 1 more

17/04/19 10:43:51 WARN scheduler.TaskSetManager: Lost task 8.3 in stage 1.0 (TID 35, hdfsnn2): TaskKilled (killed intentionally)
17/04/19 10:43:51 WARN scheduler.TaskSetManager: Lost task 1.3 in stage 1.0 (TID 31, hdfsnn2): TaskKilled (killed intentionally)
17/04/19 10:43:51 WARN scheduler.TaskSetManager: Lost task 5.3 in stage 1.0 (TID 32, hdfsnn): TaskKilled (killed intentionally)
17/04/19 10:43:51 WARN scheduler.TaskSetManager: Lost task 0.3 in stage 1.0 (TID 33, hdfsnn): TaskKilled (killed intentionally)
17/04/19 10:43:51 WARN scheduler.TaskSetManager: Lost task 4.3 in stage 1.0 (TID 34, hdfsnn): TaskKilled (killed intentionally)
17/04/19 10:43:52 WARN scheduler.TaskSetManager: Lost task 7.0 in stage 1.0 (TID 10, hdfsnn): TaskKilled (killed intentionally)
17/04/19 10:43:52 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
17/04/19 10:43:53 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on <IP_ADDRESS>:22058 in memory (size: 40.0 KB, free: 4.8 GB)
17/04/19 10:43:53 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on hdfsnn:31750 in memory (size: 40.0 KB, free: 4.8 GB)
17/04/19 10:43:53 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on hdfsnn2:26094 in memory (size: 40.0 KB, free: 4.8 GB)
17/04/19 10:43:53 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on hdfsnn2:27520 in memory (size: 40.0 KB, free: 4.8 GB)
17/04/19 10:43:53 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on hdfsnn:34069 in memory (size: 40.0 KB, free: 4.8 GB)
17/04/19 10:43:53 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on hdfsnn:41444 in memory (size: 40.0 KB, free: 4.8 GB)
17/04/19 10:43:53 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on hdfsnn2:27641 in memory (size: 40.0 KB, free: 4.8 GB)
17/04/19 10:43:53 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on hdfsnn:29014 in memory (size: 40.0 KB, free: 4.8 GB)
17/04/19 10:43:53 INFO storage.BlockManagerInfo: Removed broadcast_3_piece0 on hdfsnn2:29251 in memory (size: 40.0 KB, free: 4.8 GB)
17/04/19 10:43:53 INFO spark.ContextCleaner: Cleaned accumulator 3

@rainforc I've snipped the logs from your post since they're very long... Anyhow, it looks like you're trying to run inference.  Were you able to train successfully with these settings?  Also, I'd try --conf spark.task.cpus=1.

@WhiteFangBuck #49 added a new "registration server" that runs on the Spark driver.  You should see a log statement identifying that server/port.  Can you ensure that address/port is valid/reachable from the executors?

@leewyang The Server address/port are not reachable from the executors. The driver is launched in a Docker container.
The address of it is resolved fine when I use socket.gethostbyname(socket.gethostname()) instead of socket.gethostname().
If latter, I get the following: 2017-04-21 18:58:14,749 INFO (MainThread-505) listening for reservations at ('oal76khoyvpxn43d', 34718)
If former then I get the local container address: 2017-04-21 19:08:09,028 INFO (MainThread-757) listening for reservations at ('<IP_ADDRESS>', 60205)

@WhiteFangBuck
We are experiencing the same here:
17/04/24 16:05:53 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
args: Namespace(cluster_size=4, epochs=0, format='tfr', images='mnist/tfr/test', labels=None, mode='inference', model='mnist_model', output='predictions', rdma=False, readers=1, steps=1000, tensorboard=False)
2017-04-24T16:05:53.270745 ===== Start
2017-04-24 16:05:53,270 INFO (MainThread-7114) Reserving TFSparkNodes
2017-04-24 16:05:53,279 INFO (MainThread-7114) listening for reservations at ('tools01.data.cblab.int', 59205)
2017-04-24 16:05:53,279 INFO (MainThread-7114) Starting TensorFlow on executors
2017-04-24 16:05:53,490 INFO (MainThread-7114) Waiting for TFSparkNodes to start
2017-04-24 16:05:53,490 INFO (MainThread-7114) waiting for 4 reservations
17/04/24 16:05:53 INFO SparkContext: Starting job: foreachPartition at /home/johan/exp/TensorFlowOnSpark/tfspark.zip/com/yahoo/ml/tf/TFCluster.py:262
17/04/24 16:05:53 INFO DAGScheduler: Got job 0 (foreachPartition at /home/johan/exp/TensorFlowOnSpark/tfspark.zip/com/yahoo/ml/tf/TFCluster.py:262) with 4 output partitions
17/04/24 16:05:53 INFO DAGScheduler: Final stage: ResultStage 0 (foreachPartition at /home/johan/exp/TensorFlowOnSpark/tfspark.zip/com/yahoo/ml/tf/TFCluster.py:262)
17/04/24 16:05:53 INFO DAGScheduler: Parents of final stage: List()
17/04/24 16:05:53 INFO DAGScheduler: Missing parents: List()
17/04/24 16:05:53 INFO DAGScheduler: Submitting ResultStage 0 (PythonRDD[1] at foreachPartition at /home/johan/exp/TensorFlowOnSpark/tfspark.zip/com/yahoo/ml/tf/TFCluster.py:262), which has no missing parents
17/04/24 16:05:53 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 10.4 KB, free 10.4 KB)
17/04/24 16:05:53 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 7.4 KB, free 17.8 KB)
17/04/24 16:05:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on <IP_ADDRESS>:33512 (size: 7.4 KB, free: 511.1 MB)
17/04/24 16:05:53 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1008
17/04/24 16:05:53 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 0 (PythonRDD[1] at foreachPartition at /home/johan/exp/TensorFlowOnSpark/tfspark.zip/com/yahoo/ml/tf/TFCluster.py:262)
17/04/24 16:05:53 INFO YarnScheduler: Adding task set 0.0 with 4 tasks
17/04/24 16:05:53 INFO YarnClientSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (data03.data.cblab.int:45796) with ID 1
17/04/24 16:05:53 INFO BlockManagerMasterEndpoint: Registering block manager data03.data.cblab.int:59598 with 19.2 GB RAM, BlockManagerId(1, data03.data.cblab.int, 59598)
17/04/24 16:05:53 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, data03.data.cblab.int, partition 0,PROCESS_LOCAL, 2083 bytes)
17/04/24 16:05:54 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on data03.data.cblab.int:59598 (size: 7.4 KB, free: 19.2 GB)
2017-04-24 16:05:54,491 INFO (MainThread-7114) waiting for 4 reservations
2017-04-24 16:05:55,492 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:05:56,494 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:05:57,494 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:05:58,496 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:05:59,497 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:00,498 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:01,500 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:02,501 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:03,502 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:04,504 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:05,505 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:06,507 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:07,507 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:08,508 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:09,509 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:10,511 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:11,512 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:12,514 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:13,514 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:14,515 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:15,517 INFO (MainThread-7114) waiting for 3 reservations
2017-04-24 16:06:16,518 INFO (MainThread-7114) waiting for 3 reservations

@leewyang  I solved the problem, by add the classpath  : --conf spark.executorEnv.CLASSPATH=$($HADOOP_HDFS_HOME/bin/hadoop classpath --glob):${CLASSPATH} \

Closing, since it combines several issues (libhdfs.so + hostname resolution).
If you're still experiencing issues, please create a new/separate issue with as much detail as possible.

I tried all the above steps of setting logdir to None, providing path to libhdfs.so and also recompiling python. Still I'm stuck with the same issue of the training getting stuck at task of feeding partition.

I'm running Spark 2.2.0 on Yarn, python 3.5.
I ran the application in cluster mode with two executors with 2 GB memory each and cores per executor as 1.
Application stderr:
`SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop_store/hdfs/tmp/nm-local-dir/usercache/hduser/filecache/54/__spark_libs__7104790403017096647.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/02/27 11:18:16 INFO util.SignalUtils: Registered signal handler for TERM
18/02/27 11:18:16 INFO util.SignalUtils: Registered signal handler for HUP
18/02/27 11:18:16 INFO util.SignalUtils: Registered signal handler for INT
18/02/27 11:18:17 INFO yarn.ApplicationMaster: Preparing Local resources
18/02/27 11:18:18 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1519724908414_0003_000001
18/02/27 11:18:18 INFO spark.SecurityManager: Changing view acls to: hduser
18/02/27 11:18:18 INFO spark.SecurityManager: Changing modify acls to: hduser
18/02/27 11:18:18 INFO spark.SecurityManager: Changing view acls groups to:
18/02/27 11:18:18 INFO spark.SecurityManager: Changing modify acls groups to:
18/02/27 11:18:18 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hduser); groups with view permissions: Set(); users  with modify permissions: Set(hduser); groups with modify permissions: Set()
18/02/27 11:18:18 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
18/02/27 11:18:18 INFO yarn.ApplicationMaster: Waiting for spark context initialization...
18/02/27 11:18:21 INFO spark.SparkContext: Running Spark version 2.2.0
18/02/27 11:18:21 INFO spark.SparkContext: Submitted application: mnist_spark
18/02/27 11:18:21 INFO spark.SecurityManager: Changing view acls to: hduser
18/02/27 11:18:21 INFO spark.SecurityManager: Changing modify acls to: hduser
18/02/27 11:18:21 INFO spark.SecurityManager: Changing view acls groups to:
18/02/27 11:18:21 INFO spark.SecurityManager: Changing modify acls groups to:
18/02/27 11:18:21 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hduser); groups with view permissions: Set(); users  with modify permissions: Set(hduser); groups with modify permissions: Set()
18/02/27 11:18:21 INFO util.Utils: Successfully started service 'sparkDriver' on port 46112.
18/02/27 11:18:21 INFO spark.SparkEnv: Registering MapOutputTracker
18/02/27 11:18:21 INFO spark.SparkEnv: Registering BlockManagerMaster
18/02/27 11:18:21 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/02/27 11:18:21 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/02/27 11:18:21 INFO storage.DiskBlockManager: Created local directory at /usr/local/hadoop_store/hdfs/tmp/nm-local-dir/usercache/hduser/appcache/application_1519724908414_0003/blockmgr-1eb88fde-1edb-43c8-8580-e266606513e0
18/02/27 11:18:21 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
18/02/27 11:18:21 INFO spark.SparkEnv: Registering OutputCommitCoordinator
18/02/27 11:18:22 INFO util.log: Logging initialized @6600ms
18/02/27 11:18:22 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
18/02/27 11:18:22 INFO server.Server: jetty-9.3.z-SNAPSHOT
18/02/27 11:18:22 INFO server.Server: Started @6849ms
18/02/27 11:18:22 INFO server.AbstractConnector: Started ServerConnector@4a69e7cc{HTTP/1.1,[http/1.1]}{<IP_ADDRESS>:35715}
18/02/27 11:18:22 INFO util.Utils: Successfully started service 'SparkUI' on port 35715.
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4f9f1ffd{/jobs,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@bcfc63d{/jobs/json,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@585f6a93{/jobs/job,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6d5cade1{/jobs/job/json,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4598bb0d{/stages,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6de59a33{/stages/json,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@174a84d{/stages/stage,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5197acdb{/stages/stage/json,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@39af6375{/stages/pool,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@acfab37{/stages/pool/json,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@66c99a0c{/storage,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@669ff33e{/storage/json,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@17c1f4d9{/storage/rdd,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6c63ee4a{/storage/rdd/json,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@59c74e9d{/environment,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3865844c{/environment/json,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5a66b6e7{/executors,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3c56e8b9{/executors/json,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5ff5c3cd{/executors/threadDump,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@569ac35d{/executors/threadDump/json,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4bf2626a{/static,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3196db98{/,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@20bf2d51{/api,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@33b62897{/jobs/job/kill,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@980b282{/stages/stage/kill,null,AVAILABLE,@Spark}
18/02/27 11:18:22 INFO ui.SparkUI: Bound SparkUI to <IP_ADDRESS>, and started at http://<IP_ADDRESS>:35715
18/02/27 11:18:22 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
18/02/27 11:18:22 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1519724908414_0003 and attemptId Some(appattempt_1519724908414_0003_000001)
18/02/27 11:18:22 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33485.
18/02/27 11:18:22 INFO netty.NettyBlockTransferService: Server created on <IP_ADDRESS>:33485
18/02/27 11:18:22 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/02/27 11:18:22 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, <IP_ADDRESS>, 33485, None)
18/02/27 11:18:22 INFO storage.BlockManagerMasterEndpoint: Registering block manager <IP_ADDRESS>:33485 with 366.3 MB RAM, BlockManagerId(driver, <IP_ADDRESS>, 33485, None)
18/02/27 11:18:22 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, <IP_ADDRESS>, 33485, None)
18/02/27 11:18:22 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, <IP_ADDRESS>, 33485, None)
18/02/27 11:18:23 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6e9a85d4{/metrics/json,null,AVAILABLE,@Spark}
18/02/27 11:18:23 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@<IP_ADDRESS>:46112)
18/02/27 11:18:23 INFO yarn.ApplicationMaster:
YARN executor launch context:
env:
CLASSPATH -> {{PWD}}{{PWD}}/spark_conf{{PWD}}/spark_libs/$HADOOP_CONF_DIR$HADOOP_COMMON_HOME/share/hadoop/common/$HADOOP_COMMON_HOME/share/hadoop/common/lib/$HADOOP_HDFS_HOME/share/hadoop/hdfs/$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/$HADOOP_YARN_HOME/share/hadoop/yarn/$HADOOP_YARN_HOME/share/hadoop/yarn/lib/$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*
SPARK_YARN_STAGING_DIR -> hdfs://<IP_ADDRESS>/user/hduser/.sparkStaging/application_1519724908414_0003
SPARK_USER -> hduser
SPARK_YARN_MODE -> true
PYTHONPATH -> /home/hduser/.virtualenvs/sparkenv/bin/python:/opt/spark-2.2.0/python/lib/pyspark.zip:/opt/spark-2.2.0/python/lib/py4j-0.10.4-src.zip{{PWD}}/pyfiles{{PWD}}/pyspark.zip{{PWD}}/py4j-0.10.4-src.zip{{PWD}}/tfspark.zip{{PWD}}/com.datastax.spark_spark-cassandra-connector_2.11-2.0.1.jar{{PWD}}/org.joda_joda-convert-1.2.jar{{PWD}}/commons-beanutils_commons-beanutils-1.9.3.jar{{PWD}}/io.netty_netty-all-4.0.33.Final.jar{{PWD}}/joda-time_joda-time-2.3.jar{{PWD}}/com.twitter_jsr166e-1.1.0.jar{{PWD}}/org.scala-lang_scala-reflect-2.11.8.jar{{PWD}}/commons-collections_commons-collections-3.2.2.jar
LD_LIBRARY_PATH -> /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server:/usr/local/hadoop/lib/native
command:
{{JAVA_HOME}}/bin/java \
-server \
-Xmx2048m \
-Djava.io.tmpdir={{PWD}}/tmp \
'-Dspark.cassandra.connection.port=9042' \
-Dspark.yarn.app.container.log.dir=<LOG_DIR> \
-XX:OnOutOfMemoryError='kill %p' \
org.apache.spark.executor.CoarseGrainedExecutorBackend \
--driver-url \
spark://CoarseGrainedScheduler@<IP_ADDRESS>:46112 \
--executor-id \
 \
--hostname \
 \
--cores \
1 \
--app-id \
application_1519724908414_0003 \
--user-class-path \
file:$PWD/app.jar \
--user-class-path \
file:$PWD/com.datastax.spark_spark-cassandra-connector_2.11-2.0.1.jar \
--user-class-path \
file:$PWD/org.joda_joda-convert-1.2.jar \
--user-class-path \
file:$PWD/commons-beanutils_commons-beanutils-1.9.3.jar \
--user-class-path \
file:$PWD/io.netty_netty-all-4.0.33.Final.jar \
--user-class-path \
file:$PWD/joda-time_joda-time-2.3.jar \
--user-class-path \
file:$PWD/com.twitter_jsr166e-1.1.0.jar \
--user-class-path \
file:$PWD/org.scala-lang_scala-reflect-2.11.8.jar \
--user-class-path \
file:$PWD/commons-collections_commons-collections-3.2.2.jar \
1><LOG_DIR>/stdout \
2><LOG_DIR>/stderr
resources:
joda-time_joda-time-2.3.jar -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/joda-time_joda-time-2.3.jar" } size: 581571 timestamp:<PHONE_NUMBER>149 type: FILE visibility: PRIVATE
py4j-0.10.4-src.zip -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/py4j-0.10.4-src.zip" } size: 74096 timestamp:<PHONE_NUMBER>676 type: FILE visibility: PRIVATE
com.datastax.spark_spark-cassandra-connector_2.11-2.0.1.jar -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/com.datastax.spark_spark-cassandra-connector_2.11-2.0.1.jar" } size: 8492963 timestamp:<PHONE_NUMBER>073 type: FILE visibility: PRIVATE
spark_conf -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/spark_conf.zip" } size: 98605 timestamp:<PHONE_NUMBER>748 type: ARCHIVE visibility: PRIVATE
commons-collections_commons-collections-3.2.2.jar -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/commons-collections_commons-collections-3.2.2.jar" } size: 588337 timestamp:<PHONE_NUMBER>625 type: FILE visibility: PRIVATE
io.netty_netty-all-4.0.33.Final.jar -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/io.netty_netty-all-4.0.33.Final.jar" } size: 2112017 timestamp:<PHONE_NUMBER>131 type: FILE visibility: PRIVATE
com.twitter_jsr166e-1.1.0.jar -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/com.twitter_jsr166e-1.1.0.jar" } size: 62226 timestamp:<PHONE_NUMBER>563 type: FILE visibility: PRIVATE
tfspark.zip -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/tfspark.zip" } size: 31565 timestamp:<PHONE_NUMBER>707 type: FILE visibility: PRIVATE
pyspark.zip -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/pyspark.zip" } size: 480115 timestamp:<PHONE_NUMBER>658 type: FILE visibility: PRIVATE
org.joda_joda-convert-1.2.jar -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/org.joda_joda-convert-1.2.jar" } size: 38460 timestamp:<PHONE_NUMBER>091 type: FILE visibility: PRIVATE
pyfiles/mnist_dist.py -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/mnist_dist.py" } size: 6409 timestamp:<PHONE_NUMBER>720 type: FILE visibility: PRIVATE
spark_libs -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/__spark_libs__7104790403017096647.zip" } size: 208179347 timestamp:<PHONE_NUMBER>960 type: ARCHIVE visibility: PRIVATE
org.scala-lang_scala-reflect-2.11.8.jar -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/org.scala-lang_scala-reflect-2.11.8.jar" } size: 4573750 timestamp:<PHONE_NUMBER>594 type: FILE visibility: PRIVATE
commons-beanutils_commons-beanutils-1.9.3.jar -> resource { scheme: "hdfs" host: "<IP_ADDRESS>" port: -1 file: "/user/hduser/.sparkStaging/application_1519724908414_0003/commons-beanutils_commons-beanutils-1.9.3.jar" } size: 246174 timestamp:<PHONE_NUMBER>108 type: FILE visibility: PRIVATE
===============================================================================
18/02/27 11:18:23 INFO client.RMProxy: Connecting to ResourceManager at /<IP_ADDRESS>:8030
18/02/27 11:18:23 INFO yarn.YarnRMClient: Registering the ApplicationMaster
18/02/27 11:18:23 INFO yarn.YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 2432 MB memory (including 384 MB of overhead)
18/02/27 11:18:23 INFO yarn.YarnAllocator: Submitted 2 unlocalized container requests.
18/02/27 11:18:23 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
18/02/27 11:18:24 INFO impl.AMRMClientImpl: Received new token for : ip-172-20-6-194.us-east-2.compute.internal:38444
18/02/27 11:18:24 INFO yarn.YarnAllocator: Launching container container_1519724908414_0003_01_000002 on host ip-172-20-6-194.us-east-2.compute.internal for executor with ID 1
18/02/27 11:18:24 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
18/02/27 11:18:24 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
18/02/27 11:18:24 INFO impl.ContainerManagementProtocolProxy: Opening proxy : ip-172-20-6-194.us-east-2.compute.internal:38444
18/02/27 11:18:25 INFO yarn.YarnAllocator: Launching container container_1519724908414_0003_01_000003 on host ip-172-20-6-194.us-east-2.compute.internal for executor with ID 2
18/02/27 11:18:25 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
18/02/27 11:18:25 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
18/02/27 11:18:25 INFO impl.ContainerManagementProtocolProxy: Opening proxy : ip-172-20-6-194.us-east-2.compute.internal:38444
18/02/27 11:18:28 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (<IP_ADDRESS>:42568) with ID 1
18/02/27 11:18:28 INFO storage.BlockManagerMasterEndpoint: Registering block manager ip-172-20-6-194.us-east-2.compute.internal:46180 with 912.3 MB RAM, BlockManagerId(1, ip-172-20-6-194.us-east-2.compute.internal, 46180, None)
18/02/27 11:18:29 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (<IP_ADDRESS>:42572) with ID 2
18/02/27 11:18:29 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
18/02/27 11:18:29 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
18/02/27 11:18:29 INFO storage.BlockManagerMasterEndpoint: Registering block manager ip-172-20-6-194.us-east-2.compute.internal:43305 with 912.3 MB RAM, BlockManagerId(2, ip-172-20-6-194.us-east-2.compute.internal, 43305, None)
18/02/27 11:18:29 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 278.4 KB, free 366.0 MB)
18/02/27 11:18:29 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.3 KB, free 366.0 MB)
18/02/27 11:18:29 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on <IP_ADDRESS>:33485 (size: 23.3 KB, free: 366.3 MB)
18/02/27 11:18:29 INFO spark.SparkContext: Created broadcast 0 from textFile at NativeMethodAccessorImpl.java:0
18/02/27 11:18:30 INFO memory.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 278.4 KB, free 365.7 MB)
18/02/27 11:18:30 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 23.3 KB, free 365.7 MB)
18/02/27 11:18:30 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on <IP_ADDRESS>:33485 (size: 23.3 KB, free: 366.3 MB)
18/02/27 11:18:30 INFO spark.SparkContext: Created broadcast 1 from textFile at NativeMethodAccessorImpl.java:0
18/02/27 11:18:30 INFO mapred.FileInputFormat: Total input paths to process : 10
18/02/27 11:18:30 INFO mapred.FileInputFormat: Total input paths to process : 10
18/02/27 11:18:30 INFO spark.SparkContext: Starting job: foreachPartition at /usr/local/hadoop_store/hdfs/tmp/nm-local-dir/usercache/hduser/appcache/application_1519724908414_0003/container_1519724908414_0003_01_000001/tfspark.zip/tensorflowonspark/TFCluster.py:271
18/02/27 11:18:30 INFO scheduler.DAGScheduler: Got job 0 (foreachPartition at /usr/local/hadoop_store/hdfs/tmp/nm-local-dir/usercache/hduser/appcache/application_1519724908414_0003/container_1519724908414_0003_01_000001/tfspark.zip/tensorflowonspark/TFCluster.py:271) with 2 output partitions
18/02/27 11:18:30 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (foreachPartition at /usr/local/hadoop_store/hdfs/tmp/nm-local-dir/usercache/hduser/appcache/application_1519724908414_0003/container_1519724908414_0003_01_000001/tfspark.zip/tensorflowonspark/TFCluster.py:271)
18/02/27 11:18:30 INFO scheduler.DAGScheduler: Parents of final stage: List()
18/02/27 11:18:30 INFO scheduler.DAGScheduler: Missing parents: List()
18/02/27 11:18:30 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (PythonRDD[8] at foreachPartition at /usr/local/hadoop_store/hdfs/tmp/nm-local-dir/usercache/hduser/appcache/application_1519724908414_0003/container_1519724908414_0003_01_000001/tfspark.zip/tensorflowonspark/TFCluster.py:271), which has no missing parents
18/02/27 11:18:30 INFO memory.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 14.2 KB, free 365.7 MB)
18/02/27 11:18:30 INFO memory.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.6 KB, free 365.7 MB)
18/02/27 11:18:30 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on <IP_ADDRESS>:33485 (size: 9.6 KB, free: 366.2 MB)
18/02/27 11:18:30 INFO spark.SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1006
18/02/27 11:18:30 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (PythonRDD[8] at foreachPartition at /usr/local/hadoop_store/hdfs/tmp/nm-local-dir/usercache/hduser/appcache/application_1519724908414_0003/container_1519724908414_0003_01_000001/tfspark.zip/tensorflowonspark/TFCluster.py:271) (first 15 tasks are for partitions Vector(0, 1))
18/02/27 11:18:30 INFO cluster.YarnClusterScheduler: Adding task set 0.0 with 2 tasks
18/02/27 11:18:30 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, ip-172-20-6-194.us-east-2.compute.internal, executor 1, partition 0, PROCESS_LOCAL, 4800 bytes)
18/02/27 11:18:30 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, ip-172-20-6-194.us-east-2.compute.internal, executor 2, partition 1, PROCESS_LOCAL, 4800 bytes)
18/02/27 11:18:30 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-172-20-6-194.us-east-2.compute.internal:43305 (size: 9.6 KB, free: 912.3 MB)
18/02/27 11:18:30 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on ip-172-20-6-194.us-east-2.compute.internal:46180 (size: 9.6 KB, free: 912.3 MB)
18/02/27 11:18:33 INFO spark.SparkContext: Starting job: collect at PythonRDD.scala:458
18/02/27 11:18:33 INFO scheduler.DAGScheduler: Got job 1 (collect at PythonRDD.scala:458) with 10 output partitions
18/02/27 11:18:33 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (collect at PythonRDD.scala:458)
18/02/27 11:18:33 INFO scheduler.DAGScheduler: Parents of final stage: List()
18/02/27 11:18:33 INFO scheduler.DAGScheduler: Missing parents: List()
18/02/27 11:18:33 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (PythonRDD[10] at RDD at PythonRDD.scala:48), which has no missing parents
18/02/27 11:18:33 INFO memory.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 17.3 KB, free 365.7 MB)
18/02/27 11:18:33 INFO memory.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 8.0 KB, free 365.7 MB)
18/02/27 11:18:33 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on <IP_ADDRESS>:33485 (size: 8.0 KB, free: 366.2 MB)
18/02/27 11:18:33 INFO spark.SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:1006
18/02/27 11:18:33 INFO scheduler.DAGScheduler: Submitting 10 missing tasks from ResultStage 1 (PythonRDD[10] at RDD at PythonRDD.scala:48) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9))
18/02/27 11:18:33 INFO cluster.YarnClusterScheduler: Adding task set 1.0 with 10 tasks
18/02/27 11:18:34 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, ip-172-20-6-194.us-east-2.compute.internal, executor 2, partition 0, NODE_LOCAL, 5465 bytes)
18/02/27 11:18:34 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 3896 ms on ip-172-20-6-194.us-east-2.compute.internal (executor 2) (1/2)
18/02/27 11:18:34 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on ip-172-20-6-194.us-east-2.compute.internal:43305 (size: 8.0 KB, free: 912.3 MB)
18/02/27 11:18:34 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on ip-172-20-6-194.us-east-2.compute.internal:43305 (size: 23.3 KB, free: 912.3 MB)
18/02/27 11:18:36 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on ip-172-20-6-194.us-east-2.compute.internal:43305 (size: 23.3 KB, free: 912.2 MB)`
Application stdout:
args: Namespace(batch_size=100, cluster_size=2, epochs=1, format='csv', images='mnist/csv/train/images', labels='mnist/csv/train/labels', mode='train', model='mnist_model', output='predictions', rdma=False, readers=1, steps=1000, tensorboard=False) 2018-02-27T11:18:29.401564 ===== Start zipping images and labels 2018-02-27 11:18:30,284 INFO (MainThread-8426) Reserving TFSparkNodes  2018-02-27 11:18:30,284 INFO (MainThread-8426) worker node range range(1, 2), ps node range range(0, 1) 2018-02-27 11:18:30,286 INFO (MainThread-8426) listening for reservations at ('<IP_ADDRESS>', 43222) 2018-02-27 11:18:30,291 INFO (MainThread-8426) Starting TensorFlow on executors 2018-02-27 11:18:30,302 INFO (MainThread-8426) Waiting for TFSparkNodes to start 2018-02-27 11:18:30,302 INFO (MainThread-8426) waiting for 2 reservations 2018-02-27 11:18:31,303 INFO (MainThread-8426) waiting for 2 reservations 2018-02-27 11:18:32,304 INFO (MainThread-8426) waiting for 1 reservations 2018-02-27 11:18:33,305 INFO (MainThread-8426) all reservations completed 2018-02-27 11:18:33,306 INFO (MainThread-8426) All TFSparkNodes started 2018-02-27 11:18:33,306 INFO (MainThread-8426) {'host': '<IP_ADDRESS>', 'ppid': 8645, 'addr': '/tmp/pymp-_wm5_j2q/listener-jvy2z75u', 'worker_num': 1, 'task_index': 0, 'job_name': 'worker', 'authkey': b'\x1eB\xf11\x853H\xfa\xa0\x04d\xf3\xe6\xa0u8', 'tb_pid': 0, 'tb_port': 0, 'port': 42775} 2018-02-27 11:18:33,306 INFO (MainThread-8426) {'host': '<IP_ADDRESS>', 'ppid': 8647, 'addr': ('<IP_ADDRESS>', 35332), 'worker_num': 0, 'task_index': 0, 'job_name': 'ps', 'authkey': b'\x0bK\xe8H\xbc\x94M\x1b\x95F\x14T\xbe\x89\x9f\xbb', 'tb_pid': 0, 'tb_port': 0, 'port': 44200} 2018-02-27 11:18:33,306 INFO (MainThread-8426) Feeding training data
Worker stderr:
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hadoop_store/hdfs/tmp/nm-local-dir/usercache/hduser/filecache/54/__spark_libs__7104790403017096647.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/02/27 11:18:26 INFO executor.CoarseGrainedExecutorBackend: Started daemon with process name: 8540@ip-172-20-6-194 18/02/27 11:18:26 INFO util.SignalUtils: Registered signal handler for TERM 18/02/27 11:18:26 INFO util.SignalUtils: Registered signal handler for HUP 18/02/27 11:18:26 INFO util.SignalUtils: Registered signal handler for INT 18/02/27 11:18:28 INFO spark.SecurityManager: Changing view acls to: hduser 18/02/27 11:18:28 INFO spark.SecurityManager: Changing modify acls to: hduser 18/02/27 11:18:28 INFO spark.SecurityManager: Changing view acls groups to:  18/02/27 11:18:28 INFO spark.SecurityManager: Changing modify acls groups to:  18/02/27 11:18:28 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hduser); groups with view permissions: Set(); users  with modify permissions: Set(hduser); groups with modify permissions: Set() 18/02/27 11:18:28 INFO client.TransportClientFactory: Successfully created connection to /<IP_ADDRESS>:46112 after 65 ms (0 ms spent in bootstraps) 18/02/27 11:18:28 INFO spark.SecurityManager: Changing view acls to: hduser 18/02/27 11:18:28 INFO spark.SecurityManager: Changing modify acls to: hduser 18/02/27 11:18:28 INFO spark.SecurityManager: Changing view acls groups to:  18/02/27 11:18:28 INFO spark.SecurityManager: Changing modify acls groups to:  18/02/27 11:18:28 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hduser); groups with view permissions: Set(); users  with modify permissions: Set(hduser); groups with modify permissions: Set() 18/02/27 11:18:28 INFO client.TransportClientFactory: Successfully created connection to /<IP_ADDRESS>:46112 after 2 ms (0 ms spent in bootstraps) 18/02/27 11:18:28 INFO storage.DiskBlockManager: Created local directory at /usr/local/hadoop_store/hdfs/tmp/nm-local-dir/usercache/hduser/appcache/application_1519724908414_0003/blockmgr-0215cf5c-5cdd-46be-9a91-1fdae24ca364 18/02/27 11:18:28 INFO memory.MemoryStore: MemoryStore started with capacity 912.3 MB 18/02/27 11:18:29 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@<IP_ADDRESS>:46112 18/02/27 11:18:29 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver 18/02/27 11:18:29 INFO executor.Executor: Starting executor ID 2 on host ip-172-20-6-194.us-east-2.compute.internal 18/02/27 11:18:29 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 43305. 18/02/27 11:18:29 INFO netty.NettyBlockTransferService: Server created on ip-172-20-6-194.us-east-2.compute.internal:43305 18/02/27 11:18:29 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 18/02/27 11:18:29 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(2, ip-172-20-6-194.us-east-2.compute.internal, 43305, None) 18/02/27 11:18:29 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(2, ip-172-20-6-194.us-east-2.compute.internal, 43305, None) 18/02/27 11:18:29 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(2, ip-172-20-6-194.us-east-2.compute.internal, 43305, None) 18/02/27 11:18:30 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 1 18/02/27 11:18:30 INFO executor.Executor: Running task 1.0 in stage 0.0 (TID 1) 18/02/27 11:18:30 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 2 18/02/27 11:18:30 INFO client.TransportClientFactory: Successfully created connection to /<IP_ADDRESS>:33485 after 2 ms (0 ms spent in bootstraps) 18/02/27 11:18:30 INFO memory.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 9.6 KB, free 912.3 MB) 18/02/27 11:18:30 INFO broadcast.TorrentBroadcast: Reading broadcast variable 2 took 171 ms 18/02/27 11:18:31 INFO memory.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 14.2 KB, free 912.3 MB) 2018-02-27 11:18:32,287 INFO (MainThread-8664) connected to server at ('<IP_ADDRESS>', 43222) 2018-02-27 11:18:32,288 INFO (MainThread-8664) TFSparkNode.reserve: {'job_name': 'worker', 'addr': '/tmp/pymp-_wm5_j2q/listener-jvy2z75u', 'tb_pid': 0, 'worker_num': 1, 'task_index': 0, 'host': '<IP_ADDRESS>', 'authkey': b'\x1eB\xf11\x853H\xfa\xa0\x04d\xf3\xe6\xa0u8', 'port': 42775, 'tb_port': 0, 'ppid': 8645} 2018-02-27 11:18:34,291 INFO (MainThread-8664) node: {'job_name': 'ps', 'addr': ('<IP_ADDRESS>', 35332), 'worker_num': 0, 'task_index': 0, 'host': '<IP_ADDRESS>', 'port': 44200, 'authkey': b'\x0bK\xe8H\xbc\x94M\x1b\x95F\x14T\xbe\x89\x9f\xbb', 'tb_pid': 0, 'tb_port': 0, 'ppid': 8647} 2018-02-27 11:18:34,291 INFO (MainThread-8664) node: {'job_name': 'worker', 'addr': '/tmp/pymp-_wm5_j2q/listener-jvy2z75u', 'worker_num': 1, 'task_index': 0, 'host': '<IP_ADDRESS>', 'port': 42775, 'authkey': b'\x1eB\xf11\x853H\xfa\xa0\x04d\xf3\xe6\xa0u8', 'tb_pid': 0, 'tb_port': 0, 'ppid': 8645} 2018-02-27 11:18:34,296 INFO (MainThread-8664) Starting TensorFlow worker:0 as worker on cluster node 1 on background process 18/02/27 11:18:34 INFO python.PythonRunner: Times: total = 3203, boot = 751, init = 130, finish = 2322 18/02/27 11:18:34 INFO executor.Executor: Finished task 1.0 in stage 0.0 (TID 1). 1757 bytes result sent to driver 18/02/27 11:18:34 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 2 18/02/27 11:18:34 INFO executor.Executor: Running task 0.0 in stage 1.0 (TID 2) 18/02/27 11:18:34 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 3 18/02/27 11:18:34 INFO memory.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 8.0 KB, free 912.3 MB) 18/02/27 11:18:34 INFO broadcast.TorrentBroadcast: Reading broadcast variable 3 took 19 ms 18/02/27 11:18:34 INFO memory.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 17.3 KB, free 912.3 MB) 18/02/27 11:18:34 INFO rdd.HadoopRDD: Input split: hdfs://<IP_ADDRESS>/user/hduser/mnist/csv/train/images/part-00000:0+9338236 18/02/27 11:18:34 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 0 18/02/27 11:18:34 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.3 KB, free 912.2 MB) 18/02/27 11:18:34 INFO broadcast.TorrentBroadcast: Reading broadcast variable 0 took 23 ms 18/02/27 11:18:34 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 328.3 KB, free 911.9 MB) 2018-02-27 11:18:36,231 INFO (MainThread-8761) 1: ======== worker:0 ======== 2018-02-27 11:18:36,231 INFO (MainThread-8761) 1: Cluster spec: {'ps': ['<IP_ADDRESS>:44200'], 'worker': ['<IP_ADDRESS>:42775']} 2018-02-27 11:18:36,231 INFO (MainThread-8761) 1: Using CPU 2018-02-27 11:18:36.232036: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA 2018-02-27 11:18:36.233759: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> <IP_ADDRESS>:44200} 2018-02-27 11:18:36.233815: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> localhost:42775} 2018-02-27 11:18:36.234268: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:324] Started server with target: grpc://localhost:42775 tensorflow model path: hdfs://<IP_ADDRESS>/user/hduser/mnist_model 18/02/27 11:18:36 INFO rdd.HadoopRDD: Input split: hdfs://<IP_ADDRESS>/user/hduser/mnist/csv/train/labels/part-00000:0+204800 18/02/27 11:18:36 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 1 18/02/27 11:18:36 INFO memory.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 23.3 KB, free 911.9 MB) 18/02/27 11:18:36 INFO broadcast.TorrentBroadcast: Reading broadcast variable 1 took 39 ms 18/02/27 11:18:36 INFO memory.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 328.3 KB, free 911.6 MB) 2018-02-27 11:18:37,142 INFO (MainThread-8796) Connected to TFSparkNode.mgr on <IP_ADDRESS>, ppid=8645, state='running' 2018-02-27 11:18:37,219 INFO (MainThread-8796) mgr.state='running' 2018-02-27 11:18:37,220 INFO (MainThread-8796) Feeding partition <itertools.chain object at 0x7fe882ae65c0> into input queue <multiprocessing.queues.JoinableQueue object at 0x7fe882ae6eb8> 18/02/27 11:18:39 INFO python.PythonRunner: Times: total = 2670, boot = 22, init = 147, finish = 2501 18/02/27 11:18:39 INFO python.PythonRunner: Times: total = 201, boot = 30, init = 77, finish = 94
Let me know if anything else is needed.

Have you tried using: hdfs://default/user/hduser/mnist_model?
Normally, you should see this log statement in your worker.  The fact that you didn't usually implies a "path to libhdfs.so" issue (or the related issue where TF doesn't understand your HDFS URL).  I'm going to assume that this path was never written to... (which is another symptom).
So:

try the path with the "default" keyword (which is a TF-specific alias to the namenode).
ensure that libhdfs.so exists in the same path on all your nodes, and make sure you are passing that path into spark as part of your --conf spark.executorEnv. LD_LIBRARY_PATH.

I tried using the path you've mentioned and its still stuck at the same place.
Only other thing I can think of... given that it looks like you have access to the hadoop nodes, try to run a simple python shell on one of those nodes that reads a file from HDFS, e.g. something like:
# python
> import tensorflow as tf
> f = tf.gfile.GFile("hdfs://default/user/hduser/foo.txt")
> txt = f.read()

If that doesn't work, then perhaps there's something missing in your TF setup.  If it does work, then I'd try to run the same code in your map_fun of your TFoS job.

@leewyang
I tried, with errors like this:
>>> txt = f.read() hdfsBuilderConnect(forceNewInstance=0, nn=<IP_ADDRESS>:9000, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error: (unable to get stack trace for java.lang.NoClassDefFoundError exception: ExceptionUtils::getStackTrace error.) Traceback (most recent call last):   File "<stdin>", line 1, in <module>   File "/home/master/anaconda3/lib/python3.5/site-packages/tensorflow/python/lib/io/file_io.py", line 119, in read     self._preread_check()   File "/home/master/anaconda3/lib/python3.5/site-packages/tensorflow/python/lib/io/file_io.py", line 79, in _preread_check     compat.as_bytes(self.__name), 1024 * 512, status)   File "/home/master/anaconda3/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__     c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.NotFoundError: Unknown error 255
could you know what should i do now?

@yuezhilanyi did you provide the path to libhdfs.so in your command line?

@leewyang

How to resolve it ?

@chapzq77 please file a new git issue w/ more details (since that screenshot doesn't provide enough information to help you).

spark2-submit
--master yarn
--deploy-mode cluster
--queue default
--num-executors 100
--executor-memory 8G
--executor-cores 1
--py-files TensorFlowOnSpark/tfspark.zip,TensorFlowOnSpark/examples/mnist/spark/mnist_dist.py
--conf spark.dynamicAllocation.enabled=false
--conf spark.yarn.maxAppAttempts=1
--archives hdfs:///user/${user}/anaconda3.zip#anaconda3
--conf spark.executorEnv.LD_LIBRARY_PATH="$JAVA_HOME/jre/lib/amd64/server:/opt/cloudera/parcels/CDH/lib64"
TensorFlowOnSpark/examples/mnist/spark/mnist_spark.py
--images mnist/csv/train/images
--labels mnist/csv/train/labels
--mode train
--model hdfs:///user/${user}/mnist_model
export PYTHON_ROOT=./anaconda3
export PYSPARK_PYTHON=${PYTHON_ROOT}/bin/python
export SPARK_YARN_USER_ENV="PYSPARK_PYTHON=anaconda3/bin/python"
export PATH=${PYTHON_ROOT}/bin/:$PATH
export JAVA_HOME=/usr/java/jdk1.8.0_192
tensorflow==1.10
tensorflowonspark==1.4.4
python==3.6.5
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/worker.py", line 377, in main
process()
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/worker.py", line 372, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/rdd.py", line 2499, in pipeline_func
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/rdd.py", line 2499, in pipeline_func
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/rdd.py", line 2499, in pipeline_func
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/rdd.py", line 352, in func
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/rdd.py", line 801, in func
File "/data9/yarn/nm/usercache/11085386/appcache/application_1569730221745_97713/container_e07_1569730221745_97713_01_000001/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 417, in _train
Exception: Timeout while feeding partition

If set logdir =None, it can run and the model is empty! But if not, the error is following:
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/worker.py", line 377, in main
process()
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/worker.py", line 372, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/rdd.py", line 2499, in pipeline_func
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/rdd.py", line 2499, in pipeline_func
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/rdd.py", line 2499, in pipeline_func
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/rdd.py", line 352, in func
File "/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/lib/spark2/python/lib/pyspark.zip/pyspark/rdd.py", line 801, in func
File "/data9/yarn/nm/usercache/11085386/appcache/application_1569730221745_97713/container_e07_1569730221745_97713_01_000001/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 417, in _train
Exception: Timeout while feeding partition

That symptom is usually means that spark.executorEnv.LD_LIBRARY_PATH needs to be pointing to the directory that contains libhdfs.so.  Make sure that this file exists on ALL of your executors at that same path.  If that still doesn't work, try going down to a single executor to see how that behaves.


Ionic 2 error, "Cannot find module dgram"

In an Ionic 2 TypeScript project, I've installed the bonjour npm module, then imported it like this:
import bonjour from 'bonjour';
That doesn't throw errors. Bt as soon as I refer to bonjour, I get the runtime error
Cannot find module "dgram"
Has anyone successfully used this plugin with Ionic 2?

I don't think this is Ionic related. I'm getting the same error with a simple NodeJS project.

I'm getting the same error on<EMAIL_ADDRESS>using
import Bonjour from 'bonjour';
and then
Bonjour.find({ type: 'http' }, function (service) {
        console.log('Found an HTTP server:', service)
      })

Found an alternative https://ionicframework.com/docs/native/zeroconf/, I would have preferred this "bonjour" module but I see no other alternative to go to Zeroconf.

The module dgram is not available in client side Ionic code. Ionic runs on cordova and not on the node js system which means that this package cannot be used in Ionic for the moment. The Zeroconf Ionic plugin is a great alternative, though it is only suited for ios and android.
When programming with the zeroconf plugin, you can use safari to debug when you run on an ios device. Find the 'developer' menu in settings and you can see your device listed there.
When you can't run on a device but in a browser, look up 'mock' to 'imitate' a service being added so you can develop the things you need.

Hi,
I went into the dgram folder and it is pretty empty (just a readme) :

This package name is not currently in use, but was formerly occupied by another package. To avoid malicious use, npm is hanging on to the package name, but loosely, and we'll probably give it to you if you want it.
You may adopt this package by contacting<EMAIL_ADDRESS>and requesting the name.

So as suggested I wrote to npm, and they answered :

Thanks for reaching out!
The dgram package on npm was empty and didn't do anything, however many packages mistakenly depended on it. npm, Inc. has taken ownership of it.
It's also a built-in Node module. If you've depended on dgram, you can safely remove it from your package dependencies.
You can learn more about dgram in Node here: https://nodejs.org/api/dgram.html

And so I went to the web site suggested but it doesn't seems it was empty actually....
Whatever, I tried to comment out things with dgram ( in multicast-dns package ) or tweek it but it seems it was useful at some points.
 I'll go to multicast-dns see what may be done.


TypeScript SDK causes errors when user browses in Incognito mode

Code examples:
import { init } from '@amplitude/analytics-browser'
init(process.env.AMPLITUDE_TOKEN, null, {
  serverZone: ServerZone.EU,
})

import {
  setOptOut,
  Identify,
  identify,
  setUserId,
} from '@amplitude/analytics-browser'

if (isUserInternal(ua)) {
// Opt-out internal users from Amplitude tracking
  setOptOut(true)
} else {
  const userIdentify = new Identify()
  setUserId(ua?.id)
  userIdentify.set('userName', ua?.name)
  userIdentify.set('userEmail', ua?.username)
  identify(userIdentify)
}

Expected Behavior
I'm aware that Amplitude & other analytics tools can be blocked in the browser's Incognito mode and by script blocking tools. However, I would expect Amplitude to not track the user in this case, not to throw errors and crash my application.
Current Behavior
In browser incognito mode, Amplitude is probably blocked. init method seems to work ok (doesn't throw any errors), but then when executing setOptOut or setUserId methods, user gets the following errors:
TypeError: Cannot set properties of undefined (setting 'optOut')
at AmplitudeBrowser.setOptOut

TypeError: Cannot set properties of undefined (setting 'userId')
at AmplitudeBrowser.setUserId

Possible Solution
This problem doesn't exist in JavaScript SDK. I needed to switch back to using JavaScript SDK (which is marked as legacy now) to solve this problem.
I suppose this problem could be caused by a lack of instance object in TypeScript SDK. There is no amplitude object to check if it exists and no getInstance method. Maybe merging this pull request could be helpful:
https://github.com/amplitude/Amplitude-TypeScript/pull/106
Maybe a method to check if Amplitude is initialized could solve this problem: https://github.com/amplitude/Amplitude-TypeScript/issues/169
Steps to Reproduce

Initialize Amplitude TypeScript SDK
Use setOptOut or setUserId methods
Run application in browser incognito mode

Environment

TypeScript SDK Version: "@amplitude/analytics-browser": "^1.0.3"
Installation Method: Yarn
Browser and Version: Chrome Version 103.0.5060.134 (Incognito mode)

@kevinpagtakhan , are you working on this issue? If not, I will take a look. Thanks!


Feature Request: Presets

Hi there,
Is it possible to create a field that lets you Save and Open 'presets' of all the options.
That would be great for theme developers to include various preset styles of the theme,
or let the user save current options as a preset.
This could also be used by Agencies to display multiple variations of a theme to a client for approval.
I made something quick and dirty based on this that worked:
$('img.kirki_image_preset').click(function() {
		var api = wp.customize;
		var data = $( this ).data( 'options' );

		if ( typeof data !== 'undefined' ) {
			$.each( data, function( id, value ) {
				console.log(id);
				console.log(value);
				var setting = api( id );
				if(setting){
					setting.set( value );
				}
			} );
		}
		api.previewer.refresh();
	} );

The img.kirki_image_preset was just a image with a option array inside data-option="".

NICE... +1

That's nice idea, many developer need it.

Any news? :) I can see that Redux Customizer has this.

Hey guys, just an update on this one.
We'll be building a "preset" control pretty soon.
@tomtev do you have something functional? If you do could you please submit a pull request?

Hey guys!
Just an update here.
I've started implementing this and it's semi-working.
By semi-working I mean that you can define presets like this:
Kirki::add_field( 'kirki_demo', array(
	'type'        => 'preset',
	'settings'    => 'preset_demo',
	'label'       => __( 'Preset control', 'kirki-demo' ),
	'description' => __( 'Bulk-changes the value of other controls.', 'kirki-demo' ),
	'section'     => 'select',
	'default'     => '1',
	'priority'    => 10,
	'multiple'    => 3,
	'choices'     => array(
		'1' => array(
			'label'    => __( 'Option 1', 'kirki-demo' ),
			'settings' => array(
				'select_demo'          => 'red',
				'select_multiple_demo' => array( 'option-1', 'option-2' ),
			),
		),
		'2' => array(
			'label'    => __( 'Option 2', 'kirki-demo' ),
			'settings' => array(
				'select_demo'          => 'green',
				'select_multiple_demo' => array( 'option-4', 'option-1' ),
			),
		),
	),
) );

You can also see it in the demo theme: https://github.com/aristath/kirki/blob/cac7a24fc71308755dcf97c7227c77107a68d2b5/demo-theme/functions.php#L502-L527
When you change the value, the choices are bulk-changed and their values can be previewed.
However, what does not change is the value as seen in the controls themselves.
If someone can help out I'm sure a lot of us can use this control!
NOTE: For the time being this control is a select control, but as soon as we get the basics working I'm sure we'll be able to find a way to make it work with other control types as well.

Update: I tried making the control values update using this: https://github.com/aristath/kirki/blob/27e9997a0019d1cfd19492030fdb485f6675a6a8/assets/js/controls/preset.js#L30-L84 and failed miserably.
I feel I'm on the right track but my JS voodoo is pretty limited. :hankey:

Closing this one in favour of https://github.com/aristath/kirki/issues/393 so that we may better track the progress there.


Upper bound with refinement defaults to Object and not to Any

The following code compiles:
trait A:
  type T

trait B extends A:
  type U
  type T <: U

But after changing the definition of  of A to:
trait A:
  type T <: { }

The code fails with:
error overriding type T in trait A with bounds <: Object;
  type T with bounds <: B.this.U has incompatible type

It can be made compile with the following:
trait A:
  type T <: Any { }

This is not really a bug, but it is somewhat surprising that a refinement changes the upper bound from Any to Object. It means that a solution is to write down the Any upper bound explicitly on every refinement. It seems a little bit inconsistent with the default upper bound being <: Any in other cases (and not <: Object). I noticed that a single line could be changed to make the default "parent" of a refinement Any instead of Object:
--- a/compiler/src/dotty/tools/dotc/typer/Typer.scala
+++ b/compiler/src/dotty/tools/dotc/typer/Typer.scala
@@ -2209,7 +2209,7 @@ class Typer(@constructorOnly nestingLevel: Int = 0) extends Namer
   }

   def typedRefinedTypeTree(tree: untpd.RefinedTypeTree)(using Context): TypTree = {
-    val tpt1 = if (tree.tpt.isEmpty) TypeTree(defn.ObjectType) else typedAheadType(tree.tpt)
+    val tpt1 = if (tree.tpt.isEmpty) TypeTree(defn.AnyType) else typedAheadType(tree.tpt)
     val refineClsDef = desugar.refinedTypeToClass(tpt1, tree.refinements).withSpan(tree.span)
     val refineCls = createSymbol(refineClsDef).asClass
     val TypeDef(_, impl: Template) = typed(refineClsDef): @unchecked

My question is: Is there an inherent reason why the default is Object instead of Any or is there a chance a pull request would be accepted to change the default?

This behavior is as specified. I think I failed to carry over the spec text from 2.13 to 3.4, but in the Scala 2.13 specification you can clearly find:
https://scala-lang.org/files/archive/spec/2.13/03-types.html#compound-types

A compound type may also consist of just a refinement {R} with no preceding component types. Such a type is equivalent to AnyRef {R}.

Changing it would be a breaking change requiring a SIP, and I don't think it is likely to be accepted.


ARTEMIS-2249 - update cpp example to use correct client libs

https://issues.apache.org/jira/browse/ARTEMIS-2249

This must be env related, I only changed an example

I will merge this today.


AWS API lies about Internet Gateway detachment status during cloudstrap-teardown

This blocks teardown

It's safe to re-run cloudstrap-teardown, in this scenario. Ideally, this should not be required. I'll see what I can wrangle.


[VIS] States from multiple CPUS per node reporting

How to report in a single line the states from multiple CPUs per node
Ensure that "interesting" states are reported
e.g. use two lines with min-max information

e.g. report minimum and maximum temperature of cores in a node (vertically)


Enable cudaSupport, upstream various packages, LLM additions

A few unimportant (for us) Python regressions: numpyro/arviz/pytensor/pymc, skorch, timm

Nice!


Password changes for existing dummy user

Hi,
Is it possible to change password for the existing user on dummy authorizer?

from pyftpdlib.authorizers import DummyAuthorizer
authorizer = DummyAuthorizer()
authorizer.add_user('user', '12345', os.getcwd(), perm='elradfmwMT')
authorizer.user_table['user']['pwd'] = 'newpassword'


✨ Isolate Language Component in Library

Is there an existing issue for this?

[X] I have searched the existing issues

Suggestion
The json files in lang in library should be within the app projects.
For example
📂 admin-app
    📂 src
         📂 lang
             en.json
             MessageId.ts
📂 result-app
    📂 src
         📂 lang
             en.json
             MessageId.ts

This actually means a lot of changes on the BaseComponents to allow this. The idea is to create a Language component that can be reused without duplicating code between the app applications.

Possible Implementation
Example of a Generic Message
import { MessageDescriptor } from '@formatjs/intl';

const OverloadMessageId = 'overload';

const PlaceHolderMessageId = 'placeholder';

const placeholderMessage = 'Message Not Found';

export default class GenericMessage<T extends string | number> implements MessageDescriptor {
    id: T | typeof OverloadMessageId | typeof PlaceHolderMessageId;

    defaultMessage: string;

    description?: string;

    constructor(
        id: T | typeof OverloadMessageId | typeof PlaceHolderMessageId = PlaceHolderMessageId,
        defaultMessage = placeholderMessage,
        description?: string
    ) {
        this.id = id;
        this.defaultMessage = defaultMessage;
        this.description = description;
    }
}

export type GenericMessageDescriptor = MessageDescriptor;

Example of a generic piece to pull the languages from library.
import EnumDictionary from '@electionguard-ui/library/src/utils/EnumDictionary';

import Locale from './Locale';
import Language from './Language';

/**
 * Get language for locale based on list of supported languages
 * @param supportedLanguages
 * @returns Language
 */
const getLanguage = (supportedLanguages: EnumDictionary<Locale, Language>): Language => {
    const locale = navigator.language;

    if (locale in Locale) {
        return supportedLanguages[locale as Locale];
    }

    return supportedLanguages.default;
};

export default getLanguage;

Anything else?
No response

@keithrfung I'd like to pick this up as well, can you assign this to me?

@deepto98 you got it


Keeping row position after quick-filtering

Description of the feature, modification, idea or suggestion
I frequently use the CTRL-K for quick-filtering to hone in on particular processes/PIDs. However, sometimes it's not the process I'm interested in, but rather, wanting to locate it to find its child or sibling process, and/or its relative position in the tree. When clearing the quick-filter to show the tree, however, the main pane's row selector is also cleared.
Not a huge deal, as an alternative it necessitates keyboard letter-selecting to it (multiple hits necessary or having to bring up the full CTRL-F searh and lock-on to a process that way.
But if it'd be possible to add an option to "Keep row position upon quick-filter clear", it would be most useful and much appreciated!
Proposed implementation details (optional)
No response

Added to latest nightly. Enable by changing RestoreSelectionAfterSearch to 1


Android Basics: Fragments Navigation component

URL of codelab
https://developer.android.com/codelabs/basic-android-kotlin-training-fragments-navigation-component?continue=https%3A%2F%2Fdeveloper.android.com%2Fcourses%2Fpathways%2Fandroid-basics-kotlin-unit-3-pathway-2%23codelab-https%3A%2F%2Fdeveloper.android.com%2Fcodelabs%2Fbasic-android-kotlin-training-fragments-navigation-component#8
In which task and step of the codelab can this issue be found?
Last step when running.
Describe the problem
Navigating to WordListFragment gives a black screen and fails to go up.
Steps to reproduce?

Open app
Click on any letter
See black screen

Versions
Android Studio version: Android Studio Arctic Fox | 2020.3.1 Patch 3
API version of the emulator: API 30
Additional information
Include screenshots if they would be useful in clarifying the problem.

@darwinyip
Thank you for reaching out to us. In order to help you with troubleshooting, we need to review the code snippet with the error. First we recommend to update your Android Studio version, then download the recently updated code and follow the steps provided in the codelab. If the error happens again, feel free to reopen the ticket.
Happy coding!


Support globus token via URL query parameter in search-api

confirmed with Chuck M., he only needs this for the file service due to the image libraries they are using.


New or removed repositories (added repos 1, removed repos: 0)

The following repos were added

italia/keycloak-spid-provider

#69


Roadmap

Now that I've forked Bork, I should have a plan for its future. Looking at @mattly's 1.0 roadmap has informed a lot of where I want to go with Bork in the future. This is subject to change as things develop, but I would like to continue this work from that foundation where I can.
Finish fleshing out each of the types

I just recently added outdated / update support and tests to npm, and there are more types where these are missing. I'd like to complete these.

I'd like to look through the types as they are now (this quote is from 2016, after all) and see where these can be added.
Add 'remove' commands to types, and an opposite of the ok assertion

This is something I've been thinking about for a while, and in theory it's not terribly hard. Mostly it's come down to semantics for what the opposite of ok is. So perhaps we alias ok to should or pass and then provide should-not or fail. So for example pass brew asserts the presence of homebrew, fail brew asserts the absence of it.

This would be a good one to sink my teeth into. It doesn't form part of my personal computing workflow, but for work previously it's been part of my IaC strategy to remove a bunch of packages from a fresh Linux install to clean it up a bit. So I see the merit of doing this. Just, now it's me that has to make that semantics decision. 😅
Improved CLI help

I see this as a two-pronged effort:

Man pages
a separate help system for individual types to get more details about their usage

No more to add here; I think both of these goals are reasonable. This also feeds into the next point, and some of the work I've started to do with bork docgen will inform the hosted documentation. Each type's desc function seems to be the logical place to start for per-type usage information.
Static Website

I have borksh.com and a SSG tool I like working with already, I just need to figure out the Info Architecture and start putting things together. I see this site as serving two purposes: Marketing and Documentation. Marketing is the what and why, and Documentation is the how. I could use the most input on the marketing side of things. I have a rough plan for the docs that include both guides (to help learn) and reference (to help remind), and some good docs-focused friends to ping ideas off of.

I've begun this with the GitHub Pages site that I build from docs/, and the bork docgen which spits out each type's desc into a Jekyll-compatible Markdown file. It's really just fleshing out the text and making sure the documentation is properly thorough. borksh.com has since expired, but if this picks up enough steam I'll register it again.

I was always impressed by Bork's community, and anyone who had an interest in contributing to the original project is more than welcome here. If anyone else is still out there using Bork, I would love to read your ~hate mail~ thoughts in the comments.

I have added a man page using ronn, and an Actions workflow that will compile it for us. I would like to tweak this so that it generates the man page in Markdown as well for the website (hence the sed hack in the workflow).
I think the best thing here would be to figure out some sort of overarching documentation strategy, as currently the same thing is now being said in three different places (man page, README, and website).


 component=Controller.Run subsys=bgp-control-plane
[07:37:34] level=info msg="Cilium BGP Control Plane Controller woken for reconciliation" component=Controller.Run subsys=bgp-control-plane
[07:37:34] level=info msg="Registering BGP servers for policy with local ASN 65001" component=manager.registerBGPServer subsys=bgp-control-plane
[07:37:34] level=info msg="Reconciling peers for virtual router with local ASN 65001" component=manager.neighborReconciler subsys=bgp-control-plane
[07:37:34] level=info msg="Adding peer <IP_ADDRESS>/32 65011 to local ASN 65001" component=manager.neighborReconciler subsys=bgp-control-plane
[07:37:34] level=info msg="Add a peer configuration" Key=<IP_ADDRESS> Topic=Peer asn=65001 component=gobgp.BgpServerInstance subsys=bgp-control-plane
[07:37:34] level=info msg="Done reconciling peers for virtual router with local ASN 65001" component=manager.neighborReconcilersubsys=bgp-control-plane
[07:37:34] level=info msg="Successfully registered GoBGP servers for policy with local ASN 65001" component=manager.registerBGPServersubsys=bgp-control-plane
[07:37:34] time="2023-05-24T07:33:39Z" level=info msg="GoBGP test instance: Peer Event: {65001 IDLE}"
[07:37:34] time="2023-05-24T07:33:39Z" level=info msg="GoBGP test instance: Peer Event: {65001 ACTIVE}"
[07:37:34] level=info msg="Cilium BGP Control Plane Controller woken for reconciliation" component=Controller.Run subsys=bgp-control-plane
[07:37:34] level=info msg="Reconciling peers for virtual router with local ASN 65001" component=manager.neighborReconciler subsys=bgp-control-plane
[07:37:34] level=info msg="Adding peer <IP_ADDRESS>/32 65012 to local ASN 65001" component=manager.neighborReconciler subsys=bgp-control-plane
[07:37:34] level=info msg="Add a peer configuration" Key=<IP_ADDRESS> Topic=Peer asn=65001 component=gobgp.BgpServerInstance subsys=bgp-control-plane
[07:37:34] level=info msg="Done reconciling peers for virtual router with local ASN 65001" component=manager.neighborReconcilersubsys=bgp-control-plane
[07:37:34] level=info msg="type:STATE  peer:{conf:{local_asn:65001  neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65011}  state:{local_asn:65001 neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65011  session_state:IDLE  router_id:\"<nil>\"}  transport:{local_address:\"<nil>\"}}"
[07:37:34] level=info msg="type:STATE  peer:{conf:{local_asn:65001  neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65011}  state:{local_asn:65001 neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65011  session_state:ACTIVE  router_id:\"<nil>\"}  transport:{local_address:\"<nil>\"}}"
[07:37:34] level=info msg="type:STATE  peer:{conf:{local_asn:65001  neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65012}  state:{local_asn:65001 neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65012  session_state:IDLE  router_id:\"<nil>\"}  transport:{local_address:\"<nil>\"}}"
[07:37:34] level=info msg="type:STATE  peer:{conf:{local_asn:65001  neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65012}  state:{local_asn:65001 neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65012  session_state:ACTIVE  router_id:\"<nil>\"}  transport:{local_address:\"<nil>\"}}"
[07:37:34] time="2023-05-24T07:33:46Z" level=debug msg="try to connect" Key=<IP_ADDRESS> Topic=Peer asn=65011 component=tests.BGP subsys=basic
[07:37:34] time="2023-05-24T07:33:46Z" level=debug msg="state changed" Key=<IP_ADDRESS> Topic=Peer asn=65011 component=tests.BGP new=BGP_FSM_OPENSENTold=BGP_FSM_ACTIVE reason=new-connection subsys=basic
[07:37:34] time="2023-05-24T07:33:46Z" level=info msg="GoBGP test instance: Peer Event: {65001 OPENSENT}"
[07:37:34] level=info msg="type:STATE  peer:{conf:{local_asn:65001  neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65011}  state:{local_asn:65001 neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65011  session_state:OPENSENT  router_id:\"<nil>\"}  transport:{local_address:\"<IP_ADDRESS>\" local_port:1790  remote_port:40939}}"
[07:37:34] level=info msg="type:STATE  peer:{conf:{local_asn:65001  neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65011}  state:{local_asn:65001 neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65011  session_state:OPENCONFIRM  router_id:\"<IP_ADDRESS>\"}  transport:{local_address:\"<IP_ADDRESS>\" local_port:1790  remote_port:40939}}"
[07:37:34] time="2023-05-24T07:33:46Z" level=debug msg="state changed" Key=<IP_ADDRESS> Topic=Peer asn=65011 component=tests.BGPnew=BGP_FSM_OPENCONFIRM old=BGP_FSM_OPENSENT reason=open-msg-received subsys=basic
[07:37:34] time="2023-05-24T07:33:46Z" level=info msg="GoBGP test instance: Peer Event: {65001 OPENCONFIRM}"
[07:37:34] time="2023-05-24T07:33:46Z" level=info msg="Peer Up" Key=<IP_ADDRESS> State=BGP_FSM_OPENCONFIRM Topic=Peer asn=65011 component=tests.BGPsubsys=basic
[07:37:34] time="2023-05-24T07:33:46Z" level=debug msg="state changed" Key=<IP_ADDRESS> Topic=Peer asn=65011 component=tests.BGPnew=BGP_FSM_ESTABLISHED old=BGP_FSM_OPENCONFIRM reason=open-msg-negotiated subsys=basic
[07:37:34] time="2023-05-24T07:33:46Z" level=info msg="GoBGP test instance: Peer Event: {65001 ESTABLISHED}"
[07:37:34] time="2023-05-24T07:33:46Z" level=info msg="GoBGP test instance: Peer Event: {0 IDLE}"
[07:37:34] time="2023-05-24T07:33:46Z" level=info msg="GoBGP test instance: Peer Event: {65001 IDLE}"
[07:37:34] time="2023-05-24T07:33:46Z" level=info msg="GoBGP test instance: Peer Event: {65001 ACTIVE}"
[07:37:34] level=info msg="Peer Up" Key=<IP_ADDRESS> State=BGP_FSM_OPENCONFIRM Topic=Peer asn=65001 component=gobgp.BgpServerInstancesubsys=bgp-control-plane
[07:37:34] level=info msg="type:STATE  peer:{conf:{local_asn:65001  neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65011}  state:{local_asn:65001 neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65011  session_state:ESTABLISHED  router_id:\"<IP_ADDRESS>\"}  transport:{local_address:\"<IP_ADDRESS>\" local_port:1790  remote_port:40939}}"
[07:37:34] time="2023-05-24T07:33:48Z" level=debug msg="try to connect" Key=<IP_ADDRESS> Topic=Peer asn=65011 component=tests.BGP subsys=basic
[07:37:34] time="2023-05-24T07:33:48Z" level=debug msg="state changed" Key=<IP_ADDRESS> Topic=Peer asn=65011 component=tests.BGP new=BGP_FSM_OPENSENTold=BGP_FSM_ACTIVE reason=new-connection subsys=basic
[07:37:34] time="2023-05-24T07:33:48Z" level=info msg="GoBGP test instance: Peer Event: {65001 OPENSENT}"
[07:37:34] level=info msg="type:STATE  peer:{conf:{local_asn:65001  neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65012}  state:{local_asn:65001 neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65012  session_state:OPENSENT  router_id:\"<nil>\"}  transport:{local_address:\"<IP_ADDRESS>\" local_port:1790  remote_port:56241}}"
[07:37:34] level=info msg="type:STATE  peer:{conf:{local_asn:65001  neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65012}  state:{local_asn:65001 neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65012  session_state:OPENCONFIRM  router_id:\"<IP_ADDRESS>\"}  transport:{local_address:\"<IP_ADDRESS>\" local_port:1790  remote_port:56241}}"
[07:37:34] time="2023-05-24T07:33:48Z" level=debug msg="state changed" Key=<IP_ADDRESS> Topic=Peer asn=65011 component=tests.BGPnew=BGP_FSM_OPENCONFIRM old=BGP_FSM_OPENSENT reason=open-msg-received subsys=basic
[07:37:34] time="2023-05-24T07:33:48Z" level=info msg="GoBGP test instance: Peer Event: {65001 OPENCONFIRM}"
[07:37:34] time="2023-05-24T07:33:48Z" level=info msg="Peer Up" Key=<IP_ADDRESS> State=BGP_FSM_OPENCONFIRM Topic=Peer asn=65011 component=tests.BGPsubsys=basic
[07:37:34] time="2023-05-24T07:33:48Z" level=debug msg="state changed" Key=<IP_ADDRESS> Topic=Peer asn=65011 component=tests.BGPnew=BGP_FSM_ESTABLISHED old=BGP_FSM_OPENCONFIRM reason=open-msg-negotiated subsys=basic
[07:37:34] time="2023-05-24T07:33:48Z" level=info msg="GoBGP test instance: Peer Event: {65001 ESTABLISHED}"
[07:37:34] level=info msg="Peer Up" Key=<IP_ADDRESS> State=BGP_FSM_OPENCONFIRM Topic=Peer asn=65001 component=gobgp.BgpServerInstancesubsys=bgp-control-plane
[07:37:34]     neighbor_test.go:117:
[07:37:34]         	Error Trace:	/home/vagrant/go/src/github.com/cilium/cilium/pkg/bgpv1/test/neighbor_test.go:117
[07:37:34]         	Error:      	elements differ
[07:37:34]
[07:37:34]         	            	extra elements in list A:
[07:37:34]         	            	([]interface {}) (len=1) {
[07:37:34]         	            	 (test.peeringState) {
[07:37:34]         	            	  peerASN: (uint32) 65012,
[07:37:34]         	            	  peerAddr: (string) (len=12) "<IP_ADDRESS>",
[07:37:34]         	            	  peerSession: (string) (len=11) "established"
[07:37:34]         	            	 }
[07:37:34]         	            	}
[07:37:34]
[07:37:34]
[07:37:34]         	            	extra elements in list B:
[07:37:34]         	            	([]interface {}) (len=1) {
[07:37:34]         	            	 (test.peeringState) {
[07:37:34]         	            	  peerASN: (uint32) 65012,
[07:37:34]         	            	  peerAddr: (string) (len=12) "<IP_ADDRESS>",
[07:37:34]         	            	  peerSession: (string) (len=12) "open_confirm"
[07:37:34]         	            	 }
[07:37:34]         	            	}
[07:37:34]
[07:37:34]
[07:37:34]         	            	listA:
[07:37:34]         	            	([]test.peeringState) (len=2) {
[07:37:34]         	            	 (test.peeringState) {
[07:37:34]         	            	  peerASN: (uint32) 65011,
[07:37:34]         	            	  peerAddr: (string) (len=12) "<IP_ADDRESS>",
[07:37:34]         	            	  peerSession: (string) (len=11) "established"
[07:37:34]         	            	 },
[07:37:34]         	            	 (test.peeringState) {
[07:37:34]         	            	  peerASN: (uint32) 65012,
[07:37:34]         	            	  peerAddr: (string) (len=12) "<IP_ADDRESS>",
[07:37:34]         	            	  peerSession: (string) (len=11) "established"
[07:37:34]         	            	 }
[07:37:34]         	            	}
[07:37:34]
[07:37:34]
[07:37:34]         	            	listB:
[07:37:34]         	            	([]test.peeringState) (len=2) {
[07:37:34]         	            	 (test.peeringState) {
[07:37:34]         	            	  peerASN: (uint32) 65011,
[07:37:34]         	            	  peerAddr: (string) (len=12) "<IP_ADDRESS>",
[07:37:34]         	            	  peerSession: (string) (len=11) "established"
[07:37:34]         	            	 },
[07:37:34]         	            	 (test.peeringState) {
[07:37:34]         	            	  peerASN: (uint32) 65012,
[07:37:34]         	            	  peerAddr: (string) (len=12) "<IP_ADDRESS>",
[07:37:34]         	            	  peerSession: (string) (len=12) "open_confirm"
[07:37:34]         	            	 }
[07:37:34]         	            	}
[07:37:34]         	Test:       	Test_NeighborAddDel/add_two_neighbors
[07:37:34]         	Messages:   	add two neighbors
[07:37:34] level=info msg="type:STATE  peer:{conf:{local_asn:65001  neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65012}  state:{local_asn:65001 neighbor_address:\"<IP_ADDRESS>\"  peer_asn:65012  session_state:ESTABLISHED  router_id:\"<IP_ADDRESS>\"}  transport:{local_address:\"<IP_ADDRESS>\" local_port:1790  remote_port:56241}}"

I believe this is an issue in our test logic, taking a look.


Create Insertion Sort

Add in sorting folder of python

Hi @akshitagupta15june, can I work on this issue ?

ofcourse! open for all

i have added insertion sort


Improve contributing guidelines

After yesterday livestream, I think there is room to improve "Getting a development environment set up" (Courtney struggle with our instructions)
So I'm going to send a proposal, CONTRIBUTING.md and including a development environment setup.

PR done and approved: https://github.com/Esri/arcgis-js-vscode-snippets/pull/48


Add extra fields

Make extra fields editable
Fix #810

Should now also support FEP-fb2a:
"attachment": [
  {
    "type": "PropertyValue",
    "name": "Pronouns",
    "value": "
he/him/his
"
  },
  {
    "type": "Note",
    "name": "Pronouns",
    "content": "
he/him/his
"
  },
  {
    "type": "PropertyValue",
    "name": "Blog",
    "value": "
localhost
"
  },
  {
    "type": "Link",
    "name": "Blog",
    "href": "http://localhost:8076/",
    "rel": [
      "me",
      "noopener"
    ]
  }
]

/cc @johnspurlock

To me it was surprising to arrive in the Gutenberg editor when I clicked "Add new", it felt like a bug.

Coming in late to this discussion, to say that I experienced the same thing. To be honest, I'm still not sure I added the new field correctly. I created a new entry, with a title and just the link in the post content, but that didn't seem to work. I had to go back and add text with a link on it for it to appear in the profile page.
I like the idea of making this editable directly through the Profile page.

Coming in late to this discussion, to say that I experienced the same thing. To be honest, I'm still not sure I added the new field correctly. I created a new entry, with a title and just the link in the post content, but that didn't seem to work. I had to go back and add text with a link on it for it to appear in the profile page.

This might be a bug, I will check this.

I like the idea of making this editable directly through the Profile page.

We can add that feature, but I still like having the Block editor to help with text and HTML. The extra fields are not limited to URLs, so why not profit from the build in features of WordPress?

The extra fields are not limited to URLs, so why not profit from the build in features of WordPress?

This is true. Maybe we could display a small rich text editable input field on the profile page?

@jeherve Auto-Link comes with #849 btw.

Thank you!


changing threshold

Hi,
How can we change the threshold of an model for mtcnn.

You can inject the threshold values in the constructor of the MTCNN object. By default the thresholds are these 3 values [0.6, 0.7, 0.7].
    mtcnn = MTCNN(steps_threshold=[0.6, 0.7, 0.7])

Hi,
Thanks for your reply and actually my doubt is when multiple faces detected
in an same image how we crop the highest confidence score image.
When I used your face detection code I faced the problem is
If 3 faces are detected then it was cropping first one and next cropping
second one but by replacing the first one saving only second image.
So please help me in this problem.
On 18-Mar-2019 2:16 PM, "Iván de Paz Centeno"<EMAIL_ADDRESS>wrote:

You can inject the threshold values in the constructor of the MTCNN
object. By default the thresholds are these 3 values [0.6, 0.7, 0.7].
mtcnn = MTCNN(steps_threshold=[0.6, 0.7, 0.7])

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ipazc/mtcnn/issues/30#issuecomment-473819416, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AqwPGfTxViQlQxpCyOwVA9n9O5KtGaZWks5vX1JRgaJpZM4b5Cju
.

I guess your question is not about the implementation of MTCNN but about how to use the results of MTCNN. Am I right?

No I am understand the mtcnn but I am confused in getting cropped results
for image with three persons and I am new to python and much perfect in
python.
So I am confused in cropping the images with highest confidence score
On 18-Mar-2019 5:57 PM, "Iván de Paz Centeno"<EMAIL_ADDRESS>wrote:

I guess your question is not about the implementation of MTCNN but about
how to use the results of MTCNN. Am I right?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ipazc/mtcnn/issues/30#issuecomment-473889538, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AqwPGWRWv2dTa04mtG8Ot_0nWxUMZeAcks5vX4Y9gaJpZM4b5Cju
.

I'm sorry, issues page is not intended to solve such questions/problems outside of MTCNN domain. You might better try asking on stackoverflow.

Ok thanks
On 18-Mar-2019 6:17 PM, "Iván de Paz Centeno"<EMAIL_ADDRESS>wrote:

I'm sorry, issues page is not intended to solve such questions/problems
outside of MTCNN domain. You might better try asking on stackoverflow.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ipazc/mtcnn/issues/30#issuecomment-473895696, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AqwPGeZ-muLG2oQRqfe8C7QeP1zSFWRvks5vX4ragaJpZM4b5Cju
.


Sponsorship Paymaster Accounting and Gas limits testing on public testnet

Feature or Improvement Description
Test updated paymaster and finetune
Benefits & Outcomes
send several transactions using this paymaster for below cases
i. mint NFT/transfer erc20 with premium 0 (1e6)
ii. do the above with premium 10%
iii. do the above with premium 20%
iv. check fee collector balance to verify premiums
v. do transactions a few times to warm up the slots
NOTE: postOpGasLimit must be higher than unaccountedGasCost set otherwise it will revert
vi. catch possible reverts from bundler
vii. accounting checks. make sure paymaster's deposit on entrypoint is always higher than sum of dappIdBalances
outcome is to set gas limits and unacocuntedGasCost in proper way such that paymaster always makes profit, and the dapp is charged accurately.
do simulations on tenderly to figure out gas limits and set config accordingly, and then repeat the transactions.
Any References?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct.

tx # 3.
Slots have been warmed up
Premium : 10%
Paymaster's current deposit on EP:<PHONE_NUMBER>6024771
Dapp's current balance on paymaster:<PHONE_NUMBER>8285575
Fee collector's current balance:<PHONE_NUMBER>7474
setting unaccounted EP gas to: 15000
tx hash: https://sepolia.basescan.org/tx/0xa3d598122ae7b0540b284049b03750fe6986ee63789be2efce95287d662e9654
tenderly: https://dashboard.tenderly.co/livingrock777/biconomy/tx/84532/0xa3d598122ae7b0540b284049b03750fe6986ee63789be2efce95287d662e9654/gas-usage
bundler paid: 0.000082001141128017 =<PHONE_NUMBER>0000
entrypoint paid to bundler: 0.000421575907908534 (huge difference) =<PHONE_NUMBER>00000
Paymaster’s balance on EP:<PHONE_NUMBER>8116237

deducted for the paymaster:  =<PHONE_NUMBER>00000
Dapp’s deposit on the EP:<PHONE_NUMBER>1836452
Deducted from the dapp:  =<PHONE_NUMBER>00000  =
Fee collector's balance on pm:<PHONE_NUMBER>1265
fee collector's collected premium(bal change) =<PHONE_NUMBER>0000
Info from event:
adjustedGasCost=charge:<PHONE_NUMBER>61707 (charged from the dapp would be more than this cause of penalties?)
premium=<PHONE_NUMBER>3791 (not an exact match maybe cause of converting scientific notation..)
deducted from dapp =<PHONE_NUMBER>00000
Actual gas cost emitted by entrypoint:<PHONE_NUMBER>08534 (maybe it passes low to the postOp but then after penalties increases it and sends?)

on latest deployed paymaster on base sepolia
https://sepolia.basescan.org/address/0x00000005858b63176f39f9cfcb51ab4e97b84a2a
paymaster deposit on EP:<PHONE_NUMBER>0347448
dapp's deposit on paymaster:<PHONE_NUMBER>0958858
fee collector's balance on pm:<PHONE_NUMBER>85730
setting unaccounted gas : 15000


use slimerjs as engine for casperjs

In need webfonts to work with the test, so is it possible to pass an option to casperjs to run with slimerjs engine?

+1 I'm currently looking at moving to slimerjs for better browser support too.