MNIST example hits an error about 'AutoProxy[get_queue]' object has no attribute 'put'

I am trying to spark-submit the MNIST example on a 3 node Hadoop v3.0+Spark v2.3 cluster and not sure what to do with the error below I encountered.
spark-submit \ --master yarn \ --deploy-mode cluster \ --queue default \ --num-executors 2 \ --executor-memory 24G \ --py-files /home/junzhang22/TensorFlowOnSpark/examples/mnist/spark/mnist_dist.py \ --conf spark.dynamicAllocation.enabled=false \ --conf spark.yarn.maxAppAttempts=1 \ /home/junzhang22/TensorFlowOnSpark/examples/mnist/spark/mnist_spark.py \ --images hdfs://dlaas-185:9000/home/junzhang22/mnist/csv/train/images \ --labels hdfs://dlaas-185:9000/home/junzhang22/mnist/csv/train/labels \ --mode train \ --model mnist_model
Driver stacktrace: 2018-03-19 17:24:25 INFO  DAGScheduler:54 - Job 1 failed: collect at PythonRDD.scala:153, took 13.917543 s Traceback (most recent call last):   File "mnist_spark.py", line 69, in <module>     cluster.train(dataRDD, args.epochs)   File "/usr/lib/python2.7/site-packages/tensorflowonspark/TFCluster.py", line 90, in train     unionRDD.foreachPartition(TFSparkNode.train(self.cluster_info, self.cluster_meta, qname))   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/pyspark.zip/pyspark/rdd.py", line 814, in foreachPartition   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/pyspark.zip/pyspark/rdd.py", line 1056, in count   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/pyspark.zip/pyspark/rdd.py", line 1047, in sum   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/pyspark.zip/pyspark/rdd.py", line 921, in fold   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/pyspark.zip/pyspark/rdd.py", line 824, in collect   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1160, in __call__   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/py4j-0.10.6-src.zip/py4j/protocol.py", line 320, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 9 in stage 1.0 failed 4 times, most recent failure: Lost task 9.3 in stage 1.0 (TID 20, dlaas-183, executor 2): org.apache.spark.api.python.PythonException: Traceback (most recent call last):   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000003/pyspark.zip/pyspark/worker.py", line 229, in main     process()   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000003/pyspark.zip/pyspark/worker.py", line 224, in process     serializer.dump_stream(func(split_index, iterator), outfile)   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/pyspark.zip/pyspark/rdd.py", line 2438, in pipeline_func   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/pyspark.zip/pyspark/rdd.py", line 2438, in pipeline_func   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/pyspark.zip/pyspark/rdd.py", line 2438, in pipeline_func   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/pyspark.zip/pyspark/rdd.py", line 362, in func   File "/usr/local/hadoop-3.0.0/tmp/nm-local-dir/usercache/root/appcache/application_1520844482373_0026/container_1520844482373_0026_01_000001/pyspark.zip/pyspark/rdd.py", line 809, in func   File "/usr/lib/python2.7/site-packages/tensorflowonspark/TFSparkNode.py", line 367, in _train     queue.put(item, block=True) AttributeError: 'AutoProxy[get_queue]' object has no attribute 'put'
From the log, I can see that cluster has been set up and MNIST data had been partitioned into multiple tasks as well. The only thing I suspect is that I installed tensorflowonspark via pip and its version is
tensorflowonspark (1.2.1)  - Deep learning with TensorFlow on Apache Spark clusters   INSTALLED: 1.2.1 (latest)
Anyone who happens to run into the same issue, please comment and suggest. Thanks.

For Hadoop/YARN, you will need to make sure that tensorflowonspark is installed on all of your grid nodes.  Otherwise, you'll have to use the --py-files tfspark.zip method.

@leewyang Thanks for the reply. Initially, I installed tensorflowonspark on all my worker nodes via pip. After I pip uninstalled and explicitly specified tfspark.zip in my spark-submit statement; the same error persisted. Since the error AttributeError: 'AutoProxy[get_queue]' object has no attribute 'put' seems to be some Python version issue, I then upgraded python from 2.7.5 to 2.7.12. I have tried the upgrade by both passing the complied Pyhton zip in spark-submit and directly linking nodes' python bin to 2.7.12; howver, neither really helped. Not sure if I need to downgrade Hadoop(v3.0) and Spark(v2.3) version as well. BTW, is there any plan to dockerize the task (to get rid of all these version issues) since K8S is integrated into Spark as well.

There aren't any short-term plans to dockerize the task, although that would be an eventual goal...
Can you grab the full yarn logs and check for any errors on the executors?

the same issue i met, with hadoop 2.9 , spark2.3 and python3.6.
Is the problem solved ?

@markfengyunzhou can you send me your spark-submit command line and the yarn logs for a failure?  (There's a chance that this is just a symptom of an upstream failure).

@markfengyunzhou Can you try using:
PYSPARK_DRIVER_PYTHON=Python/bin/python3
PYSPARK_PYTHON=Python/bin/python3

Also, those are just the driver logs, if you can grab the full yarn logs via: yarn logs -applicationId <your_appId>, you'll be able to see the executor logs (and hopefully some root cause errors).

@leewyang
thank you very much for your attention.
PYSPARK_DRIVER_PYTHON=Python/bin/python3
PYSPARK_PYTHON=Python/bin/python3

After modification, it does not solve the problem.
and the full yarn logs got by yarn logs -applicationId <your_appId> below.
End of LogType:prelaunch.err.This log file belongs to a running container (container_1525333084924_0016_01_000002) and so may not be complete.
******************************************************************************

Container: container_1525333084924_0016_01_000002 on hpe02:36742
LogAggregationType: LOCAL
================================================================
LogType:stdout
LogLastModifiedTime:星期一 五月 07 09:22:41 +0800 2018
LogLength:8346
LogContents:
2018-05-07 09:22:26 INFO  CoarseGrainedExecutorBackend:2608 - Started daemon with process name: 10871@hpe02
2018-05-07 09:22:26 INFO  SignalUtils:54 - Registered signal handler for TERM
2018-05-07 09:22:26 INFO  SignalUtils:54 - Registered signal handler for HUP
2018-05-07 09:22:26 INFO  SignalUtils:54 - Registered signal handler for INT
2018-05-07 09:22:27 INFO  SecurityManager:54 - Changing view acls to: hpe
2018-05-07 09:22:27 INFO  SecurityManager:54 - Changing modify acls to: hpe
2018-05-07 09:22:27 INFO  SecurityManager:54 - Changing view acls groups to:
2018-05-07 09:22:27 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-05-07 09:22:27 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hpe); groups with view permissions: Set(); users  with modify permissions: Set(hpe); groups with modify permissions: Set()
2018-05-07 09:22:27 INFO  TransportClientFactory:267 - Successfully created connection to hpe01/<IP_ADDRESS>:41862 after 120 ms (0 ms spent in bootstraps)
2018-05-07 09:22:28 INFO  SecurityManager:54 - Changing view acls to: hpe
2018-05-07 09:22:28 INFO  SecurityManager:54 - Changing modify acls to: hpe
2018-05-07 09:22:28 INFO  SecurityManager:54 - Changing view acls groups to:
2018-05-07 09:22:28 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-05-07 09:22:28 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hpe); groups with view permissions: Set(); users  with modify permissions: Set(hpe); groups with modify permissions: Set()
2018-05-07 09:22:28 INFO  TransportClientFactory:267 - Successfully created connection to hpe01/<IP_ADDRESS>:41862 after 3 ms (0 ms spent in bootstraps)
2018-05-07 09:22:28 INFO  DiskBlockManager:54 - Created local directory at /tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/blockmgr-74879816-83af-4565-bac0-34a7367fbe7b
2018-05-07 09:22:28 INFO  MemoryStore:54 - MemoryStore started with capacity 10.5 GB
2018-05-07 09:22:28 INFO  CoarseGrainedExecutorBackend:54 - Connecting to driver: spark://CoarseGrainedScheduler@hpe01:41862
2018-05-07 09:22:28 INFO  CoarseGrainedExecutorBackend:54 - Successfully registered with driver
2018-05-07 09:22:28 INFO  Executor:54 - Starting executor ID 1 on host hpe02
2018-05-07 09:22:28 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40311.
2018-05-07 09:22:28 INFO  NettyBlockTransferService:54 - Server created on hpe02:40311
2018-05-07 09:22:28 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-05-07 09:22:28 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(1, hpe02, 40311, None)
2018-05-07 09:22:28 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(1, hpe02, 40311, None)
2018-05-07 09:22:28 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(1, hpe02, 40311, None)
2018-05-07 09:22:30 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 1
2018-05-07 09:22:30 INFO  Executor:54 - Running task 1.0 in stage 0.0 (TID 1)
2018-05-07 09:22:30 INFO  TorrentBroadcast:54 - Started reading broadcast variable 2
2018-05-07 09:22:30 INFO  TransportClientFactory:267 - Successfully created connection to hpe01/<IP_ADDRESS>:38930 after 4 ms (0 ms spent in bootstraps)
2018-05-07 09:22:30 INFO  MemoryStore:54 - Block broadcast_2_piece0 stored as bytes in memory (estimated size 10.1 KB, free 10.5 GB)
2018-05-07 09:22:30 INFO  TorrentBroadcast:54 - Reading broadcast variable 2 took 173 ms
2018-05-07 09:22:30 INFO  MemoryStore:54 - Block broadcast_2 stored as values in memory (estimated size 14.3 KB, free 10.5 GB)
2018-05-07 09:22:36 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 3
2018-05-07 09:22:36 INFO  Executor:54 - Running task 0.0 in stage 1.0 (TID 3)
2018-05-07 09:22:36 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 6
2018-05-07 09:22:36 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 9
2018-05-07 09:22:36 INFO  Executor:54 - Running task 2.0 in stage 1.0 (TID 6)
2018-05-07 09:22:36 INFO  Executor:54 - Running task 5.0 in stage 1.0 (TID 9)
2018-05-07 09:22:36 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 11
2018-05-07 09:22:36 INFO  Executor:54 - Running task 8.0 in stage 1.0 (TID 11)
2018-05-07 09:22:36 INFO  TorrentBroadcast:54 - Started reading broadcast variable 3
2018-05-07 09:22:36 INFO  MemoryStore:54 - Block broadcast_3_piece0 stored as bytes in memory (estimated size 8.0 KB, free 10.5 GB)
2018-05-07 09:22:36 INFO  TorrentBroadcast:54 - Reading broadcast variable 3 took 29 ms
2018-05-07 09:22:36 INFO  MemoryStore:54 - Block broadcast_3 stored as values in memory (estimated size 14.9 KB, free 10.5 GB)
2018-05-07 09:22:36 INFO  PythonRunner:54 - Times: total = 5345, boot = 796, init = 94, finish = 4455
2018-05-07 09:22:36 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00005:0+11173834
2018-05-07 09:22:36 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00002:0+11214784
2018-05-07 09:22:36 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00000:0+9338236
2018-05-07 09:22:36 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00008:0+11194141
2018-05-07 09:22:36 INFO  TorrentBroadcast:54 - Started reading broadcast variable 0
2018-05-07 09:22:36 INFO  Executor:54 - Finished task 1.0 in stage 0.0 (TID 1). 1310 bytes result sent to driver
2018-05-07 09:22:36 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.2 KB, free 10.5 GB)
2018-05-07 09:22:36 INFO  TorrentBroadcast:54 - Reading broadcast variable 0 took 51 ms
2018-05-07 09:22:36 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 324.4 KB, free 10.5 GB)
2018-05-07 09:22:37 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00005:0+245760
2018-05-07 09:22:37 INFO  TorrentBroadcast:54 - Started reading broadcast variable 1
2018-05-07 09:22:37 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00008:0+245760
2018-05-07 09:22:37 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00000:0+204800
2018-05-07 09:22:37 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00002:0+245760
2018-05-07 09:22:37 INFO  MemoryStore:54 - Block broadcast_1_piece0 stored as bytes in memory (estimated size 23.2 KB, free 10.5 GB)
2018-05-07 09:22:37 INFO  TorrentBroadcast:54 - Reading broadcast variable 1 took 29 ms
2018-05-07 09:22:37 INFO  MemoryStore:54 - Block broadcast_1 stored as values in memory (estimated size 324.4 KB, free 10.5 GB)
2018-05-07 09:22:38 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 14
2018-05-07 09:22:38 INFO  Executor:54 - Running task 6.1 in stage 1.0 (TID 14)
2018-05-07 09:22:38 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00006:0+11214285
2018-05-07 09:22:38 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00006:0+245760
2018-05-07 09:22:40 INFO  PythonRunner:54 - Times: total = 2566, boot = 26, init = 180, finish = 2360
2018-05-07 09:22:40 INFO  PythonRunner:54 - Times: total = 302, boot = 7, init = 182, finish = 113
2018-05-07 09:22:40 INFO  PythonRunner:54 - Times: total = 2951, boot = 19, init = 233, finish = 2699
2018-05-07 09:22:40 INFO  PythonRunner:54 - Times: total = 311, boot = 17, init = 173, finish = 121
2018-05-07 09:22:40 INFO  PythonRunner:54 - Times: total = 3000, boot = 9, init = 240, finish = 2751
2018-05-07 09:22:40 INFO  PythonRunner:54 - Times: total = 337, boot = 6, init = 188, finish = 143
2018-05-07 09:22:40 INFO  PythonRunner:54 - Times: total = 3020, boot = 32, init = 214, finish = 2774
2018-05-07 09:22:40 INFO  PythonRunner:54 - Times: total = 285, boot = 30, init = 150, finish = 105
2018-05-07 09:22:41 INFO  PythonRunner:54 - Times: total = 2746, boot = 8, init = 8, finish = 2730
2018-05-07 09:22:41 INFO  PythonRunner:54 - Times: total = 175, boot = 9, init = 40, finish = 126
End of LogType:stdout.This log file belongs to a running container (container_1525333084924_0016_01_000002) and so may not be complete.
***********************************************************************

Container: container_1525333084924_0016_01_000002 on hpe02:36742
LogAggregationType: LOCAL
================================================================
LogType:stderr
LogLastModifiedTime:星期一 五月 07 09:22:39 +0800 2018
LogLength:4793
LogContents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop-hpe/nm-local-dir/filecache/88/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hpe/hadoop-2.9.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2018-05-07 09:22:35,356 INFO (MainThread-11016) connected to server at ('<IP_ADDRESS>', 42715)
2018-05-07 09:22:35,358 INFO (MainThread-11016) TFSparkNode.reserve: {'executor_id': 1, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 0, 'port': 34589, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-_nhkb93k/listener-jcb_p0ze', 'authkey': b'\x98JB\x83M\x91G\x8b\xb54Or\xa4\x85=X'}
2018-05-07 09:22:36,361 INFO (MainThread-11016) node: {'executor_id': 0, 'host': '<IP_ADDRESS>', 'job_name': 'ps', 'task_index': 0, 'port': 40395, 'tb_pid': 0, 'tb_port': 0, 'addr': ('<IP_ADDRESS>', 42995), 'authkey': b'z\xf6x\xc9\xfe"LM\x87_i3_\xebh\xef'}
2018-05-07 09:22:36,361 INFO (MainThread-11016) node: {'executor_id': 1, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 0, 'port': 34589, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-_nhkb93k/listener-jcb_p0ze', 'authkey': b'\x98JB\x83M\x91G\x8b\xb54Or\xa4\x85=X'}
2018-05-07 09:22:36,362 INFO (MainThread-11016) node: {'executor_id': 2, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 1, 'port': 45638, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-tm696uno/listener-kmgfbw71', 'authkey': b'\x04\xc69\xdd&aG\x0e\x86\xaa\xfb\x16z\\\xb9\xc7'}
2018-05-07 09:22:36,383 INFO (MainThread-11016) Starting TensorFlow worker:0 as worker on cluster node 1 on background process
2018-05-07 09:22:36,405 INFO (MainThread-11092) 1: ======== worker:0 ========
2018-05-07 09:22:36,406 INFO (MainThread-11092) 1: Cluster spec: {'ps': ['<IP_ADDRESS>:40395'], 'worker': ['<IP_ADDRESS>:34589', '<IP_ADDRESS>:45638']}
2018-05-07 09:22:36,406 INFO (MainThread-11092) 1: Using CPU
2018-05-07 09:22:36.410466: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
2018-05-07 09:22:36.418100: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> <IP_ADDRESS>:40395}
2018-05-07 09:22:36.418166: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> localhost:34589, 1 -> <IP_ADDRESS>:45638}
2018-05-07 09:22:36.421670: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:332] Started server with target: grpc://localhost:34589
tensorflow model path: hdfs://hpe01:9000/user/hpe/mnist_model
2018-05-07 09:22:38,031 INFO (MainThread-11220) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=1, state='running'
2018-05-07 09:22:38,041 INFO (MainThread-11217) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=1, state='running'
2018-05-07 09:22:38,058 INFO (MainThread-11220) mgr.state='running'
2018-05-07 09:22:38,059 INFO (MainThread-11220) Feeding partition <itertools.chain object at 0x7fc487a68358> into input queue <multiprocessing.queues.JoinableQueue object at 0x7fc487a685f8>
2018-05-07 09:22:38,064 INFO (MainThread-11213) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=1, state='running'
2018-05-07 09:22:38,068 INFO (MainThread-11227) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=1, state='running'
2018-05-07 09:22:38,081 INFO (MainThread-11217) mgr.state='running'
2018-05-07 09:22:38,082 INFO (MainThread-11217) Feeding partition <itertools.chain object at 0x7fc487a68358> into input queue <multiprocessing.queues.JoinableQueue object at 0x7fc487a685f8>
2018-05-07 09:22:38,113 INFO (MainThread-11227) mgr.state='running'
2018-05-07 09:22:38,114 INFO (MainThread-11227) Feeding partition <itertools.chain object at 0x7fc487a68358> into input queue <multiprocessing.queues.JoinableQueue object at 0x7fc487a685f8>
2018-05-07 09:22:38,117 INFO (MainThread-11213) mgr.state='running'
2018-05-07 09:22:38,118 INFO (MainThread-11213) Feeding partition <itertools.chain object at 0x7fc487a68358> into input queue <multiprocessing.queues.JoinableQueue object at 0x7fc487a685f8>
2018-05-07 09:22:39,154 INFO (MainThread-11312) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=1, state='running'
2018-05-07 09:22:39,252 INFO (MainThread-11312) mgr.state='running'
2018-05-07 09:22:39,254 INFO (MainThread-11312) Feeding partition <itertools.chain object at 0x7fc487a68358> into input queue <multiprocessing.queues.JoinableQueue object at 0x7fc487a685f8>
End of LogType:stderr.This log file belongs to a running container (container_1525333084924_0016_01_000002) and so may not be complete.
***********************************************************************

Container: container_1525333084924_0016_01_000002 on hpe02:36742
LogAggregationType: LOCAL
================================================================
LogType:prelaunch.out
LogLastModifiedTime:星期一 五月 07 09:22:25 +0800 2018
LogLength:70
LogContents:
Setting up env variables
Setting up job resources
Launching container
End of LogType:prelaunch.out.This log file belongs to a running container (container_1525333084924_0016_01_000002) and so may not be complete.
******************************************************************************

End of LogType:prelaunch.err.This log file belongs to a running container (container_1525333084924_0016_01_000003) and so may not be complete.
******************************************************************************

Container: container_1525333084924_0016_01_000003 on hpe01:33415
LogAggregationType: LOCAL
================================================================
LogType:stdout
LogLastModifiedTime:星期一 五月 07 09:22:38 +0800 2018
LogLength:13592
LogContents:
2018-05-07 09:22:26 INFO  CoarseGrainedExecutorBackend:2608 - Started daemon with process name: 27768@hpe01
2018-05-07 09:22:26 INFO  SignalUtils:54 - Registered signal handler for TERM
2018-05-07 09:22:26 INFO  SignalUtils:54 - Registered signal handler for HUP
2018-05-07 09:22:26 INFO  SignalUtils:54 - Registered signal handler for INT
2018-05-07 09:22:27 INFO  SecurityManager:54 - Changing view acls to: hpe
2018-05-07 09:22:27 INFO  SecurityManager:54 - Changing modify acls to: hpe
2018-05-07 09:22:27 INFO  SecurityManager:54 - Changing view acls groups to:
2018-05-07 09:22:27 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-05-07 09:22:27 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hpe); groups with view permissions: Set(); users  with modify permissions: Set(hpe); groups with modify permissions: Set()
2018-05-07 09:22:27 INFO  TransportClientFactory:267 - Successfully created connection to hpe01/<IP_ADDRESS>:41862 after 166 ms (0 ms spent in bootstraps)
2018-05-07 09:22:28 INFO  SecurityManager:54 - Changing view acls to: hpe
2018-05-07 09:22:28 INFO  SecurityManager:54 - Changing modify acls to: hpe
2018-05-07 09:22:28 INFO  SecurityManager:54 - Changing view acls groups to:
2018-05-07 09:22:28 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-05-07 09:22:28 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hpe); groups with view permissions: Set(); users  with modify permissions: Set(hpe); groups with modify permissions: Set()
2018-05-07 09:22:28 INFO  TransportClientFactory:267 - Successfully created connection to hpe01/<IP_ADDRESS>:41862 after 3 ms (0 ms spent in bootstraps)
2018-05-07 09:22:28 INFO  DiskBlockManager:54 - Created local directory at /tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/blockmgr-1f36d212-3c46-453e-bcfb-d6c6b45aec3f
2018-05-07 09:22:28 INFO  MemoryStore:54 - MemoryStore started with capacity 10.5 GB
2018-05-07 09:22:28 INFO  CoarseGrainedExecutorBackend:54 - Connecting to driver: spark://CoarseGrainedScheduler@hpe01:41862
2018-05-07 09:22:28 INFO  CoarseGrainedExecutorBackend:54 - Successfully registered with driver
2018-05-07 09:22:28 INFO  Executor:54 - Starting executor ID 2 on host hpe01
2018-05-07 09:22:28 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33489.
2018-05-07 09:22:28 INFO  NettyBlockTransferService:54 - Server created on hpe01:33489
2018-05-07 09:22:28 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-05-07 09:22:28 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(2, hpe01, 33489, None)
2018-05-07 09:22:28 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(2, hpe01, 33489, None)
2018-05-07 09:22:28 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(2, hpe01, 33489, None)
2018-05-07 09:22:30 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 0
2018-05-07 09:22:30 INFO  Executor:54 - Running task 0.0 in stage 0.0 (TID 0)
2018-05-07 09:22:30 INFO  TorrentBroadcast:54 - Started reading broadcast variable 2
2018-05-07 09:22:30 INFO  TransportClientFactory:267 - Successfully created connection to hpe01/<IP_ADDRESS>:38930 after 3 ms (0 ms spent in bootstraps)
2018-05-07 09:22:30 INFO  MemoryStore:54 - Block broadcast_2_piece0 stored as bytes in memory (estimated size 10.1 KB, free 10.5 GB)
2018-05-07 09:22:30 INFO  TorrentBroadcast:54 - Reading broadcast variable 2 took 150 ms
2018-05-07 09:22:31 INFO  MemoryStore:54 - Block broadcast_2 stored as values in memory (estimated size 14.3 KB, free 10.5 GB)
2018-05-07 09:22:36 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 5
2018-05-07 09:22:36 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 8
2018-05-07 09:22:36 INFO  Executor:54 - Running task 3.0 in stage 1.0 (TID 5)
2018-05-07 09:22:36 INFO  Executor:54 - Running task 6.0 in stage 1.0 (TID 8)
2018-05-07 09:22:36 INFO  TorrentBroadcast:54 - Started reading broadcast variable 3
2018-05-07 09:22:36 INFO  MemoryStore:54 - Block broadcast_3_piece0 stored as bytes in memory (estimated size 8.0 KB, free 10.5 GB)
2018-05-07 09:22:36 INFO  TorrentBroadcast:54 - Reading broadcast variable 3 took 28 ms
2018-05-07 09:22:36 INFO  MemoryStore:54 - Block broadcast_3 stored as values in memory (estimated size 14.9 KB, free 10.5 GB)
2018-05-07 09:22:36 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00003:0+11226100
2018-05-07 09:22:36 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00006:0+11214285
2018-05-07 09:22:36 INFO  TorrentBroadcast:54 - Started reading broadcast variable 0
2018-05-07 09:22:36 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.2 KB, free 10.5 GB)
2018-05-07 09:22:36 INFO  TorrentBroadcast:54 - Reading broadcast variable 0 took 26 ms
2018-05-07 09:22:36 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 324.4 KB, free 10.5 GB)
2018-05-07 09:22:37 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00003:0+245760
2018-05-07 09:22:37 INFO  TorrentBroadcast:54 - Started reading broadcast variable 1
2018-05-07 09:22:37 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00006:0+245760
2018-05-07 09:22:37 INFO  MemoryStore:54 - Block broadcast_1_piece0 stored as bytes in memory (estimated size 23.2 KB, free 10.5 GB)
2018-05-07 09:22:37 INFO  TorrentBroadcast:54 - Reading broadcast variable 1 took 27 ms
2018-05-07 09:22:37 INFO  MemoryStore:54 - Block broadcast_1 stored as values in memory (estimated size 324.4 KB, free 10.5 GB)
2018-05-07 09:22:38 ERROR Executor:91 - Exception in task 6.0 in stage 1.0 (TID 8)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000003/pyspark.zip/pyspark/worker.py", line 229, in main
    process()
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000003/pyspark.zip/pyspark/worker.py", line 224, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/pyspark.zip/pyspark/rdd.py", line 2438, in pipeline_func
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/pyspark.zip/pyspark/rdd.py", line 2438, in pipeline_func
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/pyspark.zip/pyspark/rdd.py", line 2438, in pipeline_func
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/pyspark.zip/pyspark/rdd.py", line 362, in func
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/pyspark.zip/pyspark/rdd.py", line 809, in func
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 394, in _train
AttributeError: 'AutoProxy[get_queue]' object has no attribute 'put'

	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:298)
	at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:438)
	at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:421)
	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:252)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
	at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
	at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
	at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
	at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:939)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:939)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2018-05-07 09:22:38 ERROR Executor:91 - Exception in task 3.0 in stage 1.0 (TID 5)
org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000003/pyspark.zip/pyspark/worker.py", line 229, in main
    process()
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000003/pyspark.zip/pyspark/worker.py", line 224, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/pyspark.zip/pyspark/rdd.py", line 2438, in pipeline_func
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/pyspark.zip/pyspark/rdd.py", line 2438, in pipeline_func
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/pyspark.zip/pyspark/rdd.py", line 2438, in pipeline_func
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/pyspark.zip/pyspark/rdd.py", line 362, in func
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/pyspark.zip/pyspark/rdd.py", line 809, in func
  File "/tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/container_1525333084924_0016_01_000001/tfspark.zip/tensorflowonspark/TFSparkNode.py", line 394, in _train
AttributeError: 'AutoProxy[get_queue]' object has no attribute 'put'

	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:298)
	at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:438)
	at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:421)
	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:252)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$class.foreach(Iterator.scala:893)
	at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
	at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
	at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
	at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
	at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
	at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
	at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:939)
	at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:939)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
End of LogType:stdout.This log file belongs to a running container (container_1525333084924_0016_01_000003) and so may not be complete.
***********************************************************************

Container: container_1525333084924_0016_01_000003 on hpe01:33415
LogAggregationType: LOCAL
================================================================
LogType:stderr
LogLastModifiedTime:星期一 五月 07 09:22:42 +0800 2018
LogLength:3433
LogContents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop-hpe/nm-local-dir/filecache/87/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hpe/hadoop-2.9.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2018-05-07 09:22:35,033 INFO (MainThread-27925) connected to server at ('<IP_ADDRESS>', 42715)
2018-05-07 09:22:35,036 INFO (MainThread-27925) TFSparkNode.reserve: {'executor_id': 0, 'host': '<IP_ADDRESS>', 'job_name': 'ps', 'task_index': 0, 'port': 40395, 'tb_pid': 0, 'tb_port': 0, 'addr': ('<IP_ADDRESS>', 42995), 'authkey': b'z\xf6x\xc9\xfe"LM\x87_i3_\xebh\xef'}
2018-05-07 09:22:37,078 INFO (MainThread-27925) node: {'executor_id': 0, 'host': '<IP_ADDRESS>', 'job_name': 'ps', 'task_index': 0, 'port': 40395, 'tb_pid': 0, 'tb_port': 0, 'addr': ('<IP_ADDRESS>', 42995), 'authkey': b'z\xf6x\xc9\xfe"LM\x87_i3_\xebh\xef'}
2018-05-07 09:22:37,078 INFO (MainThread-27925) node: {'executor_id': 1, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 0, 'port': 34589, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-_nhkb93k/listener-jcb_p0ze', 'authkey': b'\x98JB\x83M\x91G\x8b\xb54Or\xa4\x85=X'}
2018-05-07 09:22:37,078 INFO (MainThread-27925) node: {'executor_id': 2, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 1, 'port': 45638, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-tm696uno/listener-kmgfbw71', 'authkey': b'\x04\xc69\xdd&aG\x0e\x86\xaa\xfb\x16z\\\xb9\xc7'}
2018-05-07 09:22:37,098 INFO (MainThread-27925) Starting TensorFlow ps:0 as ps on cluster node 0 on background process
2018-05-07 09:22:38,151 INFO (MainThread-28031) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=0, state='running'
2018-05-07 09:22:38,152 INFO (MainThread-28034) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=0, state='running'
2018-05-07 09:22:38,619 INFO (MainThread-28031) mgr.state='running'
2018-05-07 09:22:38,619 INFO (MainThread-28034) mgr.state='running'
2018-05-07 09:22:38,619 INFO (MainThread-28031) Feeding partition <itertools.chain object at 0x7f0e05cd2860> into input queue None
2018-05-07 09:22:38,620 INFO (MainThread-28034) Feeding partition <itertools.chain object at 0x7f0e05cd2860> into input queue None
2018-05-07 09:22:42,276 INFO (MainThread-28001) 0: ======== ps:0 ========
2018-05-07 09:22:42,277 INFO (MainThread-28001) 0: Cluster spec: {'ps': ['<IP_ADDRESS>:40395'], 'worker': ['<IP_ADDRESS>:34589', '<IP_ADDRESS>:45638']}
2018-05-07 09:22:42,277 INFO (MainThread-28001) 0: Using CPU
2018-05-07 09:22:42.280937: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
2018-05-07 09:22:42.290145: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> localhost:40395}
2018-05-07 09:22:42.290232: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> <IP_ADDRESS>:34589, 1 -> <IP_ADDRESS>:45638}
2018-05-07 09:22:42.295658: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:333] Started server with target: grpc://localhost:40395
End of LogType:stderr.This log file belongs to a running container (container_1525333084924_0016_01_000003) and so may not be complete.
***********************************************************************

Container: container_1525333084924_0016_01_000003 on hpe01:33415
LogAggregationType: LOCAL
================================================================
LogType:prelaunch.out
LogLastModifiedTime:星期一 五月 07 09:22:25 +0800 2018
LogLength:70
LogContents:
Setting up env variables
Setting up job resources
Launching container
End of LogType:prelaunch.out.This log file belongs to a running container (container_1525333084924_0016_01_000003) and so may not be complete.
******************************************************************************

End of LogType:prelaunch.err.This log file belongs to a running container (container_1525333084924_0016_01_000004) and so may not be complete.
******************************************************************************

Container: container_1525333084924_0016_01_000004 on hpe03:35000
LogAggregationType: LOCAL
================================================================
LogType:stdout
LogLastModifiedTime:星期一 五月 07 09:23:03 +0800 2018
LogLength:8508
LogContents:
2018-05-07 09:22:48 INFO  CoarseGrainedExecutorBackend:2608 - Started daemon with process name: 10107@hpe03
2018-05-07 09:22:48 INFO  SignalUtils:54 - Registered signal handler for TERM
2018-05-07 09:22:48 INFO  SignalUtils:54 - Registered signal handler for HUP
2018-05-07 09:22:48 INFO  SignalUtils:54 - Registered signal handler for INT
2018-05-07 09:22:49 INFO  SecurityManager:54 - Changing view acls to: hpe
2018-05-07 09:22:49 INFO  SecurityManager:54 - Changing modify acls to: hpe
2018-05-07 09:22:49 INFO  SecurityManager:54 - Changing view acls groups to:
2018-05-07 09:22:49 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-05-07 09:22:49 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hpe); groups with view permissions: Set(); users  with modify permissions: Set(hpe); groups with modify permissions: Set()
2018-05-07 09:22:50 INFO  TransportClientFactory:267 - Successfully created connection to hpe01/<IP_ADDRESS>:41862 after 121 ms (0 ms spent in bootstraps)
2018-05-07 09:22:50 INFO  SecurityManager:54 - Changing view acls to: hpe
2018-05-07 09:22:50 INFO  SecurityManager:54 - Changing modify acls to: hpe
2018-05-07 09:22:50 INFO  SecurityManager:54 - Changing view acls groups to:
2018-05-07 09:22:50 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-05-07 09:22:50 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hpe); groups with view permissions: Set(); users  with modify permissions: Set(hpe); groups with modify permissions: Set()
2018-05-07 09:22:50 INFO  TransportClientFactory:267 - Successfully created connection to hpe01/<IP_ADDRESS>:41862 after 3 ms (0 ms spent in bootstraps)
2018-05-07 09:22:50 INFO  DiskBlockManager:54 - Created local directory at /tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/blockmgr-7bf15713-3c5b-4958-b839-ab72638a9a53
2018-05-07 09:22:50 INFO  MemoryStore:54 - MemoryStore started with capacity 10.5 GB
2018-05-07 09:22:50 INFO  CoarseGrainedExecutorBackend:54 - Connecting to driver: spark://CoarseGrainedScheduler@hpe01:41862
2018-05-07 09:22:50 INFO  CoarseGrainedExecutorBackend:54 - Successfully registered with driver
2018-05-07 09:22:50 INFO  Executor:54 - Starting executor ID 3 on host hpe03
2018-05-07 09:22:50 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38332.
2018-05-07 09:22:50 INFO  NettyBlockTransferService:54 - Server created on hpe03:38332
2018-05-07 09:22:50 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-05-07 09:22:50 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(3, hpe03, 38332, None)
2018-05-07 09:22:51 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(3, hpe03, 38332, None)
2018-05-07 09:22:51 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(3, hpe03, 38332, None)
2018-05-07 09:22:52 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 2
2018-05-07 09:22:52 INFO  Executor:54 - Running task 2.0 in stage 0.0 (TID 2)
2018-05-07 09:22:52 INFO  TorrentBroadcast:54 - Started reading broadcast variable 2
2018-05-07 09:22:52 INFO  TransportClientFactory:267 - Successfully created connection to hpe01/<IP_ADDRESS>:38930 after 4 ms (0 ms spent in bootstraps)
2018-05-07 09:22:52 INFO  MemoryStore:54 - Block broadcast_2_piece0 stored as bytes in memory (estimated size 10.1 KB, free 10.5 GB)
2018-05-07 09:22:52 INFO  TorrentBroadcast:54 - Reading broadcast variable 2 took 173 ms
2018-05-07 09:22:52 INFO  MemoryStore:54 - Block broadcast_2 stored as values in memory (estimated size 14.3 KB, free 10.5 GB)
2018-05-07 09:22:57 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 4
2018-05-07 09:22:57 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 7
2018-05-07 09:22:57 INFO  Executor:54 - Running task 1.0 in stage 1.0 (TID 4)
2018-05-07 09:22:57 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 10
2018-05-07 09:22:57 INFO  Executor:54 - Running task 4.0 in stage 1.0 (TID 7)
2018-05-07 09:22:57 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 12
2018-05-07 09:22:57 INFO  Executor:54 - Running task 7.0 in stage 1.0 (TID 10)
2018-05-07 09:22:57 INFO  Executor:54 - Running task 9.0 in stage 1.0 (TID 12)
2018-05-07 09:22:57 INFO  TorrentBroadcast:54 - Started reading broadcast variable 3
2018-05-07 09:22:57 INFO  TransportClientFactory:267 - Successfully created connection to hpe02/<IP_ADDRESS>:40311 after 5 ms (0 ms spent in bootstraps)
2018-05-07 09:22:58 INFO  MemoryStore:54 - Block broadcast_3_piece0 stored as bytes in memory (estimated size 8.0 KB, free 10.5 GB)
2018-05-07 09:22:58 INFO  TorrentBroadcast:54 - Reading broadcast variable 3 took 136 ms
2018-05-07 09:22:58 INFO  MemoryStore:54 - Block broadcast_3 stored as values in memory (estimated size 14.9 KB, free 10.5 GB)
2018-05-07 09:22:58 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00007:0+11201024
2018-05-07 09:22:58 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00001:0+11231804
2018-05-07 09:22:58 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00009:0+10449019
2018-05-07 09:22:58 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00004:0+11212767
2018-05-07 09:22:58 INFO  TorrentBroadcast:54 - Started reading broadcast variable 0
2018-05-07 09:22:58 INFO  MemoryStore:54 - Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.2 KB, free 10.5 GB)
2018-05-07 09:22:58 INFO  TorrentBroadcast:54 - Reading broadcast variable 0 took 27 ms
2018-05-07 09:22:58 INFO  MemoryStore:54 - Block broadcast_0 stored as values in memory (estimated size 324.4 KB, free 10.5 GB)
2018-05-07 09:22:58 INFO  PythonRunner:54 - Times: total = 6123, boot = 743, init = 80, finish = 5300
2018-05-07 09:22:58 INFO  Executor:54 - Finished task 2.0 in stage 0.0 (TID 2). 1353 bytes result sent to driver
2018-05-07 09:22:59 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00007:0+245760
2018-05-07 09:22:59 INFO  TorrentBroadcast:54 - Started reading broadcast variable 1
2018-05-07 09:22:59 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00009:0+229120
2018-05-07 09:22:59 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00001:0+245760
2018-05-07 09:22:59 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00004:0+245760
2018-05-07 09:22:59 INFO  MemoryStore:54 - Block broadcast_1_piece0 stored as bytes in memory (estimated size 23.2 KB, free 10.5 GB)
2018-05-07 09:22:59 INFO  TorrentBroadcast:54 - Reading broadcast variable 1 took 35 ms
2018-05-07 09:22:59 INFO  MemoryStore:54 - Block broadcast_1 stored as values in memory (estimated size 324.4 KB, free 10.5 GB)
2018-05-07 09:23:00 INFO  CoarseGrainedExecutorBackend:54 - Got assigned task 13
2018-05-07 09:23:00 INFO  Executor:54 - Running task 3.1 in stage 1.0 (TID 13)
2018-05-07 09:23:00 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/images/part-00003:0+11226100
2018-05-07 09:23:00 INFO  HadoopRDD:54 - Input split: hdfs://hpe01:9000/data/mnist/csv/train/labels/part-00003:0+245760
2018-05-07 09:23:02 INFO  PythonRunner:54 - Times: total = 2715, boot = 17, init = 160, finish = 2538
2018-05-07 09:23:02 INFO  PythonRunner:54 - Times: total = 248, boot = 9, init = 130, finish = 109
2018-05-07 09:23:02 INFO  PythonRunner:54 - Times: total = 2801, boot = 10, init = 176, finish = 2615
2018-05-07 09:23:02 INFO  PythonRunner:54 - Times: total = 269, boot = 7, init = 141, finish = 121
2018-05-07 09:23:02 INFO  PythonRunner:54 - Times: total = 2847, boot = 23, init = 143, finish = 2681
2018-05-07 09:23:02 INFO  PythonRunner:54 - Times: total = 263, boot = 18, init = 121, finish = 124
2018-05-07 09:23:02 INFO  PythonRunner:54 - Times: total = 2862, boot = 29, init = 181, finish = 2652
2018-05-07 09:23:02 INFO  PythonRunner:54 - Times: total = 260, boot = 16, init = 114, finish = 130
2018-05-07 09:23:03 INFO  PythonRunner:54 - Times: total = 2435, boot = 9, init = 8, finish = 2418
2018-05-07 09:23:03 INFO  PythonRunner:54 - Times: total = 161, boot = 10, init = 10, finish = 141
End of LogType:stdout.This log file belongs to a running container (container_1525333084924_0016_01_000004) and so may not be complete.
***********************************************************************

Container: container_1525333084924_0016_01_000004 on hpe03:35000
LogAggregationType: LOCAL
================================================================
LogType:stderr
LogLastModifiedTime:星期一 五月 07 09:30:41 +0800 2018
LogLength:13512
LogContents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop-hpe/nm-local-dir/filecache/87/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hpe/hadoop-2.9.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2018-05-07 09:22:56,709 INFO (MainThread-10250) connected to server at ('<IP_ADDRESS>', 42715)
2018-05-07 09:22:56,711 INFO (MainThread-10250) TFSparkNode.reserve: {'executor_id': 2, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 1, 'port': 45638, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-tm696uno/listener-kmgfbw71', 'authkey': b'\x04\xc69\xdd&aG\x0e\x86\xaa\xfb\x16z\\\xb9\xc7'}
2018-05-07 09:22:58,793 INFO (MainThread-10250) node: {'executor_id': 0, 'host': '<IP_ADDRESS>', 'job_name': 'ps', 'task_index': 0, 'port': 40395, 'tb_pid': 0, 'tb_port': 0, 'addr': ('<IP_ADDRESS>', 42995), 'authkey': b'z\xf6x\xc9\xfe"LM\x87_i3_\xebh\xef'}
2018-05-07 09:22:58,793 INFO (MainThread-10250) node: {'executor_id': 1, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 0, 'port': 34589, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-_nhkb93k/listener-jcb_p0ze', 'authkey': b'\x98JB\x83M\x91G\x8b\xb54Or\xa4\x85=X'}
2018-05-07 09:22:58,793 INFO (MainThread-10250) node: {'executor_id': 2, 'host': '<IP_ADDRESS>', 'job_name': 'worker', 'task_index': 1, 'port': 45638, 'tb_pid': 0, 'tb_port': 0, 'addr': '/tmp/pymp-tm696uno/listener-kmgfbw71', 'authkey': b'\x04\xc69\xdd&aG\x0e\x86\xaa\xfb\x16z\\\xb9\xc7'}
2018-05-07 09:22:58,810 INFO (MainThread-10250) Starting TensorFlow worker:1 as worker on cluster node 2 on background process
2018-05-07 09:22:58,830 INFO (MainThread-10335) 2: ======== worker:1 ========
2018-05-07 09:22:58,831 INFO (MainThread-10335) 2: Cluster spec: {'ps': ['<IP_ADDRESS>:40395'], 'worker': ['<IP_ADDRESS>:34589', '<IP_ADDRESS>:45638']}
2018-05-07 09:22:58,831 INFO (MainThread-10335) 2: Using CPU
2018-05-07 09:22:58.833362: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
2018-05-07 09:22:58.839880: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> <IP_ADDRESS>:40395}
2018-05-07 09:22:58.839934: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> <IP_ADDRESS>:34589, 1 -> localhost:45638}
2018-05-07 09:22:58.843093: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:332] Started server with target: grpc://localhost:45638
tensorflow model path: hdfs://hpe01:9000/user/hpe/mnist_model
2018-05-07 09:22:59,378 INFO (MainThread-10335) Graph was finalized.
2018-05-07 09:22:59,703 INFO (MainThread-10463) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=2, state='running'
2018-05-07 09:22:59,710 INFO (MainThread-10452) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=2, state='running'
2018-05-07 09:22:59,724 INFO (MainThread-10469) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=2, state='running'
2018-05-07 09:22:59,731 INFO (MainThread-10466) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=2, state='running'
2018-05-07 09:22:59,745 INFO (MainThread-10463) mgr.state='running'
2018-05-07 09:22:59,746 INFO (MainThread-10463) Feeding partition <itertools.chain object at 0x7fb7d98ea358> into input queue <multiprocessing.queues.JoinableQueue object at 0x7fb7d98ea5f8>
2018-05-07 09:22:59,747 INFO (MainThread-10452) mgr.state='running'
2018-05-07 09:22:59,748 INFO (MainThread-10452) Feeding partition <itertools.chain object at 0x7fb7d98ea358> into input queue <multiprocessing.queues.JoinableQueue object at 0x7fb7d98ea5f8>
2018-05-07 09:22:59,764 INFO (MainThread-10469) mgr.state='running'
2018-05-07 09:22:59,765 INFO (MainThread-10469) Feeding partition <itertools.chain object at 0x7fb7d98ea358> into input queue <multiprocessing.queues.JoinableQueue object at 0x7fb7d98ea5f8>
2018-05-07 09:22:59,778 INFO (MainThread-10466) mgr.state='running'
2018-05-07 09:22:59,780 INFO (MainThread-10466) Feeding partition <itertools.chain object at 0x7fb7d98ea358> into input queue <multiprocessing.queues.JoinableQueue object at 0x7fb7d98ea5f8>
2018-05-07 09:23:00,786 INFO (MainThread-10553) Connected to TFSparkNode.mgr on <IP_ADDRESS>, executor=2, state='running'
2018-05-07 09:23:00,830 INFO (MainThread-10553) mgr.state='running'
2018-05-07 09:23:00,832 INFO (MainThread-10553) Feeding partition <itertools.chain object at 0x7fb7d98ea358> into input queue <multiprocessing.queues.JoinableQueue object at 0x7fb7d98ea5f8>
2018-05-07 09:23:04.392866: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session c7cea3785511f7ee with config:
2018-05-07 09:23:04,513 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:23:34.786975: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session e6f1c776525e4380 with config:
2018-05-07 09:23:34,843 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:24:04.881590: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session fb79d5bb5075c753 with config:
2018-05-07 09:24:04,935 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:24:35.165303: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session bcde1636c5140cc2 with config:
2018-05-07 09:24:35,218 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:25:05.968857: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session c538e315143d99c6 with config:
2018-05-07 09:25:06,021 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:25:36.028765: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session f56ff319b6085cc0 with config:
2018-05-07 09:25:36,077 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:26:07.098008: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session 3773bb2871300c8c with config:
2018-05-07 09:26:07,152 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:26:37.504513: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session 9eff0976308830d6 with config:
2018-05-07 09:26:37,555 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:27:07.655913: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session 7779a1b6b86bc58b with config:
2018-05-07 09:27:07,709 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:27:37.931022: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session dbadd3b944776685 with config:
2018-05-07 09:27:37,984 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:28:08.910850: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session a8e2f40c794437dc with config:
2018-05-07 09:28:08,960 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:28:39.453483: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session 9135cc65a1150abd with config:
2018-05-07 09:28:39,503 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:29:09.808256: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session 7397c55ff304c069 with config:
2018-05-07 09:29:09,862 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:29:40.492260: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session a9557c044290e432 with config:
2018-05-07 09:29:40,541 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:30:10.867679: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session 863652ccb1f92ec3 with config:
2018-05-07 09:30:10,917 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
2018-05-07 09:30:41.410422: I tensorflow/core/distributed_runtime/master_session.cc:1136] Start master session cb54d3f85fb4596e with config:
2018-05-07 09:30:41,464 INFO (MainThread-10335) Waiting for model to be ready.  Ready_for_local_init_op:  Variables not initialized: layer/hidden_layer/hid_w, layer/hidden_layer/hid_b, layer/softmax_layer/sm_w, layer/softmax_layer/sm_b, global_step, layer/hidden_layer/hid_w/Adagrad, layer/hidden_layer/hid_b/Adagrad, layer/softmax_layer/sm_w/Adagrad, layer/softmax_layer/sm_b/Adagrad, ready: None
End of LogType:stderr.This log file belongs to a running container (container_1525333084924_0016_01_000004) and so may not be complete.
***********************************************************************

Container: container_1525333084924_0016_01_000004 on hpe03:35000
LogAggregationType: LOCAL
================================================================
LogType:prelaunch.out
LogLastModifiedTime:星期一 五月 07 09:22:47 +0800 2018
LogLength:70
LogContents:
Setting up env variables
Setting up job resources
Launching container
End of LogType:prelaunch.out.This log file belongs to a running container (container_1525333084924_0016_01_000004) and so may not be complete.
******************************************************************************

End of LogType:prelaunch.err.This log file belongs to a running container (container_1525333084924_0016_01_000001) and so may not be complete.
******************************************************************************

Container: container_1525333084924_0016_01_000001 on hpe01:33415
LogAggregationType: LOCAL
================================================================
LogType:stdout
LogLastModifiedTime:星期一 五月 07 09:22:38 +0800 2018
LogLength:79227
LogContents:
2018-05-07 09:22:16 INFO  SignalUtils:54 - Registered signal handler for TERM
2018-05-07 09:22:16 INFO  SignalUtils:54 - Registered signal handler for HUP
2018-05-07 09:22:16 INFO  SignalUtils:54 - Registered signal handler for INT
2018-05-07 09:22:17 INFO  SecurityManager:54 - Changing view acls to: hpe
2018-05-07 09:22:17 INFO  SecurityManager:54 - Changing modify acls to: hpe
2018-05-07 09:22:17 INFO  SecurityManager:54 - Changing view acls groups to:
2018-05-07 09:22:17 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-05-07 09:22:17 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hpe); groups with view permissions: Set(); users  with modify permissions: Set(hpe); groups with modify permissions: Set()
2018-05-07 09:22:17 INFO  ApplicationMaster:54 - Preparing Local resources
2018-05-07 09:22:19 INFO  ApplicationMaster:54 - ApplicationAttemptId: appattempt_1525333084924_0016_000001
2018-05-07 09:22:19 INFO  ApplicationMaster:54 - Starting the user application in a separate Thread
2018-05-07 09:22:19 INFO  ApplicationMaster:54 - Waiting for spark context initialization...
2018-05-07 09:22:21 INFO  SparkContext:54 - Running Spark version 2.3.0
2018-05-07 09:22:21 INFO  SparkContext:54 - Submitted application: mnist_spark
2018-05-07 09:22:21 INFO  SecurityManager:54 - Changing view acls to: hpe
2018-05-07 09:22:21 INFO  SecurityManager:54 - Changing modify acls to: hpe
2018-05-07 09:22:21 INFO  SecurityManager:54 - Changing view acls groups to:
2018-05-07 09:22:21 INFO  SecurityManager:54 - Changing modify acls groups to:
2018-05-07 09:22:21 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hpe); groups with view permissions: Set(); users  with modify permissions: Set(hpe); groups with modify permissions: Set()
2018-05-07 09:22:21 INFO  Utils:54 - Successfully started service 'sparkDriver' on port 41862.
2018-05-07 09:22:21 INFO  SparkEnv:54 - Registering MapOutputTracker
2018-05-07 09:22:21 INFO  SparkEnv:54 - Registering BlockManagerMaster
2018-05-07 09:22:21 INFO  BlockManagerMasterEndpoint:54 - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
2018-05-07 09:22:21 INFO  BlockManagerMasterEndpoint:54 - BlockManagerMasterEndpoint up
2018-05-07 09:22:21 INFO  DiskBlockManager:54 - Created local directory at /tmp/hadoop-hpe/nm-local-dir/usercache/hpe/appcache/application_1525333084924_0016/blockmgr-820f86be-5e2f-4a4d-a753-c18c4ab60867
2018-05-07 09:22:21 INFO  MemoryStore:54 - MemoryStore started with capacity 366.3 MB
2018-05-07 09:22:21 INFO  SparkEnv:54 - Registering OutputCommitCoordinator
2018-05-07 09:22:21 INFO  log:192 - Logging initialized @6393ms
2018-05-07 09:22:21 INFO  JettyUtils:54 - Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
2018-05-07 09:22:21 INFO  Server:346 - jetty-9.3.z-SNAPSHOT
2018-05-07 09:22:21 INFO  Server:414 - Started @6524ms
2018-05-07 09:22:21 INFO  AbstractConnector:278 - Started ServerConnector@4ce4051c{HTTP/1.1,[http/1.1]}{<IP_ADDRESS>:45576}
2018-05-07 09:22:21 INFO  Utils:54 - Successfully started service 'SparkUI' on port 45576.
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@37055cf6{/jobs,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@149026db{/jobs/json,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4d05ec8a{/jobs/job,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@40c1b1aa{/jobs/job/json,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@55f2f197{/stages,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3771261f{/stages/json,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@eccfa76{/stages/stage,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3f8ee6ac{/stages/stage/json,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@56f8b461{/stages/pool,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7fab322c{/stages/pool/json,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@751af76b{/storage,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3de9c4e2{/storage/json,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2e212482{/storage/rdd,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@8bd5938{/storage/rdd/json,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@2abe56eb{/environment,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5903598a{/environment/json,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7ea1731c{/executors,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@1c379169{/executors/json,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@4466850a{/executors/threadDump,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@76a5b05{/executors/threadDump/json,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7226f167{/static,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@6f7b00d7{/,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@3b9b078e{/api,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@5aa0cd22{/jobs/job/kill,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@27ed3bbb{/stages/stage/kill,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  SparkUI:54 - Bound SparkUI to <IP_ADDRESS>, and started at http://hpe01:45576
2018-05-07 09:22:22 INFO  YarnClusterScheduler:54 - Created YarnClusterScheduler
2018-05-07 09:22:22 INFO  SchedulerExtensionServices:54 - Starting Yarn extension services with app application_1525333084924_0016 and attemptId Some(appattempt_1525333084924_0016_000001)
2018-05-07 09:22:22 INFO  Utils:54 - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38930.
2018-05-07 09:22:22 INFO  NettyBlockTransferService:54 - Server created on hpe01:38930
2018-05-07 09:22:22 INFO  BlockManager:54 - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
2018-05-07 09:22:22 INFO  BlockManagerMaster:54 - Registering BlockManager BlockManagerId(driver, hpe01, 38930, None)
2018-05-07 09:22:22 INFO  BlockManagerMasterEndpoint:54 - Registering block manager hpe01:38930 with 366.3 MB RAM, BlockManagerId(driver, hpe01, 38930, None)
2018-05-07 09:22:22 INFO  BlockManagerMaster:54 - Registered BlockManager BlockManagerId(driver, hpe01, 38930, None)
2018-05-07 09:22:22 INFO  BlockManager:54 - Initialized BlockManager: BlockManagerId(driver, hpe01, 38930, None)
2018-05-07 09:22:22 INFO  ContextHandler:781 - Started o.s.j.s.ServletContextHandler@7d2fd91a{/metrics/json,null,AVAILABLE,@Spark}
2018-05-07 09:22:22 INFO  ApplicationMaster:54 -
===============================================================================
YARN executor launch context:
  env:
    CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/*<CPS>$HADOOP_COMMON_HOME/share/hadoop/common/lib/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/*<CPS>$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/*<CPS>$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*<CPS>{{PWD}}/__spark_conf__/__hadoop_conf__
    SPARK_YARN_STAGING_DIR -> *********(redacted)
    SPARK_USER -> *********(redacted)
    PYTHONPATH -> {{PWD}}/__pyfiles__<CPS>{{PWD}}/pyspark.zip<CPS>{{PWD}}/py4j-0.10.6-src.zip<CPS>{{PWD}}/tfspark.zip

  command:
    {{JAVA_HOME}}/bin/java \
      -server \
      -Xmx20480m \
      -Djava.io.tmpdir={{PWD}}/tmp \
      -Dspark.yarn.app.container.log.dir=<LOG_DIR> \
      -XX:OnOutOfMemoryError='kill %p' \
      org.apache.spark.executor.CoarseGrainedExecutorBackend \
      --driver-url \
      spark://CoarseGrainedScheduler@hpe01:41862 \
      --executor-id \
      <executorId> \
      --hostname \
      <hostname> \
      --cores \
      8 \
      --app-id \
      application_1525333084924_0016 \
      --user-class-path \
      file:$PWD/__app__.jar \
      --user-class-path \
      file:$PWD/tensorflow-hadoop-1.6.0.jar \
      1><LOG_DIR>/stdout \
      2><LOG_DIR>/stderr

  resources:
    __spark_libs__/scala-xml_2.11-1.0.5.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/scala-xml_2.11-1.0.5.jar" } size: 671138 timestamp:<PHONE_NUMBER>989 type: FILE visibility: PUBLIC
    __spark_libs__/xbean-asm5-shaded-4.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/xbean-asm5-shaded-4.4.jar" } size: 144660 timestamp:<PHONE_NUMBER>331 type: FILE visibility: PUBLIC
    __spark_libs__/apache-log4j-extras-1.2.17.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/apache-log4j-extras-1.2.17.jar" } size: 448794 timestamp:<PHONE_NUMBER>722 type: FILE visibility: PUBLIC
    __spark_libs__/base64-2.3.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/base64-2.3.8.jar" } size: 17008 timestamp:<PHONE_NUMBER>098 type: FILE visibility: PUBLIC
    __spark_libs__/scala-library-2.11.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/scala-library-2.11.8.jar" } size: 5744974 timestamp:<PHONE_NUMBER>849 type: FILE visibility: PUBLIC
    __spark_libs__/javolution-5.5.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/javolution-5.5.1.jar" } size: 395195 timestamp:<PHONE_NUMBER>215 type: FILE visibility: PUBLIC
    __spark_libs__/hk2-locator-2.4.0-b34.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hk2-locator-2.4.0-b34.jar" } size: 181271 timestamp:<PHONE_NUMBER>502 type: FILE visibility: PUBLIC
    __spark_libs__/javax.ws.rs-api-2.0.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/javax.ws.rs-api-2.0.1.jar" } size: 115534 timestamp:<PHONE_NUMBER>189 type: FILE visibility: PUBLIC
    __spark_libs__/parquet-encoding-1.8.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/parquet-encoding-1.8.2.jar" } size: 286071 timestamp:<PHONE_NUMBER>401 type: FILE visibility: PUBLIC
    __spark_libs__/compress-lzf-1.0.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/compress-lzf-1.0.3.jar" } size: 79845 timestamp:<PHONE_NUMBER>117 type: FILE visibility: PUBLIC
    __spark_libs__/java-xmlbuilder-1.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/java-xmlbuilder-1.1.jar" } size: 17316 timestamp:<PHONE_NUMBER>142 type: FILE visibility: PUBLIC
    __spark_libs__/parquet-jackson-1.8.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/parquet-jackson-1.8.2.jar" } size: 1048116 timestamp:<PHONE_NUMBER>529 type: FILE visibility: PUBLIC
    __spark_libs__/guava-14.0.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/guava-14.0.1.jar" } size: 2189117 timestamp:<PHONE_NUMBER>561 type: FILE visibility: PUBLIC
    __spark_libs__/jackson-module-jaxb-annotations-2.6.7.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jackson-module-jaxb-annotations-2.6.7.jar" } size: 32612 timestamp:<PHONE_NUMBER>899 type: FILE visibility: PUBLIC
    __spark_libs__/javax.annotation-api-1.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/javax.annotation-api-1.2.jar" } size: 26366 timestamp:<PHONE_NUMBER>075 type: FILE visibility: PUBLIC
    __spark_libs__/json4s-ast_2.11-3.2.11.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/json4s-ast_2.11-3.2.11.jar" } size: 82421 timestamp:<PHONE_NUMBER>679 type: FILE visibility: PUBLIC
    __spark_libs__/breeze_2.11-0.13.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/breeze_2.11-0.13.2.jar" } size: 15113382 timestamp:<PHONE_NUMBER>348 type: FILE visibility: PUBLIC
    __spark_libs__/httpclient-4.5.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/httpclient-4.5.4.jar" } size: 781831 timestamp:<PHONE_NUMBER>628 type: FILE visibility: PUBLIC
    __spark_libs__/logging-interceptor-3.8.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/logging-interceptor-3.8.1.jar" } size: 8268 timestamp:<PHONE_NUMBER>076 type: FILE visibility: PUBLIC
    __spark_libs__/stream-2.7.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/stream-2.7.0.jar" } size: 174351 timestamp:<PHONE_NUMBER>228 type: FILE visibility: PUBLIC
    __spark_libs__/commons-compress-1.4.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-compress-1.4.1.jar" } size: 241367 timestamp:<PHONE_NUMBER>744 type: FILE visibility: PUBLIC
    __spark_libs__/spark-tags_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-tags_2.11-2.3.0.jar" } size: 15456 timestamp:<PHONE_NUMBER>968 type: FILE visibility: PUBLIC
    __spark_libs__/commons-math3-3.4.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-math3-3.4.1.jar" } size: 2035066 timestamp:<PHONE_NUMBER>029 type: FILE visibility: PUBLIC
    __spark_libs__/hk2-api-2.4.0-b34.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hk2-api-2.4.0-b34.jar" } size: 178947 timestamp:<PHONE_NUMBER>474 type: FILE visibility: PUBLIC
    __spark_libs__/commons-beanutils-1.7.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-beanutils-1.7.0.jar" } size: 188671 timestamp:<PHONE_NUMBER>583 type: FILE visibility: PUBLIC
    __spark_libs__/derby-<IP_ADDRESS>.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/derby-<IP_ADDRESS>.jar" } size: 3224708 timestamp:<PHONE_NUMBER>409 type: FILE visibility: PUBLIC
    __spark_libs__/spark-unsafe_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-unsafe_2.11-2.3.0.jar" } size: 48464 timestamp:<PHONE_NUMBER>987 type: FILE visibility: PUBLIC
    tensorflow-hadoop-1.6.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/user/hpe/.sparkStaging/application_1525333084924_0016/tensorflow-hadoop-1.6.0.jar" } size: 123665 timestamp:<PHONE_NUMBER>874 type: FILE visibility: PRIVATE
    __spark_libs__/jersey-client-2.22.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jersey-client-2.22.2.jar" } size: 167421 timestamp:<PHONE_NUMBER>306 type: FILE visibility: PUBLIC
    __spark_libs__/hive-cli-1.2.1.spark2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hive-cli-1.2.1.spark2.jar" } size: 40817 timestamp:<PHONE_NUMBER>214 type: FILE visibility: PUBLIC
    __spark_libs__/antlr-2.7.7.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/antlr-2.7.7.jar" } size: 445288 timestamp:<PHONE_NUMBER>478 type: FILE visibility: PUBLIC
    __spark_libs__/hive-metastore-1.2.1.spark2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hive-metastore-1.2.1.spark2.jar" } size: 5505200 timestamp:<PHONE_NUMBER>446 type: FILE visibility: PUBLIC
    py4j-0.10.6-src.zip -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/user/hpe/.sparkStaging/application_1525333084924_0016/py4j-0.10.6-src.zip" } size: 80352 timestamp:<PHONE_NUMBER>978 type: FILE visibility: PRIVATE
    __spark_libs__/jackson-module-paranamer-2.7.9.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jackson-module-paranamer-2.7.9.jar" } size: 42858 timestamp:<PHONE_NUMBER>922 type: FILE visibility: PUBLIC
    __spark_libs__/macro-compat_2.11-1.1.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/macro-compat_2.11-1.1.1.jar" } size: 3142 timestamp:<PHONE_NUMBER>139 type: FILE visibility: PUBLIC
    __spark_libs__/okhttp-3.8.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/okhttp-3.8.1.jar" } size: 398122 timestamp:<PHONE_NUMBER>145 type: FILE visibility: PUBLIC
    __spark_libs__/curator-recipes-2.7.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/curator-recipes-2.7.1.jar" } size: 270342 timestamp:<PHONE_NUMBER>239 type: FILE visibility: PUBLIC
    __spark_libs__/jackson-mapper-asl-1.9.13.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jackson-mapper-asl-1.9.13.jar" } size: 780664 timestamp:<PHONE_NUMBER>875 type: FILE visibility: PUBLIC
    __spark_libs__/commons-net-2.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-net-2.2.jar" } size: 212453 timestamp:<PHONE_NUMBER>057 type: FILE visibility: PUBLIC
    __spark_libs__/commons-lang3-3.5.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-lang3-3.5.jar" } size: 479881 timestamp:<PHONE_NUMBER>961 type: FILE visibility: PUBLIC
    __spark_libs__/oro-2.0.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/oro-2.0.8.jar" } size: 65261 timestamp:<PHONE_NUMBER>283 type: FILE visibility: PUBLIC
    __spark_libs__/jackson-core-asl-1.9.13.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jackson-core-asl-1.9.13.jar" } size: 232248 timestamp:<PHONE_NUMBER>765 type: FILE visibility: PUBLIC
    __spark_libs__/jta-1.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jta-1.1.jar" } size: 15071 timestamp:<PHONE_NUMBER>785 type: FILE visibility: PUBLIC
    __spark_libs__/jetty-util-6.1.26.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jetty-util-6.1.26.jar" } size: 177131 timestamp:<PHONE_NUMBER>557 type: FILE visibility: PUBLIC
    __spark_libs__/gson-2.2.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/gson-2.2.4.jar" } size: 190432 timestamp:<PHONE_NUMBER>516 type: FILE visibility: PUBLIC
    __spark_libs__/kryo-shaded-3.0.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/kryo-shaded-3.0.3.jar" } size: 358390 timestamp:<PHONE_NUMBER>850 type: FILE visibility: PUBLIC
    __spark_libs__/avro-mapred-1.7.7-hadoop2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/avro-mapred-1.7.7-hadoop2.jar" } size: 180736 timestamp:<PHONE_NUMBER>070 type: FILE visibility: PUBLIC
    __spark_libs__/xercesImpl-2.9.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/xercesImpl-2.9.1.jar" } size: 1229125 timestamp:<PHONE_NUMBER>361 type: FILE visibility: PUBLIC
    __spark_libs__/jpam-1.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jpam-1.1.jar" } size: 12131 timestamp:<PHONE_NUMBER>657 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-mapreduce-client-common-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-mapreduce-client-common-2.7.3.jar" } size: 776634 timestamp:<PHONE_NUMBER>914 type: FILE visibility: PUBLIC
    __spark_libs__/chill_2.11-0.8.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/chill_2.11-0.8.4.jar" } size: 224167 timestamp:<PHONE_NUMBER>524 type: FILE visibility: PUBLIC
    __spark_libs__/parquet-hadoop-bundle-1.6.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/parquet-hadoop-bundle-1.6.0.jar" } size: 2796935 timestamp:<PHONE_NUMBER>497 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-mapreduce-client-jobclient-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-mapreduce-client-jobclient-2.7.3.jar" } size: 62304 timestamp:<PHONE_NUMBER>988 type: FILE visibility: PUBLIC
    __spark_libs__/scala-reflect-2.11.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/scala-reflect-2.11.8.jar" } size: 4573750 timestamp:<PHONE_NUMBER>963 type: FILE visibility: PUBLIC
    __spark_libs__/curator-framework-2.7.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/curator-framework-2.7.1.jar" } size: 186273 timestamp:<PHONE_NUMBER>210 type: FILE visibility: PUBLIC
    __spark_libs__/commons-beanutils-core-1.8.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-beanutils-core-1.8.0.jar" } size: 206035 timestamp:<PHONE_NUMBER>610 type: FILE visibility: PUBLIC
    __spark_libs__/commons-logging-1.1.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-logging-1.1.3.jar" } size: 62050 timestamp:<PHONE_NUMBER>986 type: FILE visibility: PUBLIC
    __spark_libs__/json4s-core_2.11-3.2.11.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/json4s-core_2.11-3.2.11.jar" } size: 589462 timestamp:<PHONE_NUMBER>703 type: FILE visibility: PUBLIC
    __spark_libs__/validation-api-1.1.0.Final.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/validation-api-1.1.0.Final.jar" } size: 63777 timestamp:<PHONE_NUMBER>313 type: FILE visibility: PUBLIC
    __spark_libs__/breeze-macros_2.11-0.13.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/breeze-macros_2.11-0.13.2.jar" } size: 187882 timestamp:<PHONE_NUMBER>378 type: FILE visibility: PUBLIC
    __spark_libs__/minlog-1.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/minlog-1.3.0.jar" } size: 5711 timestamp:<PHONE_NUMBER>322 type: FILE visibility: PUBLIC
    __spark_libs__/metrics-json-3.1.5.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/metrics-json-3.1.5.jar" } size: 15824 timestamp:<PHONE_NUMBER>281 type: FILE visibility: PUBLIC
    __spark_libs__/jets3t-0.9.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jets3t-0.9.4.jar" } size: 2046361 timestamp:<PHONE_NUMBER>509 type: FILE visibility: PUBLIC
    __spark_libs__/antlr4-runtime-4.7.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/antlr4-runtime-4.7.jar" } size: 334662 timestamp:<PHONE_NUMBER>514 type: FILE visibility: PUBLIC
    __spark_libs__/commons-cli-1.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-cli-1.2.jar" } size: 41123 timestamp:<PHONE_NUMBER>636 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-yarn-common-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-yarn-common-2.7.3.jar" } size: 1678642 timestamp:<PHONE_NUMBER>117 type: FILE visibility: PUBLIC
    __spark_libs__/bonecp-0.8.0.RELEASE.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/bonecp-0.8.0.RELEASE.jar" } size: 110600 timestamp:<PHONE_NUMBER>190 type: FILE visibility: PUBLIC
    __spark_libs__/jline-2.12.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jline-2.12.1.jar" } size: 213911 timestamp:<PHONE_NUMBER>587 type: FILE visibility: PUBLIC
    __spark_libs__/scala-compiler-2.11.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/scala-compiler-2.11.8.jar" } size: 15487351 timestamp:<PHONE_NUMBER>775 type: FILE visibility: PUBLIC
    __spark_libs__/javax.servlet-api-3.1.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/javax.servlet-api-3.1.0.jar" } size: 95806 timestamp:<PHONE_NUMBER>165 type: FILE visibility: PUBLIC
    __spark_libs__/parquet-common-1.8.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/parquet-common-1.8.2.jar" } size: 42032 timestamp:<PHONE_NUMBER>380 type: FILE visibility: PUBLIC
    __spark_libs__/avro-ipc-1.7.7.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/avro-ipc-1.7.7.jar" } size: 192993 timestamp:<PHONE_NUMBER>041 type: FILE visibility: PUBLIC
    __spark_libs__/hk2-utils-2.4.0-b34.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hk2-utils-2.4.0-b34.jar" } size: 118973 timestamp:<PHONE_NUMBER>526 type: FILE visibility: PUBLIC
    __spark_libs__/chill-java-0.8.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/chill-java-0.8.4.jar" } size: 60483 timestamp:<PHONE_NUMBER>553 type: FILE visibility: PUBLIC
    __spark_libs__/eigenbase-properties-1.1.5.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/eigenbase-properties-1.1.5.jar" } size: 18482 timestamp:<PHONE_NUMBER>435 type: FILE visibility: PUBLIC
    __spark_libs__/jaxb-api-2.2.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jaxb-api-2.2.2.jar" } size: 105134 timestamp:<PHONE_NUMBER>240 type: FILE visibility: PUBLIC
    __spark_libs__/scala-parser-combinators_2.11-1.0.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/scala-parser-combinators_2.11-1.0.4.jar" } size: 423753 timestamp:<PHONE_NUMBER>903 type: FILE visibility: PUBLIC
    __spark_libs__/RoaringBitmap-0.5.11.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/RoaringBitmap-0.5.11.jar" } size: 201928 timestamp:<PHONE_NUMBER>620 type: FILE visibility: PUBLIC
    __spark_libs__/spark-sql_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-sql_2.11-2.3.0.jar" } size: 8665004 timestamp:<PHONE_NUMBER>898 type: FILE visibility: PUBLIC
    __spark_libs__/jackson-jaxrs-1.9.13.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jackson-jaxrs-1.9.13.jar" } size: 18336 timestamp:<PHONE_NUMBER>845 type: FILE visibility: PUBLIC
    __spark_libs__/jersey-container-servlet-2.22.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jersey-container-servlet-2.22.2.jar" } size: 18098 timestamp:<PHONE_NUMBER>366 type: FILE visibility: PUBLIC
    __spark_libs__/py4j-0.10.6.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/py4j-0.10.6.jar" } size: 116859 timestamp:<PHONE_NUMBER>577 type: FILE visibility: PUBLIC
    __spark_libs__/core-1.1.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/core-1.1.2.jar" } size: 164422 timestamp:<PHONE_NUMBER>149 type: FILE visibility: PUBLIC
    __spark_libs__/libthrift-0.9.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/libthrift-0.9.3.jar" } size: 234201 timestamp:<PHONE_NUMBER>033 type: FILE visibility: PUBLIC
    __spark_libs__/jackson-dataformat-yaml-2.6.7.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jackson-dataformat-yaml-2.6.7.jar" } size: 320444 timestamp:<PHONE_NUMBER>823 type: FILE visibility: PUBLIC
    __spark_libs__/spark-hive-thriftserver_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-hive-thriftserver_2.11-2.3.0.jar" } size: 1811675 timestamp:<PHONE_NUMBER>489 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-hdfs-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-hdfs-2.7.3.jar" } size: 8316190 timestamp:<PHONE_NUMBER>853 type: FILE visibility: PUBLIC
    __spark_libs__/htrace-core-3.1.0-incubating.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/htrace-core-3.1.0-incubating.jar" } size: 1475955 timestamp:<PHONE_NUMBER>597 type: FILE visibility: PUBLIC
    __spark_libs__/orc-core-1.4.1-nohive.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/orc-core-1.4.1-nohive.jar" } size: 1441994 timestamp:<PHONE_NUMBER>229 type: FILE visibility: PUBLIC
    __spark_libs__/guice-3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/guice-3.0.jar" } size: 710492 timestamp:<PHONE_NUMBER>595 type: FILE visibility: PUBLIC
    __spark_libs__/commons-pool-1.5.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-pool-1.5.4.jar" } size: 96221 timestamp:<PHONE_NUMBER>087 type: FILE visibility: PUBLIC
    __spark_libs__/aircompressor-0.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/aircompressor-0.8.jar" } size: 130802 timestamp:<PHONE_NUMBER>428 type: FILE visibility: PUBLIC
    __spark_libs__/guice-servlet-3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/guice-servlet-3.0.jar" } size: 65012 timestamp:<PHONE_NUMBER>621 type: FILE visibility: PUBLIC
    __spark_libs__/hppc-0.7.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hppc-0.7.2.jar" } size: 1671083 timestamp:<PHONE_NUMBER>564 type: FILE visibility: PUBLIC
    __spark_libs__/spark-mllib_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-mllib_2.11-2.3.0.jar" } size: 7738543 timestamp:<PHONE_NUMBER>677 type: FILE visibility: PUBLIC
    __spark_conf__ -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/user/hpe/.sparkStaging/application_1525333084924_0016/__spark_conf__.zip" } size: 224134 timestamp:<PHONE_NUMBER>231 type: ARCHIVE visibility: PRIVATE
    __spark_libs__/spark-graphx_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-graphx_2.11-2.3.0.jar" } size: 708678 timestamp:<PHONE_NUMBER>420 type: FILE visibility: PUBLIC
    __spark_libs__/jackson-xc-1.9.13.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jackson-xc-1.9.13.jar" } size: 27084 timestamp:<PHONE_NUMBER>972 type: FILE visibility: PUBLIC
    __spark_libs__/JavaEWAH-0.3.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/JavaEWAH-0.3.2.jar" } size: 16993 timestamp:<PHONE_NUMBER>023 type: FILE visibility: PUBLIC
    __spark_libs__/activation-1.1.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/activation-1.1.1.jar" } size: 69409 timestamp:<PHONE_NUMBER>380 type: FILE visibility: PUBLIC
    __spark_libs__/objenesis-2.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/objenesis-2.1.jar" } size: 41755 timestamp:<PHONE_NUMBER>118 type: FILE visibility: PUBLIC
    __spark_libs__/netty-all-4.1.17.Final.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/netty-all-4.1.17.Final.jar" } size: 3780056 timestamp:<PHONE_NUMBER>097 type: FILE visibility: PUBLIC
    __spark_libs__/commons-crypto-1.0.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-crypto-1.0.0.jar" } size: 134595 timestamp:<PHONE_NUMBER>799 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-annotations-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-annotations-2.7.3.jar" } size: 40863 timestamp:<PHONE_NUMBER>645 type: FILE visibility: PUBLIC
    __spark_libs__/kubernetes-model-2.0.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/kubernetes-model-2.0.0.jar" } size: 7015233 timestamp:<PHONE_NUMBER>953 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-yarn-server-common-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-yarn-server-common-2.7.3.jar" } size: 388235 timestamp:<PHONE_NUMBER>145 type: FILE visibility: PUBLIC
    __spark_libs__/joda-time-2.9.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/joda-time-2.9.3.jar" } size: 627814 timestamp:<PHONE_NUMBER>614 type: FILE visibility: PUBLIC
    __spark_libs__/jodd-core-3.5.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jodd-core-3.5.2.jar" } size: 427780 timestamp:<PHONE_NUMBER>639 type: FILE visibility: PUBLIC
    __spark_libs__/api-util-1.0.0-M20.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/api-util-1.0.0-M20.jar" } size: 79912 timestamp:<PHONE_NUMBER>786 type: FILE visibility: PUBLIC
    __spark_libs__/spark-streaming_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-streaming_2.11-2.3.0.jar" } size: 2169838 timestamp:<PHONE_NUMBER>940 type: FILE visibility: PUBLIC
    __spark_libs__/aopalliance-repackaged-2.4.0-b34.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/aopalliance-repackaged-2.4.0-b34.jar" } size: 14766 timestamp:<PHONE_NUMBER>617 type: FILE visibility: PUBLIC
    __spark_libs__/json4s-jackson_2.11-3.2.11.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/json4s-jackson_2.11-3.2.11.jar" } size: 40341 timestamp:<PHONE_NUMBER>724 type: FILE visibility: PUBLIC
    __spark_libs__/apacheds-kerberos-codec-2.0.0-M15.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/apacheds-kerberos-codec-2.0.0-M15.jar" } size: 691479 timestamp:<PHONE_NUMBER>688 type: FILE visibility: PUBLIC
    __spark_libs__/parquet-format-2.3.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/parquet-format-2.3.1.jar" } size: 390733 timestamp:<PHONE_NUMBER>427 type: FILE visibility: PUBLIC
    __spark_libs__/jdo-api-3.0.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jdo-api-3.0.1.jar" } size: 201124 timestamp:<PHONE_NUMBER>282 type: FILE visibility: PUBLIC
    __spark_libs__/jersey-media-jaxb-2.22.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jersey-media-jaxb-2.22.2.jar" } size: 72733 timestamp:<PHONE_NUMBER>439 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-yarn-server-web-proxy-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-yarn-server-web-proxy-2.7.3.jar" } size: 58407 timestamp:<PHONE_NUMBER>169 type: FILE visibility: PUBLIC
    __spark_libs__/spire_2.11-0.13.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spire_2.11-0.13.0.jar" } size: 10121868 timestamp:<PHONE_NUMBER>128 type: FILE visibility: PUBLIC
    __spark_libs__/snappy-java-<IP_ADDRESS>.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/snappy-java-<IP_ADDRESS>.jar" } size: 1056168 timestamp:<PHONE_NUMBER>153 type: FILE visibility: PUBLIC
    __spark_libs__/machinist_2.11-0.6.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/machinist_2.11-0.6.1.jar" } size: 34603 timestamp:<PHONE_NUMBER>119 type: FILE visibility: PUBLIC
    __spark_libs__/datanucleus-core-3.2.10.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/datanucleus-core-3.2.10.jar" } size: 1890075 timestamp:<PHONE_NUMBER>315 type: FILE visibility: PUBLIC
    __spark_libs__/httpcore-4.4.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/httpcore-4.4.8.jar" } size: 324565 timestamp:<PHONE_NUMBER>654 type: FILE visibility: PUBLIC
    __spark_libs__/scalap-2.11.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/scalap-2.11.8.jar" } size: 802818 timestamp:<PHONE_NUMBER>880 type: FILE visibility: PUBLIC
    __spark_libs__/zstd-jni-1.3.2-2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/zstd-jni-1.3.2-2.jar" } size: 2333186 timestamp:<PHONE_NUMBER>491 type: FILE visibility: PUBLIC
    __spark_libs__/spark-kubernetes_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-kubernetes_2.11-2.3.0.jar" } size: 381117 timestamp:<PHONE_NUMBER>515 type: FILE visibility: PUBLIC
    __spark_libs__/bcprov-jdk15on-1.58.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/bcprov-jdk15on-1.58.jar" } size: 3955990 timestamp:<PHONE_NUMBER>161 type: FILE visibility: PUBLIC
    __spark_libs__/calcite-avatica-1.2.0-incubating.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/calcite-avatica-1.2.0-incubating.jar" } size: 258370 timestamp:<PHONE_NUMBER>408 type: FILE visibility: PUBLIC
    __spark_libs__/javax.inject-2.4.0-b34.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/javax.inject-2.4.0-b34.jar" } size: 5950 timestamp:<PHONE_NUMBER>119 type: FILE visibility: PUBLIC
    __spark_libs__/spire-macros_2.11-0.13.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spire-macros_2.11-0.13.0.jar" } size: 87192 timestamp:<PHONE_NUMBER>153 type: FILE visibility: PUBLIC
    __spark_libs__/datanucleus-api-jdo-3.2.6.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/datanucleus-api-jdo-3.2.6.jar" } size: 339666 timestamp:<PHONE_NUMBER>272 type: FILE visibility: PUBLIC
    __spark_libs__/commons-compiler-3.0.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-compiler-3.0.8.jar" } size: 37887 timestamp:<PHONE_NUMBER>717 type: FILE visibility: PUBLIC
    __spark_libs__/spark-repl_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-repl_2.11-2.3.0.jar" } size: 120336 timestamp:<PHONE_NUMBER>785 type: FILE visibility: PUBLIC
    __spark_libs__/commons-digester-1.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-digester-1.8.jar" } size: 143602 timestamp:<PHONE_NUMBER>847 type: FILE visibility: PUBLIC
    __spark_libs__/spark-sketch_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-sketch_2.11-2.3.0.jar" } size: 30057 timestamp:<PHONE_NUMBER>805 type: FILE visibility: PUBLIC
    tfspark.zip -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/user/hpe/.sparkStaging/application_1525333084924_0016/tfspark.zip" } size: 33111 timestamp:<PHONE_NUMBER>005 type: FILE visibility: PRIVATE
    __spark_libs__/log4j-1.2.17.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/log4j-1.2.17.jar" } size: 489884 timestamp:<PHONE_NUMBER>057 type: FILE visibility: PUBLIC
    __spark_libs__/curator-client-2.7.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/curator-client-2.7.1.jar" } size: 69500 timestamp:<PHONE_NUMBER>181 type: FILE visibility: PUBLIC
    __spark_libs__/arpack_combined_all-0.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/arpack_combined_all-0.1.jar" } size: 1194003 timestamp:<PHONE_NUMBER>826 type: FILE visibility: PUBLIC
    __spark_libs__/commons-collections-3.2.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-collections-3.2.2.jar" } size: 588337 timestamp:<PHONE_NUMBER>691 type: FILE visibility: PUBLIC
    __spark_libs__/netty-3.9.9.Final.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/netty-3.9.9.Final.jar" } size: 1330219 timestamp:<PHONE_NUMBER>359 type: FILE visibility: PUBLIC
    __spark_libs__/commons-configuration-1.6.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-configuration-1.6.jar" } size: 298829 timestamp:<PHONE_NUMBER>773 type: FILE visibility: PUBLIC
    __spark_libs__/commons-codec-1.10.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-codec-1.10.jar" } size: 284184 timestamp:<PHONE_NUMBER>661 type: FILE visibility: PUBLIC
    __spark_libs__/datanucleus-rdbms-3.2.9.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/datanucleus-rdbms-3.2.9.jar" } size: 1809447 timestamp:<PHONE_NUMBER>355 type: FILE visibility: PUBLIC
    __spark_libs__/protobuf-java-2.5.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/protobuf-java-2.5.0.jar" } size: 533455 timestamp:<PHONE_NUMBER>554 type: FILE visibility: PUBLIC
    __spark_libs__/hive-beeline-1.2.1.spark2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hive-beeline-1.2.1.spark2.jar" } size: 138464 timestamp:<PHONE_NUMBER>193 type: FILE visibility: PUBLIC
    __spark_libs__/metrics-jvm-3.1.5.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/metrics-jvm-3.1.5.jar" } size: 39283 timestamp:<PHONE_NUMBER>302 type: FILE visibility: PUBLIC
    __spark_libs__/slf4j-log4j12-1.7.16.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/slf4j-log4j12-1.7.16.jar" } size: 9939 timestamp:<PHONE_NUMBER>080 type: FILE visibility: PUBLIC
    __spark_libs__/jersey-guava-2.22.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jersey-guava-2.22.2.jar" } size: 971310 timestamp:<PHONE_NUMBER>415 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-client-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-client-2.7.3.jar" } size: 26012 timestamp:<PHONE_NUMBER>696 type: FILE visibility: PUBLIC
    __spark_libs__/api-asn1-api-1.0.0-M20.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/api-asn1-api-1.0.0-M20.jar" } size: 16560 timestamp:<PHONE_NUMBER>754 type: FILE visibility: PUBLIC
    __spark_libs__/javassist-3.18.1-GA.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/javassist-3.18.1-GA.jar" } size: 714194 timestamp:<PHONE_NUMBER>051 type: FILE visibility: PUBLIC
    __spark_libs__/lz4-java-1.4.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/lz4-java-1.4.0.jar" } size: 370119 timestamp:<PHONE_NUMBER>100 type: FILE visibility: PUBLIC
    __spark_libs__/calcite-core-1.2.0-incubating.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/calcite-core-1.2.0-incubating.jar" } size: 3519262 timestamp:<PHONE_NUMBER>466 type: FILE visibility: PUBLIC
    pyspark.zip -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/user/hpe/.sparkStaging/application_1525333084924_0016/pyspark.zip" } size: 538841 timestamp:<PHONE_NUMBER>952 type: FILE visibility: PRIVATE
    __spark_libs__/jcl-over-slf4j-1.7.16.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jcl-over-slf4j-1.7.16.jar" } size: 16430 timestamp:<PHONE_NUMBER>261 type: FILE visibility: PUBLIC
    __spark_libs__/arrow-vector-0.8.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/arrow-vector-0.8.0.jar" } size: 1270377 timestamp:<PHONE_NUMBER>947 type: FILE visibility: PUBLIC
    __spark_libs__/jersey-container-servlet-core-2.22.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jersey-container-servlet-core-2.22.2.jar" } size: 66270 timestamp:<PHONE_NUMBER>387 type: FILE visibility: PUBLIC
    __spark_libs__/commons-dbcp-1.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-dbcp-1.4.jar" } size: 160519 timestamp:<PHONE_NUMBER>821 type: FILE visibility: PUBLIC
    __spark_libs__/generex-1.0.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/generex-1.0.1.jar" } size: 13911 timestamp:<PHONE_NUMBER>488 type: FILE visibility: PUBLIC
    __spark_libs__/snappy-0.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/snappy-0.2.jar" } size: 48720 timestamp:<PHONE_NUMBER>123 type: FILE visibility: PUBLIC
    __spark_libs__/apacheds-i18n-2.0.0-M15.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/apacheds-i18n-2.0.0-M15.jar" } size: 44925 timestamp:<PHONE_NUMBER>651 type: FILE visibility: PUBLIC
    __spark_libs__/metrics-graphite-3.1.5.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/metrics-graphite-3.1.5.jar" } size: 21247 timestamp:<PHONE_NUMBER>263 type: FILE visibility: PUBLIC
    __spark_libs__/zjsonpatch-0.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/zjsonpatch-0.3.0.jar" } size: 35518 timestamp:<PHONE_NUMBER>419 type: FILE visibility: PUBLIC
    __spark_libs__/jsr305-1.3.9.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jsr305-1.3.9.jar" } size: 33015 timestamp:<PHONE_NUMBER>763 type: FILE visibility: PUBLIC
    __spark_libs__/janino-3.0.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/janino-3.0.8.jar" } size: 796326 timestamp:<PHONE_NUMBER>001 type: FILE visibility: PUBLIC
    __spark_libs__/jetty-6.1.26.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jetty-6.1.26.jar" } size: 539912 timestamp:<PHONE_NUMBER>535 type: FILE visibility: PUBLIC
    __spark_libs__/snakeyaml-1.15.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/snakeyaml-1.15.jar" } size: 269295 timestamp:<PHONE_NUMBER>101 type: FILE visibility: PUBLIC
    __spark_libs__/jersey-common-2.22.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jersey-common-2.22.2.jar" } size: 698375 timestamp:<PHONE_NUMBER>338 type: FILE visibility: PUBLIC
    __spark_libs__/spark-launcher_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-launcher_2.11-2.3.0.jar" } size: 75835 timestamp:<PHONE_NUMBER>561 type: FILE visibility: PUBLIC
    __spark_libs__/zookeeper-3.4.6.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/zookeeper-3.4.6.jar" } size: 792964 timestamp:<PHONE_NUMBER>444 type: FILE visibility: PUBLIC
    __spark_libs__/opencsv-2.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/opencsv-2.3.jar" } size: 19827 timestamp:<PHONE_NUMBER>191 type: FILE visibility: PUBLIC
    __spark_libs__/flatbuffers-1.2.0-3f79e055.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/flatbuffers-1.2.0-3f79e055.jar" } size: 10166 timestamp:<PHONE_NUMBER>461 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-mapreduce-client-shuffle-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-mapreduce-client-shuffle-2.7.3.jar" } size: 71737 timestamp:<PHONE_NUMBER>011 type: FILE visibility: PUBLIC
    __spark_libs__/jsp-api-2.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jsp-api-2.1.jar" } size: 100636 timestamp:<PHONE_NUMBER>744 type: FILE visibility: PUBLIC
    __spark_libs__/xz-1.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/xz-1.0.jar" } size: 94672 timestamp:<PHONE_NUMBER>399 type: FILE visibility: PUBLIC
    __spark_libs__/avro-1.7.7.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/avro-1.7.7.jar" } size: 436303 timestamp:<PHONE_NUMBER>010 type: FILE visibility: PUBLIC
    __pyfiles__/mnist_dist.py -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/user/hpe/.sparkStaging/application_1525333084924_0016/mnist_dist.py" } size: 5883 timestamp:<PHONE_NUMBER>027 type: FILE visibility: PRIVATE
    __spark_libs__/spark-yarn_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-yarn_2.11-2.3.0.jar" } size: 650250 timestamp:<PHONE_NUMBER>017 type: FILE visibility: PUBLIC
    __spark_libs__/paranamer-2.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/paranamer-2.8.jar" } size: 34654 timestamp:<PHONE_NUMBER>330 type: FILE visibility: PUBLIC
    __spark_libs__/spark-core_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-core_2.11-2.3.0.jar" } size: 12982728 timestamp:<PHONE_NUMBER>390 type: FILE visibility: PUBLIC
    __spark_libs__/ivy-2.4.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/ivy-2.4.0.jar" } size: 1282424 timestamp:<PHONE_NUMBER>691 type: FILE visibility: PUBLIC
    __spark_libs__/osgi-resource-locator-1.0.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/osgi-resource-locator-1.0.1.jar" } size: 20235 timestamp:<PHONE_NUMBER>307 type: FILE visibility: PUBLIC
    __spark_libs__/mesos-1.4.0-shaded-protobuf.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/mesos-1.4.0-shaded-protobuf.jar" } size: 7343426 timestamp:<PHONE_NUMBER>221 type: FILE visibility: PUBLIC
    __spark_libs__/antlr-runtime-3.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/antlr-runtime-3.4.jar" } size: 164368 timestamp:<PHONE_NUMBER>553 type: FILE visibility: PUBLIC
    __spark_libs__/hive-exec-1.2.1.spark2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hive-exec-1.2.1.spark2.jar" } size: 11498852 timestamp:<PHONE_NUMBER>341 type: FILE visibility: PUBLIC
    __spark_libs__/ST4-4.0.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/ST4-4.0.4.jar" } size: 236660 timestamp:<PHONE_NUMBER>173 type: FILE visibility: PUBLIC
    __spark_libs__/calcite-linq4j-1.2.0-incubating.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/calcite-linq4j-1.2.0-incubating.jar" } size: 442406 timestamp:<PHONE_NUMBER>497 type: FILE visibility: PUBLIC
    __spark_libs__/spark-hive_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-hive_2.11-2.3.0.jar" } size: 1304415 timestamp:<PHONE_NUMBER>452 type: FILE visibility: PUBLIC
    __spark_libs__/spark-kvstore_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-kvstore_2.11-2.3.0.jar" } size: 53007 timestamp:<PHONE_NUMBER>538 type: FILE visibility: PUBLIC
    __spark_libs__/jackson-core-2.6.7.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jackson-core-2.6.7.jar" } size: 258919 timestamp:<PHONE_NUMBER>740 type: FILE visibility: PUBLIC
    __spark_libs__/jersey-server-2.22.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jersey-server-2.22.2.jar" } size: 951701 timestamp:<PHONE_NUMBER>469 type: FILE visibility: PUBLIC
    __spark_libs__/parquet-hadoop-1.8.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/parquet-hadoop-1.8.2.jar" } size: 250377 timestamp:<PHONE_NUMBER>450 type: FILE visibility: PUBLIC
    __spark_libs__/stax-api-1.0-2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/stax-api-1.0-2.jar" } size: 23346 timestamp:<PHONE_NUMBER>209 type: FILE visibility: PUBLIC
    __spark_libs__/super-csv-2.2.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/super-csv-2.2.0.jar" } size: 93210 timestamp:<PHONE_NUMBER>270 type: FILE visibility: PUBLIC
    __spark_libs__/automaton-1.11-8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/automaton-1.11-8.jar" } size: 176285 timestamp:<PHONE_NUMBER>977 type: FILE visibility: PUBLIC
    __spark_libs__/libfb303-0.9.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/libfb303-0.9.3.jar" } size: 313702 timestamp:<PHONE_NUMBER>008 type: FILE visibility: PUBLIC
    __spark_libs__/slf4j-api-1.7.16.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/slf4j-api-1.7.16.jar" } size: 40509 timestamp:<PHONE_NUMBER>060 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-yarn-api-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-yarn-api-2.7.3.jar" } size: 2039143 timestamp:<PHONE_NUMBER>053 type: FILE visibility: PUBLIC
    __spark_libs__/javax.inject-1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/javax.inject-1.jar" } size: 2497 timestamp:<PHONE_NUMBER>097 type: FILE visibility: PUBLIC
    __spark_libs__/kubernetes-client-3.0.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/kubernetes-client-3.0.0.jar" } size: 416731 timestamp:<PHONE_NUMBER>872 type: FILE visibility: PUBLIC
    __spark_libs__/spark-mesos_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-mesos_2.11-2.3.0.jar" } size: 663338 timestamp:<PHONE_NUMBER>587 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-mapreduce-client-app-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-mapreduce-client-app-2.7.3.jar" } size: 542869 timestamp:<PHONE_NUMBER>883 type: FILE visibility: PUBLIC
    __spark_libs__/xmlenc-0.52.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/xmlenc-0.52.jar" } size: 15010 timestamp:<PHONE_NUMBER>378 type: FILE visibility: PUBLIC
    __spark_libs__/leveldbjni-all-1.8.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/leveldbjni-all-1.8.jar" } size: 1045744 timestamp:<PHONE_NUMBER>985 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-mapreduce-client-core-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-mapreduce-client-core-2.7.3.jar" } size: 1556539 timestamp:<PHONE_NUMBER>958 type: FILE visibility: PUBLIC
    __spark_libs__/stax-api-1.0.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/stax-api-1.0.1.jar" } size: 26514 timestamp:<PHONE_NUMBER>189 type: FILE visibility: PUBLIC
    __spark_libs__/univocity-parsers-2.5.9.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/univocity-parsers-2.5.9.jar" } size: 384150 timestamp:<PHONE_NUMBER>292 type: FILE visibility: PUBLIC
    __spark_libs__/arrow-format-0.8.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/arrow-format-0.8.0.jar" } size: 52076 timestamp:<PHONE_NUMBER>863 type: FILE visibility: PUBLIC
    __spark_libs__/commons-httpclient-3.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-httpclient-3.1.jar" } size: 305001 timestamp:<PHONE_NUMBER>875 type: FILE visibility: PUBLIC
    __spark_libs__/pyrolite-4.13.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/pyrolite-4.13.jar" } size: 94796 timestamp:<PHONE_NUMBER>598 type: FILE visibility: PUBLIC
    __spark_libs__/jul-to-slf4j-1.7.16.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jul-to-slf4j-1.7.16.jar" } size: 4596 timestamp:<PHONE_NUMBER>830 type: FILE visibility: PUBLIC
    __spark_libs__/hive-jdbc-1.2.1.spark2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hive-jdbc-1.2.1.spark2.jar" } size: 100680 timestamp:<PHONE_NUMBER>369 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-auth-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-auth-2.7.3.jar" } size: 94150 timestamp:<PHONE_NUMBER>670 type: FILE visibility: PUBLIC
    __spark_libs__/metrics-core-3.1.5.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/metrics-core-3.1.5.jar" } size: 120465 timestamp:<PHONE_NUMBER>244 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-yarn-client-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-yarn-client-2.7.3.jar" } size: 165867 timestamp:<PHONE_NUMBER>079 type: FILE visibility: PUBLIC
    __spark_libs__/okio-1.13.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/okio-1.13.0.jar" } size: 81811 timestamp:<PHONE_NUMBER>169 type: FILE visibility: PUBLIC
    __spark_libs__/parquet-column-1.8.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/parquet-column-1.8.2.jar" } size: 956984 timestamp:<PHONE_NUMBER>360 type: FILE visibility: PUBLIC
    __spark_libs__/stringtemplate-3.2.1.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/stringtemplate-3.2.1.jar" } size: 148627 timestamp:<PHONE_NUMBER>247 type: FILE visibility: PUBLIC
    __spark_libs__/commons-io-2.4.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-io-2.4.jar" } size: 185140 timestamp:<PHONE_NUMBER>904 type: FILE visibility: PUBLIC
    __spark_libs__/commons-lang-2.6.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/commons-lang-2.6.jar" } size: 284220 timestamp:<PHONE_NUMBER>933 type: FILE visibility: PUBLIC
    __spark_libs__/jackson-annotations-2.6.7.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jackson-annotations-2.6.7.jar" } size: 46986 timestamp:<PHONE_NUMBER>715 type: FILE visibility: PUBLIC
    __spark_libs__/spark-catalyst_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-catalyst_2.11-2.3.0.jar" } size: 8984438 timestamp:<PHONE_NUMBER>253 type: FILE visibility: PUBLIC
    __spark_libs__/orc-mapreduce-1.4.1-nohive.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/orc-mapreduce-1.4.1-nohive.jar" } size: 757423 timestamp:<PHONE_NUMBER>259 type: FILE visibility: PUBLIC
    __spark_libs__/aopalliance-1.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/aopalliance-1.0.jar" } size: 4467 timestamp:<PHONE_NUMBER>584 type: FILE visibility: PUBLIC
    __spark_libs__/jackson-module-scala_2.11-<IP_ADDRESS>.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jackson-module-scala_2.11-<IP_ADDRESS>.jar" } size: 515645 timestamp:<PHONE_NUMBER>949 type: FILE visibility: PUBLIC
    __spark_libs__/jackson-databind-<IP_ADDRESS>.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jackson-databind-<IP_ADDRESS>.jar" } size: 1165323 timestamp:<PHONE_NUMBER>798 type: FILE visibility: PUBLIC
    __spark_libs__/spark-network-shuffle_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-network-shuffle_2.11-2.3.0.jar" } size: 64847 timestamp:<PHONE_NUMBER>763 type: FILE visibility: PUBLIC
    __spark_libs__/spark-network-common_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-network-common_2.11-2.3.0.jar" } size: 2382165 timestamp:<PHONE_NUMBER>743 type: FILE visibility: PUBLIC
    __spark_libs__/arrow-memory-0.8.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/arrow-memory-0.8.0.jar" } size: 79150 timestamp:<PHONE_NUMBER>897 type: FILE visibility: PUBLIC
    __spark_libs__/jtransforms-2.4.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/jtransforms-2.4.0.jar" } size: 764569 timestamp:<PHONE_NUMBER>811 type: FILE visibility: PUBLIC
    __spark_libs__/hadoop-common-2.7.3.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/hadoop-common-2.7.3.jar" } size: 3479293 timestamp:<PHONE_NUMBER>753 type: FILE visibility: PUBLIC
    __spark_libs__/shapeless_2.11-2.3.2.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/shapeless_2.11-2.3.2.jar" } size: 3522616 timestamp:<PHONE_NUMBER>038 type: FILE visibility: PUBLIC
    __spark_libs__/spark-mllib-local_2.11-2.3.0.jar -> resource { scheme: "hdfs" host: "hpe01" port: 9000 file: "/spark/jars/spark-mllib-local_2.11-2.3.0.jar" } size: 183832 timestamp:<PHONE_NUMBER>701 type: FILE visibility: PUBLIC

===============================================================================
2018-05-07 09:22:22 INFO  RMProxy:98 - Connecting to ResourceManager at hpe01/<IP_ADDRESS>:8030
2018-05-07 09:22:22 INFO  YarnRMClient:54 - Registering the ApplicationMaster
2018-05-07 09:22:23 INFO  YarnAllocator:54 - Will request 3 executor container(s), each with 8 core(s) and 22528 MB memory (including 2048 MB of overhead)
2018-05-07 09:22:23 INFO  YarnSchedulerBackend$YarnSchedulerEndpoint:54 - ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@hpe01:41862)
2018-05-07 09:22:23 INFO  YarnAllocator:54 - Submitted 3 unlocalized container requests.
2018-05-07 09:22:23 INFO  ApplicationMaster:54 - Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
2018-05-07 09:22:23 INFO  AMRMClientImpl:360 - Received new token for : hpe02:36742
2018-05-07 09:22:23 INFO  YarnAllocator:54 - Launching container container_1525333084924_0016_01_000002 on host hpe02 for executor with ID 1
2018-05-07 09:22:23 INFO  YarnAllocator:54 - Received 1 containers from YARN, launching executors on 1 of them.
2018-05-07 09:22:23 INFO  ContainerManagementProtocolProxy:81 - yarn.client.max-cached-nodemanagers-proxies : 0
2018-05-07 09:22:23 INFO  ContainerManagementProtocolProxy:260 - Opening proxy : hpe02:36742
2018-05-07 09:22:24 INFO  AMRMClientImpl:360 - Received new token for : hpe01:33415
2018-05-07 09:22:24 INFO  AMRMClientImpl:360 - Received new token for : hpe03:35000
2018-05-07 09:22:24 INFO  YarnAllocator:54 - Launching container container_1525333084924_0016_01_000003 on host hpe01 for executor with ID 2
2018-05-07 09:22:24 INFO  YarnAllocator:54 - Launching container container_1525333084924_0016_01_000004 on host hpe03 for executor with ID 3
2018-05-07 09:22:24 INFO  YarnAllocator:54 - Received 2 containers from YARN, launching executors on 2 of them.
2018-05-07 09:22:24 INFO  ContainerManagementProtocolProxy:81 - yarn.client.max-cached-nodemanagers-proxies : 0
2018-05-07 09:22:24 INFO  ContainerManagementProtocolProxy:81 - yarn.client.max-cached-nodemanagers-proxies : 0
2018-05-07 09:22:24 INFO  ContainerManagementProtocolProxy:260 - Opening proxy : hpe01:33415
2018-05-07 09:22:24 INFO  ContainerManagementProtocolProxy:260 - Opening proxy : hpe03:35000
2018-05-07 09:22:28 INFO  YarnSchedulerBackend$YarnDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (<IP_ADDRESS>:34684) with ID 1
2018-05-07 09:22:28 INFO  YarnSchedulerBackend$YarnDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (<IP_ADDRESS>:59736) with ID 2
2018-05-07 09:22:28 INFO  BlockManagerMasterEndpoint:54 - Registering block manager hpe02:40311 with 10.5 GB RAM, BlockManagerId(1, hpe02, 40311, None)
2018-05-07 09:22:28 INFO  BlockManagerMasterEndpoint:54 - Registering block manager hpe01:33489 with 10.5 GB RAM, BlockManagerId(2, hpe01, 33489, None)
2018-05-07 09:22:29 INFO  YarnSchedulerBackend$YarnDriverEndpoint:54 - Registered executor NettyRpcEndpointRef(spark-client://Executor) (<IP_ADDRESS>:38020) with ID 3
2018-05-07 09:22:29 INFO  YarnClusterSchedulerBackend:54 - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
2018-05-07 09:22:29 INFO  YarnClusterScheduler:54 - YarnClusterScheduler.postStartHook done
2018-05-07 09:22:29 INFO  BlockManagerMasterEndpoint:54 - Registering block manager hpe03:38332 with 10.5 GB RAM, BlockManagerId(3, hpe03, 38332, None)