Failed deploy Alluxio on a Cluster with HA

Alluxio Version:
What version of Alluxio are you using?
V2.3.0
Describe the bug
A clear and concise description of what the bug is.
Alluxio has some problems when starting ha based on raft election
To Reproduce
Steps to reproduce the behavior (as minimally and precisely as possible)
Expected behavior
A clear and concise description of what you expected to happen.
Urgency
Describe the impact and urgency of the bug.
Additional context
Add any other context about the problem here.

properties
`[masters]
xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01   xx.3.67.77
xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01   xx.3.67.71
xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02   xx.3.67.72
[workers]
xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03   xx.3.67.73
[alluxio-site.properties]
xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01   xx.3.67.77:
alluxio.master.hostname=xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01
alluxio.master.mount.table.root.ufs=hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio
alluxio.master.journal.folder=hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal
alluxio.master.journal.type=EMBEDDED
alluxio.master.embedded.journal.addresses=xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200
alluxio.master.rpc.addresses=xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19998,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19998
alluxio.underfs.hdfs.configuration=/data/alluxio/core-site.xml:/data/alluxio/hdfs-site.xml
xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01   xx.3.67.71:
alluxio.master.hostname=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01
alluxio.master.mount.table.root.ufs=hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio
alluxio.master.journal.folder=hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal
alluxio.master.journal.type=EMBEDDED
alluxio.master.embedded.journal.addresses=xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200
alluxio.master.rpc.addresses=xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19998,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19998
alluxio.underfs.hdfs.configuration=/data/alluxio/core-site.xml:/data/alluxio/hdfs-site.xml
xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02   xx.3.67.72:
alluxio.master.hostname=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02
alluxio.master.mount.table.root.ufs=hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio
alluxio.master.journal.folder=hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal
alluxio.master.journal.type=EMBEDDED
alluxio.master.embedded.journal.addresses=xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200
alluxio.master.rpc.addresses=xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19998,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19998
alluxio.underfs.hdfs.configuration=/data/alluxio/core-site.xml:/data/alluxio/hdfs-site.xml
xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03   xx.3.67.73:
alluxio.master.mount.table.root.ufs=hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio
alluxio.master.journal.folder=hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal
alluxio.master.journal.type=UFS
alluxio.master.web.bind.host=xx.3.67.77
alluxio.master.web.hostname=xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01
alluxio.underfs.hdfs.configuration=/data/alluxio/core-site.xml:/data/alluxio/hdfs-site.xml
alluxio.worker.memory.size=1GB
alluxio.worker.tieredstore.levels=1
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.path=/mnt/ramdisk
[aluxio-env.sh]
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera`

[master.log]
2020-09-22 xx:32:17,013 INFO  FileSystemMasterFactory - Creating alluxio.master.file.FileSystemMaster
2020-09-22 xx:32:17,014 INFO  TableMasterFactory - Creating alluxio.master.table.TableMaster
2020-09-22 xx:32:17,013 INFO  MetaMasterFactory - Creating alluxio.master.meta.MetaMaster
2020-09-22 xx:32:17,013 INFO  MetricsMasterFactory - Creating alluxio.master.metrics.MetricsMaster
2020-09-22 xx:32:17,013 INFO  BlockMasterFactory - Creating alluxio.master.block.BlockMaster
2020-09-22 xx:32:17,077 INFO  TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01, rack=null)
2020-09-22 xx:32:17,078 INFO  UnderDatabaseRegistry - Loading udb jars from /data/alluxio/lib
2020-09-22 xx:32:17,xx1 INFO  UnderDatabaseRegistry - Registered UDBs: hive,glue
2020-09-22 xx:32:17,xx3 INFO  LayoutRegistry - Registered Table Layouts: hive
2020-09-22 xx:32:17,114 INFO  ExtensionFactoryRegistry - Loading core jars from /data/alluxio/lib
2020-09-22 xx:32:17,138 INFO  ExtensionFactoryRegistry - Loading extension jars from /data/alluxio/extensions
2020-09-22 xx:32:17,288 WARN  HdfsUnderFileSystem - Cannot create SupportedHdfsAclProvider. HDFS ACLs will not be supported.
2020-09-22 xx:32:17,347 WARN  NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-09-22 xx:32:17,782 INFO  HdfsUnderFileSystem - Successfully instantiated SupportedHdfsActiveSyncProvider
2020-09-22 xx:32:17,915 INFO  ProcessUtils - Starting Alluxio master @xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998.
2020-09-22 xx:32:17,915 INFO  RaftJournalSystem - Initializing Raft Journal System
2020-09-22 xx:32:17,922 INFO  JournalStateMachine - Initialized new journal state machine
2020-09-22 xx:32:18,005 INFO  AbstractPrimarySelector - Primary selector transitioning to SECONDARY
2020-09-22 xx:32:18,006 INFO  RaftJournalSystem - Starting Raft journal system. Cluster addresses: [xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200, xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200]. Local address: xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200
2020-09-22 xx:32:18,216 INFO  GrpcMessagingServer - Successfully started messaging server at: xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200
2020-09-22 xx:32:18,225 INFO  ServerContext - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Transitioning to FOLLOWER
2020-09-22 xx:32:18,233 INFO  AbstractPrimarySelector - Primary selector transitioning to SECONDARY
2020-09-22 xx:32:18,254 INFO  NettyUtils - EPOLL_MODE is available
2020-09-22 xx:32:35,315 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:33:03,163 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:33:28,609 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:33:52,612 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:34:16,763 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:34:42,987 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:35:08,981 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:35:30,878 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:35:58,996 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:36:25,805 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:36:46,387 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:37:13,034 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:37:38,745 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:38:07,358 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:38:32,707 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:38:58,555 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:39:20,658 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:39:48,382 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:40:12,299 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:40:35,250 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:40:58,064 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:41:19,430 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:41:49,150 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:42:18,775 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:42:41,417 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:43:02,560 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:43:23,988 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:43:52,124 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:44:21,084 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:44:49,048 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:45:18,719 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:45:39,118 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:46:00,3xx INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:46:26,657 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:46:49,562 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:47:18,980 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:47:45,590 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:48:xx,752 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:48:34,063 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:48:59,746 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:49:26,566 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:49:51,620 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:50:19,8xx INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:50:44,489 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:51:xx,733 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:51:37,980 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:52:04,182 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:52:33,396 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:52:53,681 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:53:13,958 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:53:39,429 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:54:01,251 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:54:21,822 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:54:47,230 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:55:12,113 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:55:40,639 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:56:08,720 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:56:28,744 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:56:48,993 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:57:xx,432 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:57:34,090 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:58:01,434 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:58:27,397 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:58:51,201 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:59:18,478 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:59:42,000 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]

[console.log]
`[root@xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01 bin]# ./alluxio-start.sh all Mount
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh proxy
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh proxy
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh job_worker
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh worker
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh job_master
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh master
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a master
Waiting for tasks to finish...
Task on 'xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01' fails, exit code: 255
Task on 'xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02' fails, exit code: 255
There are task failures, look at /data/alluxio/logs/task.log for details.
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a job_master
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a worker Mount
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a job_worker
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a proxy
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a proxy
Waiting for tasks to finish...
All tasks finished
Starting to monitor all remote services.
--- [ FAILED ] The master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/master.log

BEGIN
2020-09-22 10:35:30,878 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:35:58,996 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:36:25,805 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:36:46,387 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:37:13,034 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:37:38,745 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:38:07,358 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:38:32,707 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:38:58,555 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:39:20,658 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:39:48,382 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:40:12,299 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:40:35,250 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:40:58,064 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:41:19,430 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:41:49,150 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:42:18,775 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:42:41,417 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:43:02,560 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:43:23,988 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:43:52,124 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:44:21,084 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:44:49,048 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:45:18,719 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:45:39,118 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:46:00,310 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:46:26,657 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:46:49,562 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:47:18,980 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 10:47:45,590 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
<<< EOF
--- Printing the log tail for /data/alluxio/logs/master.out
--- EMPTY ---
--- [ FAILED ] The master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/master.log
BEGIN
2020-09-22 09:50:12,651 INFO  ConnectionStateManager - State change: CONNECTED
2020-09-22 09:50:12,653 INFO  CuratorFrameworkImpl - Default schema
2020-09-22 09:50:12,717 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/MetaMaster/v1/logs/0x0-0x1.
2020-09-22 09:50:12,717 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/BlockMaster/v1/logs/0x0-0x1.
2020-09-22 09:50:12,723 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x0-0x3.
2020-09-22 09:50:12,753 INFO  NettyUtils - EPOLL_MODE is available
2020-09-22 09:50:12,859 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/BlockMaster/v1/logs/0x1-0x2.
2020-09-22 09:50:12,882 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x3-0x19.
2020-09-22 09:50:29,329 WARN  RetryHandlingMetaMasterMasterClient - GetId(address=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19998) returned<PHONE_NUMBER>001608331 in 16689 ms (>=10000 ms)
2020-09-22 10:05:21,799 INFO  AbstractMaster - TableMaster: Stopped secondary master.
2020-09-22 10:05:21,799 INFO  AbstractMaster - MetaMaster: Stopped secondary master.
2020-09-22 10:05:21,799 INFO  AbstractMaster - FileSystemMaster: Stopped secondary master.
2020-09-22 10:05:21,800 INFO  AbstractMaster - BlockMaster: Stopped secondary master.
2020-09-22 10:05:21,800 INFO  AbstractMaster - MetricsMaster: Stopped secondary master.
2020-09-22 10:05:21,801 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 10:05:22,546 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 10:05:22,546 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal shutdown complete
2020-09-22 10:05:22,546 INFO  UfsJournalCheckpointThread - MetaMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 10:05:22,909 INFO  UfsJournalCheckpointThread - MetaMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 10:05:22,910 INFO  UfsJournalCheckpointThread - MetaMaster: Journal shutdown complete
2020-09-22 10:05:22,910 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 10:05:22,910 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x19-0x1c.
2020-09-22 10:05:22,917 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 10:05:22,917 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal shutdown complete
2020-09-22 10:05:22,917 INFO  UfsJournalCheckpointThread - TableMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 10:05:23,548 INFO  UfsJournalCheckpointThread - TableMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 10:05:23,548 INFO  UfsJournalCheckpointThread - TableMaster: Journal shutdown complete
2020-09-22 10:05:23,548 INFO  UfsJournalCheckpointThread - BlockMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 10:05:23,912 INFO  UfsJournalCheckpointThread - BlockMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 10:05:23,912 INFO  UfsJournalCheckpointThread - BlockMaster: Journal shutdown complete
<<< EOF
--- Printing the log tail for /data/alluxio/logs/master.out
--- [ FAILED ] The master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/master.log
BEGIN
Exception in thread "RejectingServer-/<IP_ADDRESS>:19999" java.lang.RuntimeException: java.net.BindException: Cannot assign requested address (Bind failed)
at alluxio.network.RejectingServer.run(RejectingServer.java:50)
Caused by: java.net.BindException: Cannot assign requested address (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.bind(ServerSocket.java:329)
at alluxio.network.RejectingServer.run(RejectingServer.java:47)
<<< EOF
BEGIN
2020-09-22 09:50:12,664 INFO  ConnectionStateManager - State change: CONNECTED
2020-09-22 09:50:12,666 INFO  CuratorFrameworkImpl - Default schema
2020-09-22 09:50:12,735 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x0-0x3.
2020-09-22 09:50:12,735 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/MetaMaster/v1/logs/0x0-0x1.
2020-09-22 09:50:12,735 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/BlockMaster/v1/logs/0x0-0x1.
2020-09-22 09:50:12,769 INFO  NettyUtils - EPOLL_MODE is available
2020-09-22 09:50:12,877 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/BlockMaster/v1/logs/0x1-0x2.
2020-09-22 09:50:12,898 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x3-0x19.
2020-09-22 09:50:29,163 WARN  RetryHandlingMetaMasterMasterClient - GetId(address=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19998) returned<PHONE_NUMBER>782265916 in 16508 ms (>=10000 ms)
2020-09-22 10:05:21,796 INFO  AbstractMaster - TableMaster: Stopped secondary master.
2020-09-22 10:05:21,796 INFO  AbstractMaster - MetaMaster: Stopped secondary master.
2020-09-22 10:05:21,797 INFO  AbstractMaster - FileSystemMaster: Stopped secondary master.
2020-09-22 10:05:21,797 INFO  AbstractMaster - BlockMaster: Stopped secondary master.
2020-09-22 10:05:21,797 INFO  AbstractMaster - MetricsMaster: Stopped secondary master.
2020-09-22 10:05:21,798 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 10:05:22,545 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 10:05:22,546 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal shutdown complete
2020-09-22 10:05:22,546 INFO  UfsJournalCheckpointThread - MetaMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 10:05:22,909 INFO  UfsJournalCheckpointThread - MetaMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 10:05:22,910 INFO  UfsJournalCheckpointThread - MetaMaster: Journal shutdown complete
2020-09-22 10:05:22,910 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 10:05:22,910 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x19-0x1c.
2020-09-22 10:05:22,917 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 10:05:22,917 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal shutdown complete
2020-09-22 10:05:22,917 INFO  UfsJournalCheckpointThread - TableMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 10:05:23,548 INFO  UfsJournalCheckpointThread - TableMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 10:05:23,548 INFO  UfsJournalCheckpointThread - TableMaster: Journal shutdown complete
2020-09-22 10:05:23,548 INFO  UfsJournalCheckpointThread - BlockMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 10:05:23,912 INFO  UfsJournalCheckpointThread - BlockMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 10:05:23,912 INFO  UfsJournalCheckpointThread - BlockMaster: Journal shutdown complete
<<< EOF
--- Printing the log tail for /data/alluxio/logs/master.out
BEGIN
Exception in thread "RejectingServer-/<IP_ADDRESS>:19999" java.lang.RuntimeException: java.net.BindException: Cannot assign requested address (Bind failed)
at alluxio.network.RejectingServer.run(RejectingServer.java:50)
Caused by: java.net.BindException: Cannot assign requested address (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.bind(ServerSocket.java:329)
at alluxio.network.RejectingServer.run(RejectingServer.java:47)
<<< EOF
--- [ FAILED ] The job_master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/job_master.log
BEGIN
2020-09-22 10:47:18,415 INFO  network.TieredIdentityFactory (TieredIdentityFactory.java:localIdentity) - Initialized tiered identity TieredIdentity(node=xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01, rack=null)
2020-09-22 10:47:18,435 INFO  alluxio.ProcessUtils (ProcessUtils.java:run) - Starting Alluxio job master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20001.
2020-09-22 10:47:18,435 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Initializing Raft Journal System
2020-09-22 10:47:18,443 INFO  raft.JournalStateMachine (JournalStateMachine.java:) - Initialized new journal state machine
2020-09-22 10:47:18,531 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to SECONDARY
2020-09-22 10:47:18,532 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Starting Raft journal system. Cluster addresses: [xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003, xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003, xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003]. Local address: xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003
2020-09-22 10:47:18,757 INFO  transport.GrpcMessagingServer (GrpcMessagingServer.java:lambda$listen$0) - Successfully started messaging server at: xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003
2020-09-22 10:47:18,767 INFO  state.ServerContext (ServerContext.java:transition) - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003 - Transitioning to FOLLOWER
2020-09-22 10:47:18,775 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to SECONDARY
2020-09-22 10:47:18,799 INFO  network.NettyUtils (NettyUtils.java:checkNettyEpollAvailable) - EPOLL_MODE is available
2020-09-22 10:47:34,364 INFO  state.ServerContext (ServerContext.java:setLeader) - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003 - Found leader xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003
2020-09-22 10:47:34,425 INFO  state.ServerStateMachine (ServerStateMachine.java:takeSnapshot) - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003 - Taking snapshot 1
2020-09-22 10:47:34,435 INFO  raft.JournalStateMachine (JournalStateMachine.java:snapshot) - Completed snapshot up to SN -1 in 5ms
2020-09-22 10:47:54,383 INFO  server.CopycatServer (CopycatServer.java:lambda$start$6) - Server started successfully!
2020-09-22 10:47:54,383 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Started Raft Journal System in 35851ms
<<< EOF
--- Printing the log tail for /data/alluxio/logs/job_master.out
--- EMPTY ---
--- [ FAILED ] The job_master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/job_master.log
BEGIN
2020-09-22 10:47:34,340 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to SECONDARY
2020-09-22 10:47:34,350 INFO  state.ServerContext (ServerContext.java:transition) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003 - Transitioning to LEADER
2020-09-22 10:47:34,353 INFO  state.ServerContext (ServerContext.java:setLeader) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003 - Found leader xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003
2020-09-22 10:47:34,355 INFO  server.CopycatServer (CopycatServer.java:lambda$start$6) - Server started successfully!
2020-09-22 10:47:34,355 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Started Raft Journal System in 15804ms
2020-09-22 10:47:34,365 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to PRIMARY
2020-09-22 10:47:34,365 INFO  master.FaultTolerantAlluxioJobMasterProcess (FaultTolerantAlluxioJobMasterProcess.java:start) - Transitioning from secondary to primary
2020-09-22 10:47:34,435 INFO  session.ClientSession (ClientSessionManager.java:lambda$register$2) - Registered session 3
2020-09-22 10:47:34,437 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:catchUp) - Performing catchup. Last applied SN: -1. Catchup ID: -1512486790793928800
2020-09-22 10:47:39,201 INFO  state.ClusterState (ClusterState.java:lambda$join$8) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003 - Successfully joined via xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003
2020-09-22 10:47:54,605 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:catchUp) - Caught up in 20169ms. Last sequence number from previous term: -1.
2020-09-22 10:47:54,609 INFO  master.AbstractMaster (AbstractMaster.java:stop) - JobMaster: Stopped secondary master.
2020-09-22 10:47:54,609 INFO  master.FaultTolerantAlluxioJobMasterProcess (FaultTolerantAlluxioJobMasterProcess.java:start) - Secondary stopped
2020-09-22 10:47:54,609 INFO  master.AbstractMaster (AbstractMaster.java:start) - JobMaster: Starting primary master.
2020-09-22 10:47:54,613 INFO  master.FaultTolerantAlluxioJobMasterProcess (FaultTolerantAlluxioJobMasterProcess.java:start) - Primary started
2020-09-22 10:47:54,613 INFO  master.AlluxioJobMasterProcess (AlluxioJobMasterProcess.java:startServing) - Alluxio job master web server version 2.3.0 starting (gained leadership). webAddress=/<IP_ADDRESS>:20002
2020-09-22 10:47:54,619 INFO  util.log (Log.java:initialized) - Logging initialized @36736ms
2020-09-22 10:47:54,688 INFO  web.WebServer (WebServer.java:start) - Alluxio Job Manager Master Web service starting @ /<IP_ADDRESS>:20002
2020-09-22 10:47:54,690 INFO  server.Server (Server.java:doStart) - jetty-9.2.z-SNAPSHOT
2020-09-22 10:47:54,710 INFO  handler.ContextHandler (ContextHandler.java:doStart) - Started o.e.j.s.ServletContextHandler@3d970b44{/metrics/json,null,AVAILABLE}
2020-09-22 10:47:54,711 WARN  security.SecurityHandler (ConstraintSecurityHandler.java:checkPathsWithUncoveredHttpMethods) - ServletContext@o.e.j.s.ServletContextHandler@c9d2345{/,null,STARTING} has uncovered http methods for path: /
2020-09-22 10:48:00,597 INFO  handler.ContextHandler (ContextHandler.java:doStart) - Started o.e.j.s.ServletContextHandler@c9d2345{/,null,AVAILABLE}
2020-09-22 10:48:00,603 INFO  server.ServerConnector (AbstractConnector.java:doStart) - Started ServerConnector@43a57692{HTTP/1.1}{<IP_ADDRESS>:20002}
2020-09-22 10:48:00,604 INFO  server.Server (Server.java:doStart) - Started @42721ms
2020-09-22 10:48:00,605 INFO  web.WebServer (WebServer.java:start) - Alluxio Job Manager Master Web service started @ /<IP_ADDRESS>:20002
2020-09-22 10:48:00,605 INFO  master.AlluxioJobMasterProcess (AlluxioJobMasterProcess.java:startServing) - Alluxio job master version 2.3.0 started (gained leadership). bindAddress=/<IP_ADDRESS>:20001, connectAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20001, webAddress=/<IP_ADDRESS>:20002
2020-09-22 10:48:00,605 INFO  master.AlluxioJobMasterProcess (AlluxioJobMasterProcess.java:startServingRPCServer) - Starting Alluxio job master gRPC server on address /<IP_ADDRESS>:20001
2020-09-22 10:48:00,635 INFO  master.MasterProcess (MasterProcess.java:registerServices) - registered service JOB_MASTER_CLIENT_SERVICE
2020-09-22 10:48:00,636 INFO  master.MasterProcess (MasterProcess.java:registerServices) - registered service JOB_MASTER_WORKER_SERVICE
2020-09-22 10:48:00,644 INFO  master.AlluxioJobMasterProcess (AlluxioJobMasterProcess.java:startServingRPCServer) - Started Alluxio job master gRPC server on address xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20001
--- [ FAILED ] The job_master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/job_master.log
<<< EOF
--- Printing the log tail for /data/alluxio/logs/job_master.out
BEGIN
--- EMPTY ---
2020-09-22 09:50:14,024 INFO  zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect) - Opening socket connection to server xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01/<IP_ADDRESS>:2181. Will not attempt to authenticate using SASL (unknown error)
2020-09-22 09:50:14,025 INFO  zookeeper.ClientCnxn (ClientCnxn.java:primeConnection) - Socket connection established, initiating session, client: /<IP_ADDRESS>:50484, server: xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01/<IP_ADDRESS>:2181
2020-09-22 09:50:14,026 INFO  zookeeper.ClientCnxn (ClientCnxn.java:onConnected) - Session establishment complete on server xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01/<IP_ADDRESS>:2181, sessionid = 0x37451f310cfae43, negotiated timeout = 120000
2020-09-22 09:50:14,030 INFO  state.ConnectionStateManager (ConnectionStateManager.java:postState) - State change: CONNECTED
2020-09-22 09:50:14,226 INFO  master.ZkMasterInquireClient (ZkMasterInquireClient.java:) - Creating new zookeeper client for zk@xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:2181,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:2181,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:2181/alluxio/leader
2020-09-22 09:50:14,258 INFO  network.TieredIdentityFactory (TieredIdentityFactory.java:localIdentity) - Initialized tiered identity TieredIdentity(node=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02, rack=null)
2020-09-22 09:50:14,278 INFO  extensions.ExtensionFactoryRegistry (ExtensionFactoryRegistry.java:scanLibs) - Loading core jars from /data/alluxio/lib
2020-09-22 09:50:14,308 INFO  extensions.ExtensionFactoryRegistry (ExtensionFactoryRegistry.java:scanExtensions) - Loading extension jars from /data/alluxio/extensions
2020-09-22 09:50:14,472 WARN  hdfs.HdfsUnderFileSystem (HdfsUnderFileSystem.java:) - Cannot create SupportedHdfsAclProvider. HDFS ACLs will not be supported.
2020-09-22 09:50:14,534 WARN  util.NativeCodeLoader (NativeCodeLoader.java:) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-09-22 09:50:15,034 INFO  hdfs.HdfsUnderFileSystem (HdfsUnderFileSystem.java:) - Successfully instantiated SupportedHdfsActiveSyncProvider
2020-09-22 09:50:15,043 INFO  alluxio.ProcessUtils (ProcessUtils.java:run) - Starting Alluxio job master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20001.
2020-09-22 09:50:15,045 INFO  ufs.UfsJournalCheckpointThread (UfsJournalCheckpointThread.java:runInternal) - JobMaster: Journal checkpoint thread started.
2020-09-22 10:05:19,599 INFO  master.AbstractMaster (AbstractMaster.java:stop) - JobMaster: Stopped secondary master.
2020-09-22 10:05:19,606 INFO  ufs.UfsJournalCheckpointThread (UfsJournalCheckpointThread.java:awaitTermination) - JobMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 10:05:20,045 INFO  ufs.UfsJournalCheckpointThread (UfsJournalCheckpointThread.java:runInternal) - JobMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 10:05:20,046 INFO  ufs.UfsJournalCheckpointThread (UfsJournalCheckpointThread.java:awaitTermination) - JobMaster: Journal shutdown complete
2020-09-22 10:47:18,440 INFO  network.TieredIdentityFactory (TieredIdentityFactory.java:localIdentity) - Initialized tiered identity TieredIdentity(node=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02, rack=null)
2020-09-22 10:47:18,459 INFO  alluxio.ProcessUtils (ProcessUtils.java:run) - Starting Alluxio job master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20001.
2020-09-22 10:47:18,459 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Initializing Raft Journal System
2020-09-22 10:47:18,466 INFO  raft.JournalStateMachine (JournalStateMachine.java:) - Initialized new journal state machine
2020-09-22 10:47:18,593 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to SECONDARY
2020-09-22 10:47:18,594 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Starting Raft journal system. Cluster addresses: [xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003, xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003, xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003]. Local address: xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003
2020-09-22 10:47:18,835 INFO  transport.GrpcMessagingServer (GrpcMessagingServer.java:lambda$listen$0) - Successfully started messaging server at: xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003
2020-09-22 10:47:18,846 INFO  state.ServerContext (ServerContext.java:transition) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003 - Transitioning to FOLLOWER
2020-09-22 10:47:18,854 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to SECONDARY
2020-09-22 10:47:18,877 INFO  network.NettyUtils (NettyUtils.java:checkNettyEpollAvailable) - EPOLL_MODE is available
2020-09-22 10:47:34,365 INFO  state.ServerContext (ServerContext.java:setLeader) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003 - Found leader xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003
2020-09-22 10:47:34,424 INFO  state.ServerStateMachine (ServerStateMachine.java:takeSnapshot) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003 - Taking snapshot 1
2020-09-22 10:47:34,432 INFO  raft.JournalStateMachine (JournalStateMachine.java:snapshot) - Completed snapshot up to SN -1 in 5ms
<<< EOF
--- Printing the log tail for /data/alluxio/logs/job_master.out
--- EMPTY ---
--- [ FAILED ] The worker @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/worker.log
BEGIN
2020-09-22 09:50:21,255 WARN  RetryUtils - Failed to load cluster default configuration with master (attempt 2): alluxio.exception.status.UnavailableException: Failed to handshake with master xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998 to load cluster default configuration values: UNAVAILABLE: Network closed for unknown reason
2020-09-22 09:50:21,481 WARN  RetryUtils - Failed to load cluster default configuration with master (attempt 3): alluxio.exception.status.UnavailableException: Failed to handshake with master xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998 to load cluster default configuration values: UNAVAILABLE: Network closed for unknown reason
2020-09-22 09:50:21,936 WARN  RetryUtils - Failed to load cluster default configuration with master (attempt 4): alluxio.exception.status.UnavailableException: Failed to handshake with master xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998 to load cluster default configuration values: UNAVAILABLE: Network closed for unknown reason
2020-09-22 09:50:22,778 WARN  RetryUtils - Failed to load cluster default configuration with master (attempt 5): alluxio.exception.status.UnavailableException: Failed to handshake with master xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998 to load cluster default configuration values: UNAVAILABLE: Network closed for unknown reason
2020-09-22 09:50:24,403 WARN  RetryUtils - Failed to load cluster default configuration with master (attempt 6): alluxio.exception.status.UnavailableException: Failed to handshake with master xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998 to load cluster default configuration values: UNAVAILABLE: Network closed for unknown reason
2020-09-22 09:50:27,868 INFO  TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03, rack=null)
2020-09-22 09:50:28,082 INFO  BlockWorkerFactory - Creating alluxio.worker.block.BlockWorker
2020-09-22 09:50:28,193 INFO  log - Logging initialized @7965ms
2020-09-22 09:50:28,414 INFO  GrpcDataServer - Alluxio worker gRPC server started, listening on /<IP_ADDRESS>:29999
2020-09-22 09:50:28,415 INFO  ProcessUtils - Starting Alluxio worker @xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:29999.
2020-09-22 09:50:28,418 INFO  MetricsSystem - Starting sinks with config: {}.
2020-09-22 09:50:28,586 INFO  AlluxioWorkerProcess - Started Alluxio worker @xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:29999 with id<PHONE_NUMBER>525047293
2020-09-22 09:50:28,589 INFO  WebServer - Alluxio worker web service starting @ /<IP_ADDRESS>:30000
2020-09-22 09:50:28,591 INFO  Server - jetty-9.2.z-SNAPSHOT
2020-09-22 09:50:28,622 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@58670130{/metrics/prometheus,null,AVAILABLE}
2020-09-22 09:50:28,622 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@9bd0fa6{/metrics/json,null,AVAILABLE}
2020-09-22 09:50:28,623 WARN  SecurityHandler - ServletContext@o.e.j.s.ServletContextHandler@46f699d5{/,file:/data/alluxio/webui/worker/build/,STARTING} has uncovered http methods for path: /
2020-09-22 09:50:34,502 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@46f699d5{/,file:/data/alluxio/webui/worker/build/,AVAILABLE}
2020-09-22 09:50:34,506 INFO  ServerConnector - Started ServerConnector@2257fadf{HTTP/1.1}{<IP_ADDRESS>:30000}
2020-09-22 09:50:34,506 INFO  Server - Started @14278ms
2020-09-22 09:50:34,506 INFO  WebServer - Alluxio worker web service started @ /<IP_ADDRESS>:30000
2020-09-22 09:50:34,506 INFO  AlluxioWorkerProcess - Alluxio worker version 2.3.0 started. bindHost=<IP_ADDRESS>, connectHost=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03, rpcPort=29999, webPort=30000
2020-09-22 10:05:13,373 INFO  GrpcDataServer - Shutting down Alluxio worker gRPC server at <IP_ADDRESS>/<IP_ADDRESS>:29999.
2020-09-22 10:05:13,375 INFO  AlluxioWorkerProcess - Alluxio worker ended
2020-09-22 10:05:13,375 INFO  ProcessUtils - Stopping Alluxio worker @xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:29999.
2020-09-22 10:05:17,390 INFO  ServerConnector - Stopped ServerConnector@2257fadf{HTTP/1.1}{<IP_ADDRESS>:30000}
2020-09-22 10:05:17,410 INFO  ContextHandler - Stopped o.e.j.s.ServletContextHandler@46f699d5{/,file:/data/alluxio/webui/worker/build/,UNAVAILABLE}
2020-09-22 10:05:17,411 INFO  ContextHandler - Stopped o.e.j.s.ServletContextHandler@9bd0fa6{/metrics/json,null,UNAVAILABLE}
2020-09-22 10:05:17,411 INFO  ContextHandler - Stopped o.e.j.s.ServletContextHandler@58670130{/metrics/prometheus,null,UNAVAILABLE}
2020-09-22 10:47:25,033 ERROR AlluxioWorker - Fatal error: Cannot run Alluxio worker; Unable to determine master address. Please modify alluxio-site.properties to either set alluxio.master.hostname, configure zookeeper with alluxio.zookeeper.enabled=true and alluxio.zookeeper.address=[comma-separated zookeeper master addresses], or utilize internal HA by setting alluxio.master.embedded.journal.addresses=[comma-separated alluxio master addresses]
<<< EOF
--- Printing the log tail for /data/alluxio/logs/worker.out
--- EMPTY ---
--- [ FAILED ] The job_worker @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/job_worker.log
BEGIN
2020-09-22 09:50:28,308 INFO  alluxio.ProcessUtils (ProcessUtils.java:run) - Starting Alluxio job worker.
2020-09-22 09:50:28,309 INFO  web.WebServer (WebServer.java:start) - Alluxio Job Manager Worker Web service starting @ /<IP_ADDRESS>:30003
2020-09-22 09:50:28,310 INFO  server.Server (Server.java:doStart) - jetty-9.2.z-SNAPSHOT
2020-09-22 09:50:28,332 INFO  handler.ContextHandler (ContextHandler.java:doStart) - Started o.e.j.s.ServletContextHandler@61f05988{/metrics/json,null,AVAILABLE}
2020-09-22 09:50:28,333 WARN  security.SecurityHandler (ConstraintSecurityHandler.java:checkPathsWithUncoveredHttpMethods) - ServletContext@o.e.j.s.ServletContextHandler@613a8ee1{/,null,STARTING} has uncovered http methods for path: /
2020-09-22 09:50:31,567 INFO  handler.ContextHandler (ContextHandler.java:doStart) - Started o.e.j.s.ServletContextHandler@613a8ee1{/,null,AVAILABLE}
2020-09-22 09:50:31,572 INFO  server.ServerConnector (AbstractConnector.java:doStart) - Started ServerConnector@2d66530f{HTTP/1.1}{<IP_ADDRESS>:30003}
2020-09-22 09:50:31,572 INFO  server.Server (Server.java:doStart) - Started @7921ms
2020-09-22 09:50:31,572 INFO  web.WebServer (WebServer.java:start) - Alluxio Job Manager Worker Web service started @ /<IP_ADDRESS>:30003
2020-09-22 09:50:31,574 INFO  metrics.MetricsSystem (MetricsSystem.java:startSinksFromConfig) - Starting sinks with config: {}.
2020-09-22 09:50:31,576 INFO  imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start) - Starting
2020-09-22 09:50:31,576 INFO  zookeeper.ZooKeeper (ZooKeeper.java:) - Initiating client connection, connectString=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:2181,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:2181,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@f9b5552
2020-09-22 09:50:31,577 INFO  zookeeper.ClientCnxnSocket (ClientCnxnSocket.java:initProperties) - jute.maxbuffer value is 4194304 Bytes
2020-09-22 09:50:31,577 INFO  zookeeper.ClientCnxn (ClientCnxn.java:initRequestTimeout) - zookeeper.request.timeout value is 0. feature enabled=
2020-09-22 09:50:31,578 INFO  imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start) - Default schema
2020-09-22 09:50:31,578 INFO  zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect) - Opening socket connection to server xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03/<IP_ADDRESS>:2181. Will not attempt to authenticate using SASL (unknown error)
2020-09-22 09:50:31,578 INFO  zookeeper.ClientCnxn (ClientCnxn.java:primeConnection) - Socket connection established, initiating session, client: /<IP_ADDRESS>:38100, server: xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03/<IP_ADDRESS>:2181
2020-09-22 09:50:31,580 INFO  zookeeper.ClientCnxn (ClientCnxn.java:onConnected) - Session establishment complete on server xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03/<IP_ADDRESS>:2181, sessionid = 0x27451f310c3ae8b, negotiated timeout = 60000
2020-09-22 09:50:31,581 INFO  state.ConnectionStateManager (ConnectionStateManager.java:postState) - State change: CONNECTED
2020-09-22 09:51:48,134 WARN  job.JobMasterClient (AbstractClient.java:retryRPC) - RegisterWorker(address=WorkerNetAddress{host=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03, containerHost=, rpcPort=30001, dataPort=30002, webPort=30003, domainSocketPath=, tieredIdentity=null}) returned<PHONE_NUMBER>918 in 76559 ms (>=10000 ms)
2020-09-22 09:51:48,149 INFO  worker.AlluxioJobWorkerProcess (AlluxioJobWorkerProcess.java:start) - Started Alluxio job worker with id<PHONE_NUMBER>918
2020-09-22 09:51:48,150 INFO  worker.AlluxioJobWorkerProcess (AlluxioJobWorkerProcess.java:start) - Alluxio job worker version 2.3.0 started. bindHost=/<IP_ADDRESS>:30001, connectHost=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:30001, rpcPort=30001, webPort=30003
2020-09-22 09:51:48,150 INFO  worker.AlluxioJobWorkerProcess (AlluxioJobWorkerProcess.java:startServingRPCServer) - Starting gRPC server on address xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:30001
2020-09-22 09:51:48,182 INFO  worker.AlluxioJobWorkerProcess (AlluxioJobWorkerProcess.java:startServingRPCServer) - Started gRPC server on address xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:30001
2020-09-22 10:05:12,041 INFO  worker.AlluxioJobWorkerProcess (AlluxioJobWorkerProcess.java:stop) - Stopping RPC server on Alluxio job worker @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:30001
2020-09-22 10:05:12,043 INFO  worker.AlluxioJobWorkerProcess (AlluxioJobWorkerProcess.java:start) - Alluxio job worker ended
2020-09-22 10:05:12,044 INFO  alluxio.ProcessUtils (ProcessUtils.java:run) - Stopping Alluxio job worker.
2020-09-22 10:05:12,045 INFO  server.ServerConnector (AbstractConnector.java:doStop) - Stopped ServerConnector@2d66530f{HTTP/1.1}{<IP_ADDRESS>:30003}
2020-09-22 10:05:12,065 INFO  handler.ContextHandler (ContextHandler.java:doStop) - Stopped o.e.j.s.ServletContextHandler@613a8ee1{/,null,UNAVAILABLE}
2020-09-22 10:05:12,065 INFO  handler.ContextHandler (ContextHandler.java:doStop) - Stopped o.e.j.s.ServletContextHandler@61f05988{/metrics/json,null,UNAVAILABLE}
<<< EOF
--- Printing the log tail for /data/alluxio/logs/job_worker.out
BEGIN
Cannot run Alluxio job worker; Unable to determine master address. Please modify alluxio-site.properties to either set alluxio.master.hostname, configure zookeeper with alluxio.zookeeper.enabled=true and alluxio.zookeeper.address=[comma-separated zookeeper master addresses], or utilize internal HA by setting alluxio.master.embedded.journal.addresses=[comma-separated alluxio master addresses]
<<< EOF
--- [ OK ] The proxy service @ xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01 is in a healthy state.
--- [ OK ] The proxy service @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01 is in a healthy state.
--- [ OK ] The proxy service @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02 is in a healthy state.
--- [ FAILED ] The proxy @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/proxy.log
BEGIN
2020-09-22 09:50:31,734 INFO  log - Logging initialized @370ms
2020-09-22 09:50:31,866 INFO  ZkMasterInquireClient - Creating new zookeeper client for zk@xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:2181,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:2181,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:2181/alluxio/leader
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:zookeeper.version=3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653, built on 05/03/2019 12:07 GMT
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:host.name=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:java.version=1.8.0_161
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:java.vendor=Oracle Corporation
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:java.home=/app/yw/jdk1.8.0_161/jre
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:java.class.path=/data/alluxio/conf/::/data/alluxio/assembly/alluxio-server-2.3.0.jar
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:java.io.tmpdir=/tmp
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:java.compiler=
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:os.name=Linux
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:os.arch=amd64
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:os.version=3.10.0-693.2.2.el7.x86_64
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:user.name=root
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:user.home=/root
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:user.dir=/root
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:os.memory.free=914MB
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:os.memory.max=14281MB
2020-09-22 09:50:31,909 INFO  ZooKeeper - Client environment:os.memory.total=964MB
2020-09-22 09:50:31,910 INFO  Compatibility - Using emulated InjectSessionExpiration
2020-09-22 09:50:31,967 INFO  TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03, rack=null)
2020-09-22 09:50:31,976 INFO  WebServer - Alluxio Proxy Web service starting @ /<IP_ADDRESS>:39999
2020-09-22 09:50:31,978 INFO  Server - jetty-9.2.z-SNAPSHOT
2020-09-22 09:50:31,994 WARN  SecurityHandler - ServletContext@o.e.j.s.ServletContextHandler@1b1473ab{/,null,STARTING} has uncovered http methods for path: /
2020-09-22 09:50:38,199 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@1b1473ab{/,null,AVAILABLE}
2020-09-22 09:50:38,206 INFO  ServerConnector - Started ServerConnector@6928f576{HTTP/1.1}{<IP_ADDRESS>:39999}
2020-09-22 09:50:38,206 INFO  Server - Started @6842ms
2020-09-22 09:50:38,206 INFO  WebServer - Alluxio Proxy Web service started @ /<IP_ADDRESS>:39999
2020-09-22 10:47:36,145 ERROR AlluxioProxy - Fatal error: Cannot run Alluxio proxy; Unable to determine master address. Please modify alluxio-site.properties to either set alluxio.master.hostname, configure zookeeper with alluxio.zookeeper.enabled=true and alluxio.zookeeper.address=[comma-separated zookeeper master addresses], or utilize internal HA by setting alluxio.master.embedded.journal.addresses=[comma-separated alluxio master addresses]
<<< EOF
--- Printing the log tail for /data/alluxio/logs/proxy.out
--- EMPTY ---
`

[console.log]
[root@xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01 bin]# ./alluxio-start.sh all Mount
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh proxy
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh proxy
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh job_worker
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh worker
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh job_master
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-stop.sh master
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a master
Waiting for tasks to finish...
Task on 'xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01' fails, exit code: 255
Task on 'xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02' fails, exit code: 255
There are task failures, look at /data/alluxio/logs/task.log for details.
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a job_master
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a worker Mount
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a job_worker
Waiting for tasks to finish...
All tasks finished
Executing the following command on all master nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a proxy
Waiting for tasks to finish...
All tasks finished
Executing the following command on all worker nodes and logging to /data/alluxio/logs/task.log: /data/alluxio/bin/alluxio-start.sh -a proxy
Waiting for tasks to finish...
All tasks finished
Starting to monitor all remote services.
--- [ FAILED ] The master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/master.log

BEGIN
2020-09-22 xx:35:30,878 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:35:58,996 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:36:25,805 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:36:46,387 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:37:13,034 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:37:38,745 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:38:07,358 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:38:32,707 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:38:58,555 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:39:20,658 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:39:48,382 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:40:12,299 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:40:35,250 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:40:58,064 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:41:19,430 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:41:49,150 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:42:18,775 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:42:41,417 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:43:02,560 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:43:23,988 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:43:52,124 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:44:21,084 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:44:49,048 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:45:18,719 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:45:39,118 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:46:00,3xx INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:46:26,657 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:46:49,562 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:47:18,980 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
2020-09-22 xx:47:45,590 INFO  FollowerState - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19200 - Polling members [ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19200, clientAddress=null], ServerMember[type=ACTIVE, status=AVAILABLE, serverAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19200, clientAddress=null]]
<<< EOF
--- Printing the log tail for /data/alluxio/logs/master.out
--- EMPTY ---
--- [ FAILED ] The master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/master.log
BEGIN
2020-09-22 09:50:12,651 INFO  ConnectionStateManager - State change: CONNECTED
2020-09-22 09:50:12,653 INFO  CuratorFrameworkImpl - Default schema
2020-09-22 09:50:12,717 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/MetaMaster/v1/logs/0x0-0x1.
2020-09-22 09:50:12,717 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/BlockMaster/v1/logs/0x0-0x1.
2020-09-22 09:50:12,723 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x0-0x3.
2020-09-22 09:50:12,753 INFO  NettyUtils - EPOLL_MODE is available
2020-09-22 09:50:12,859 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/BlockMaster/v1/logs/0x1-0x2.
2020-09-22 09:50:12,882 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x3-0x19.
2020-09-22 09:50:29,329 WARN  RetryHandlingMetaMasterMasterClient - GetId(address=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:19998) returned<PHONE_NUMBER>001608331 in 16689 ms (>=xx000 ms)
2020-09-22 xx:05:21,799 INFO  AbstractMaster - TableMaster: Stopped secondary master.
2020-09-22 xx:05:21,799 INFO  AbstractMaster - MetaMaster: Stopped secondary master.
2020-09-22 xx:05:21,799 INFO  AbstractMaster - FileSystemMaster: Stopped secondary master.
2020-09-22 xx:05:21,800 INFO  AbstractMaster - BlockMaster: Stopped secondary master.
2020-09-22 xx:05:21,800 INFO  AbstractMaster - MetricsMaster: Stopped secondary master.
2020-09-22 xx:05:21,801 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 xx:05:22,546 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 xx:05:22,546 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal shutdown complete
2020-09-22 xx:05:22,546 INFO  UfsJournalCheckpointThread - MetaMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 xx:05:22,909 INFO  UfsJournalCheckpointThread - MetaMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 xx:05:22,9xx INFO  UfsJournalCheckpointThread - MetaMaster: Journal shutdown complete
2020-09-22 xx:05:22,9xx INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 xx:05:22,9xx INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x19-0x1c.
2020-09-22 xx:05:22,917 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 xx:05:22,917 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal shutdown complete
2020-09-22 xx:05:22,917 INFO  UfsJournalCheckpointThread - TableMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 xx:05:23,548 INFO  UfsJournalCheckpointThread - TableMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 xx:05:23,548 INFO  UfsJournalCheckpointThread - TableMaster: Journal shutdown complete
2020-09-22 xx:05:23,548 INFO  UfsJournalCheckpointThread - BlockMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 xx:05:23,912 INFO  UfsJournalCheckpointThread - BlockMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 xx:05:23,912 INFO  UfsJournalCheckpointThread - BlockMaster: Journal shutdown complete
<<< EOF
--- Printing the log tail for /data/alluxio/logs/master.out
--- [ FAILED ] The master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/master.log
BEGIN
Exception in thread "RejectingServer-/xx.3.67.77:19999" java.lang.RuntimeException: java.net.BindException: Cannot assign requested address (Bind failed)
at alluxio.network.RejectingServer.run(RejectingServer.java:50)
Caused by: java.net.BindException: Cannot assign requested address (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.bind(ServerSocket.java:329)
at alluxio.network.RejectingServer.run(RejectingServer.java:47)
<<< EOF
BEGIN
2020-09-22 09:50:12,664 INFO  ConnectionStateManager - State change: CONNECTED
2020-09-22 09:50:12,666 INFO  CuratorFrameworkImpl - Default schema
2020-09-22 09:50:12,735 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x0-0x3.
2020-09-22 09:50:12,735 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/MetaMaster/v1/logs/0x0-0x1.
2020-09-22 09:50:12,735 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/BlockMaster/v1/logs/0x0-0x1.
2020-09-22 09:50:12,769 INFO  NettyUtils - EPOLL_MODE is available
2020-09-22 09:50:12,877 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/BlockMaster/v1/logs/0x1-0x2.
2020-09-22 09:50:12,898 INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x3-0x19.
2020-09-22 09:50:29,163 WARN  RetryHandlingMetaMasterMasterClient - GetId(address=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:19998) returned 4590935xx3782265916 in 16508 ms (>=xx000 ms)
2020-09-22 xx:05:21,796 INFO  AbstractMaster - TableMaster: Stopped secondary master.
2020-09-22 xx:05:21,796 INFO  AbstractMaster - MetaMaster: Stopped secondary master.
2020-09-22 xx:05:21,797 INFO  AbstractMaster - FileSystemMaster: Stopped secondary master.
2020-09-22 xx:05:21,797 INFO  AbstractMaster - BlockMaster: Stopped secondary master.
2020-09-22 xx:05:21,797 INFO  AbstractMaster - MetricsMaster: Stopped secondary master.
2020-09-22 xx:05:21,798 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 xx:05:22,545 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 xx:05:22,546 INFO  UfsJournalCheckpointThread - MetricsMaster: Journal shutdown complete
2020-09-22 xx:05:22,546 INFO  UfsJournalCheckpointThread - MetaMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 xx:05:22,909 INFO  UfsJournalCheckpointThread - MetaMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 xx:05:22,9xx INFO  UfsJournalCheckpointThread - MetaMaster: Journal shutdown complete
2020-09-22 xx:05:22,9xx INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 xx:05:22,9xx INFO  UfsJournalReader - Reading journal file hdfs://xx-dev-cq-ecs-dtpbu-datalake-cdh-mansger-01:8020/alluxio/journal/FileSystemMaster/v1/logs/0x19-0x1c.
2020-09-22 xx:05:22,917 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 xx:05:22,917 INFO  UfsJournalCheckpointThread - FileSystemMaster: Journal shutdown complete
2020-09-22 xx:05:22,917 INFO  UfsJournalCheckpointThread - TableMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 xx:05:23,548 INFO  UfsJournalCheckpointThread - TableMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 xx:05:23,548 INFO  UfsJournalCheckpointThread - TableMaster: Journal shutdown complete
2020-09-22 xx:05:23,548 INFO  UfsJournalCheckpointThread - BlockMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 xx:05:23,912 INFO  UfsJournalCheckpointThread - BlockMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 xx:05:23,912 INFO  UfsJournalCheckpointThread - BlockMaster: Journal shutdown complete
<<< EOF
--- Printing the log tail for /data/alluxio/logs/master.out
BEGIN
Exception in thread "RejectingServer-/xx.3.67.77:19999" java.lang.RuntimeException: java.net.BindException: Cannot assign requested address (Bind failed)
at alluxio.network.RejectingServer.run(RejectingServer.java:50)
Caused by: java.net.BindException: Cannot assign requested address (Bind failed)
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.bind(ServerSocket.java:329)
at alluxio.network.RejectingServer.run(RejectingServer.java:47)
<<< EOF
--- [ FAILED ] The job_master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/job_master.log
BEGIN
2020-09-22 xx:47:18,415 INFO  network.TieredIdentityFactory (TieredIdentityFactory.java:localIdentity) - Initialized tiered identity TieredIdentity(node=xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01, rack=null)
2020-09-22 xx:47:18,435 INFO  alluxio.ProcessUtils (ProcessUtils.java:run) - Starting Alluxio job master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20001.
2020-09-22 xx:47:18,435 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Initializing Raft Journal System
2020-09-22 xx:47:18,443 INFO  raft.JournalStateMachine (JournalStateMachine.java:) - Initialized new journal state machine
2020-09-22 xx:47:18,531 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to SECONDARY
2020-09-22 xx:47:18,532 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Starting Raft journal system. Cluster addresses: [xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003, xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003, xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003]. Local address: xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003
2020-09-22 xx:47:18,757 INFO  transport.GrpcMessagingServer (GrpcMessagingServer.java:lambda$listen$0) - Successfully started messaging server at: xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003
2020-09-22 xx:47:18,767 INFO  state.ServerContext (ServerContext.java:transition) - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003 - Transitioning to FOLLOWER
2020-09-22 xx:47:18,775 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to SECONDARY
2020-09-22 xx:47:18,799 INFO  network.NettyUtils (NettyUtils.java:checkNettyEpollAvailable) - EPOLL_MODE is available
2020-09-22 xx:47:34,364 INFO  state.ServerContext (ServerContext.java:setLeader) - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003 - Found leader xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003
2020-09-22 xx:47:34,425 INFO  state.ServerStateMachine (ServerStateMachine.java:takeSnapshot) - xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003 - Taking snapshot 1
2020-09-22 xx:47:34,435 INFO  raft.JournalStateMachine (JournalStateMachine.java:snapshot) - Completed snapshot up to SN -1 in 5ms
2020-09-22 xx:47:54,383 INFO  server.CopycatServer (CopycatServer.java:lambda$start$6) - Server started successfully!
2020-09-22 xx:47:54,383 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Started Raft Journal System in 35851ms
<<< EOF
--- Printing the log tail for /data/alluxio/logs/job_master.out
--- EMPTY ---
--- [ FAILED ] The job_master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/job_master.log
BEGIN
2020-09-22 xx:47:34,340 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to SECONDARY
2020-09-22 xx:47:34,350 INFO  state.ServerContext (ServerContext.java:transition) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003 - Transitioning to LEADER
2020-09-22 xx:47:34,353 INFO  state.ServerContext (ServerContext.java:setLeader) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003 - Found leader xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003
2020-09-22 xx:47:34,355 INFO  server.CopycatServer (CopycatServer.java:lambda$start$6) - Server started successfully!
2020-09-22 xx:47:34,355 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Started Raft Journal System in 15804ms
2020-09-22 xx:47:34,365 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to PRIMARY
2020-09-22 xx:47:34,365 INFO  master.FaultTolerantAlluxioJobMasterProcess (FaultTolerantAlluxioJobMasterProcess.java:start) - Transitioning from secondary to primary
2020-09-22 xx:47:34,435 INFO  session.ClientSession (ClientSessionManager.java:lambda$register$2) - Registered session 3
2020-09-22 xx:47:34,437 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:catchUp) - Performing catchup. Last applied SN: -1. Catchup ID: -1512486790793928800
2020-09-22 xx:47:39,201 INFO  state.ClusterState (ClusterState.java:lambda$join$8) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003 - Successfully joined via xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003
2020-09-22 xx:47:54,605 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:catchUp) - Caught up in 20169ms. Last sequence number from previous term: -1.
2020-09-22 xx:47:54,609 INFO  master.AbstractMaster (AbstractMaster.java:stop) - JobMaster: Stopped secondary master.
2020-09-22 xx:47:54,609 INFO  master.FaultTolerantAlluxioJobMasterProcess (FaultTolerantAlluxioJobMasterProcess.java:start) - Secondary stopped
2020-09-22 xx:47:54,609 INFO  master.AbstractMaster (AbstractMaster.java:start) - JobMaster: Starting primary master.
2020-09-22 xx:47:54,613 INFO  master.FaultTolerantAlluxioJobMasterProcess (FaultTolerantAlluxioJobMasterProcess.java:start) - Primary started
2020-09-22 xx:47:54,613 INFO  master.AlluxioJobMasterProcess (AlluxioJobMasterProcess.java:startServing) - Alluxio job master web server version 2.3.0 starting (gained leadership). webAddress=/<IP_ADDRESS>:20002
2020-09-22 xx:47:54,619 INFO  util.log (Log.java:initialized) - Logging initialized @36736ms
2020-09-22 xx:47:54,688 INFO  web.WebServer (WebServer.java:start) - Alluxio Job Manager Master Web service starting @ /<IP_ADDRESS>:20002
2020-09-22 xx:47:54,690 INFO  server.Server (Server.java:doStart) - jetty-9.2.z-SNAPSHOT
2020-09-22 xx:47:54,7xx INFO  handler.ContextHandler (ContextHandler.java:doStart) - Started o.e.j.s.ServletContextHandler@3d970b44{/metrics/json,null,AVAILABLE}
2020-09-22 xx:47:54,711 WARN  security.SecurityHandler (ConstraintSecurityHandler.java:checkPathsWithUncoveredHttpMethods) - ServletContext@o.e.j.s.ServletContextHandler@c9d2345{/,null,STARTING} has uncovered http methods for path: /
2020-09-22 xx:48:00,597 INFO  handler.ContextHandler (ContextHandler.java:doStart) - Started o.e.j.s.ServletContextHandler@c9d2345{/,null,AVAILABLE}
2020-09-22 xx:48:00,603 INFO  server.ServerConnector (AbstractConnector.java:doStart) - Started ServerConnector@43a57692{HTTP/1.1}{<IP_ADDRESS>:20002}
2020-09-22 xx:48:00,604 INFO  server.Server (Server.java:doStart) - Started @42721ms
2020-09-22 xx:48:00,605 INFO  web.WebServer (WebServer.java:start) - Alluxio Job Manager Master Web service started @ /<IP_ADDRESS>:20002
2020-09-22 xx:48:00,605 INFO  master.AlluxioJobMasterProcess (AlluxioJobMasterProcess.java:startServing) - Alluxio job master version 2.3.0 started (gained leadership). bindAddress=/<IP_ADDRESS>:20001, connectAddress=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20001, webAddress=/<IP_ADDRESS>:20002
2020-09-22 xx:48:00,605 INFO  master.AlluxioJobMasterProcess (AlluxioJobMasterProcess.java:startServingRPCServer) - Starting Alluxio job master gRPC server on address /<IP_ADDRESS>:20001
2020-09-22 xx:48:00,635 INFO  master.MasterProcess (MasterProcess.java:registerServices) - registered service JOB_MASTER_CLIENT_SERVICE
2020-09-22 xx:48:00,636 INFO  master.MasterProcess (MasterProcess.java:registerServices) - registered service JOB_MASTER_WORKER_SERVICE
2020-09-22 xx:48:00,644 INFO  master.AlluxioJobMasterProcess (AlluxioJobMasterProcess.java:startServingRPCServer) - Started Alluxio job master gRPC server on address xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20001
--- [ FAILED ] The job_master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/job_master.log
<<< EOF
--- Printing the log tail for /data/alluxio/logs/job_master.out
BEGIN
--- EMPTY ---
2020-09-22 09:50:14,024 INFO  zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect) - Opening socket connection to server xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01/xx.3.67.71:2181. Will not attempt to authenticate using SASL (unknown error)
2020-09-22 09:50:14,025 INFO  zookeeper.ClientCnxn (ClientCnxn.java:primeConnection) - Socket connection established, initiating session, client: /xx.3.67.72:50484, server: xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01/xx.3.67.71:2181
2020-09-22 09:50:14,026 INFO  zookeeper.ClientCnxn (ClientCnxn.java:onConnected) - Session establishment complete on server xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01/xx.3.67.71:2181, sessionid = 0x37451f3xxcfae43, negotiated timeout = 120000
2020-09-22 09:50:14,030 INFO  state.ConnectionStateManager (ConnectionStateManager.java:postState) - State change: CONNECTED
2020-09-22 09:50:14,226 INFO  master.ZkMasterInquireClient (ZkMasterInquireClient.java:) - Creating new zookeeper client for zk@xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:2181,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:2181,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:2181/alluxio/leader
2020-09-22 09:50:14,258 INFO  network.TieredIdentityFactory (TieredIdentityFactory.java:localIdentity) - Initialized tiered identity TieredIdentity(node=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02, rack=null)
2020-09-22 09:50:14,278 INFO  extensions.ExtensionFactoryRegistry (ExtensionFactoryRegistry.java:scanLibs) - Loading core jars from /data/alluxio/lib
2020-09-22 09:50:14,308 INFO  extensions.ExtensionFactoryRegistry (ExtensionFactoryRegistry.java:scanExtensions) - Loading extension jars from /data/alluxio/extensions
2020-09-22 09:50:14,472 WARN  hdfs.HdfsUnderFileSystem (HdfsUnderFileSystem.java:) - Cannot create SupportedHdfsAclProvider. HDFS ACLs will not be supported.
2020-09-22 09:50:14,534 WARN  util.NativeCodeLoader (NativeCodeLoader.java:) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-09-22 09:50:15,034 INFO  hdfs.HdfsUnderFileSystem (HdfsUnderFileSystem.java:) - Successfully instantiated SupportedHdfsActiveSyncProvider
2020-09-22 09:50:15,043 INFO  alluxio.ProcessUtils (ProcessUtils.java:run) - Starting Alluxio job master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20001.
2020-09-22 09:50:15,045 INFO  ufs.UfsJournalCheckpointThread (UfsJournalCheckpointThread.java:runInternal) - JobMaster: Journal checkpoint thread started.
2020-09-22 xx:05:19,599 INFO  master.AbstractMaster (AbstractMaster.java:stop) - JobMaster: Stopped secondary master.
2020-09-22 xx:05:19,606 INFO  ufs.UfsJournalCheckpointThread (UfsJournalCheckpointThread.java:awaitTermination) - JobMaster: Journal checkpointer shutdown has been initiated.
2020-09-22 xx:05:20,045 INFO  ufs.UfsJournalCheckpointThread (UfsJournalCheckpointThread.java:runInternal) - JobMaster: Journal checkpoint thread has been shutdown. No new logs have been found during the quiet period.
2020-09-22 xx:05:20,046 INFO  ufs.UfsJournalCheckpointThread (UfsJournalCheckpointThread.java:awaitTermination) - JobMaster: Journal shutdown complete
2020-09-22 xx:47:18,440 INFO  network.TieredIdentityFactory (TieredIdentityFactory.java:localIdentity) - Initialized tiered identity TieredIdentity(node=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02, rack=null)
2020-09-22 xx:47:18,459 INFO  alluxio.ProcessUtils (ProcessUtils.java:run) - Starting Alluxio job master @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20001.
2020-09-22 xx:47:18,459 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Initializing Raft Journal System
2020-09-22 xx:47:18,466 INFO  raft.JournalStateMachine (JournalStateMachine.java:) - Initialized new journal state machine
2020-09-22 xx:47:18,593 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to SECONDARY
2020-09-22 xx:47:18,594 INFO  raft.RaftJournalSystem (RaftJournalSystem.java:startInternal) - Starting Raft journal system. Cluster addresses: [xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:20003, xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003, xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003]. Local address: xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003
2020-09-22 xx:47:18,835 INFO  transport.GrpcMessagingServer (GrpcMessagingServer.java:lambda$listen$0) - Successfully started messaging server at: xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003
2020-09-22 xx:47:18,846 INFO  state.ServerContext (ServerContext.java:transition) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003 - Transitioning to FOLLOWER
2020-09-22 xx:47:18,854 INFO  master.AbstractPrimarySelector (AbstractPrimarySelector.java:setState) - Primary selector transitioning to SECONDARY
2020-09-22 xx:47:18,877 INFO  network.NettyUtils (NettyUtils.java:checkNettyEpollAvailable) - EPOLL_MODE is available
2020-09-22 xx:47:34,365 INFO  state.ServerContext (ServerContext.java:setLeader) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003 - Found leader xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:20003
2020-09-22 xx:47:34,424 INFO  state.ServerStateMachine (ServerStateMachine.java:takeSnapshot) - xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:20003 - Taking snapshot 1
2020-09-22 xx:47:34,432 INFO  raft.JournalStateMachine (JournalStateMachine.java:snapshot) - Completed snapshot up to SN -1 in 5ms
<<< EOF
--- Printing the log tail for /data/alluxio/logs/job_master.out
--- EMPTY ---
--- [ FAILED ] The worker @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/worker.log
BEGIN
2020-09-22 09:50:21,255 WARN  RetryUtils - Failed to load cluster default configuration with master (attempt 2): alluxio.exception.status.UnavailableException: Failed to handshake with master xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998 to load cluster default configuration values: UNAVAILABLE: Network closed for unknown reason
2020-09-22 09:50:21,481 WARN  RetryUtils - Failed to load cluster default configuration with master (attempt 3): alluxio.exception.status.UnavailableException: Failed to handshake with master xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998 to load cluster default configuration values: UNAVAILABLE: Network closed for unknown reason
2020-09-22 09:50:21,936 WARN  RetryUtils - Failed to load cluster default configuration with master (attempt 4): alluxio.exception.status.UnavailableException: Failed to handshake with master xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998 to load cluster default configuration values: UNAVAILABLE: Network closed for unknown reason
2020-09-22 09:50:22,778 WARN  RetryUtils - Failed to load cluster default configuration with master (attempt 5): alluxio.exception.status.UnavailableException: Failed to handshake with master xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998 to load cluster default configuration values: UNAVAILABLE: Network closed for unknown reason
2020-09-22 09:50:24,403 WARN  RetryUtils - Failed to load cluster default configuration with master (attempt 6): alluxio.exception.status.UnavailableException: Failed to handshake with master xx-dev-cq-ecs-dtpbu-datalake-cdh-edge-01:19998 to load cluster default configuration values: UNAVAILABLE: Network closed for unknown reason
2020-09-22 09:50:27,868 INFO  TieredIdentityFactory - Initialized tiered identity TieredIdentity(node=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03, rack=null)
2020-09-22 09:50:28,082 INFO  BlockWorkerFactory - Creating alluxio.worker.block.BlockWorker
2020-09-22 09:50:28,193 INFO  log - Logging initialized @7965ms
2020-09-22 09:50:28,414 INFO  GrpcDataServer - Alluxio worker gRPC server started, listening on /<IP_ADDRESS>:29999
2020-09-22 09:50:28,415 INFO  ProcessUtils - Starting Alluxio worker @xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:29999.
2020-09-22 09:50:28,418 INFO  MetricsSystem - Starting sinks with config: {}.
2020-09-22 09:50:28,586 INFO  AlluxioWorkerProcess - Started Alluxio worker @xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:29999 with id<PHONE_NUMBER>525047293
2020-09-22 09:50:28,589 INFO  WebServer - Alluxio worker web service starting @ /<IP_ADDRESS>:30000
2020-09-22 09:50:28,591 INFO  Server - jetty-9.2.z-SNAPSHOT
2020-09-22 09:50:28,622 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@58670130{/metrics/prometheus,null,AVAILABLE}
2020-09-22 09:50:28,622 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@9bd0fa6{/metrics/json,null,AVAILABLE}
2020-09-22 09:50:28,623 WARN  SecurityHandler - ServletContext@o.e.j.s.ServletContextHandler@46f699d5{/,file:/data/alluxio/webui/worker/build/,STARTING} has uncovered http methods for path: /
2020-09-22 09:50:34,502 INFO  ContextHandler - Started o.e.j.s.ServletContextHandler@46f699d5{/,file:/data/alluxio/webui/worker/build/,AVAILABLE}
2020-09-22 09:50:34,506 INFO  ServerConnector - Started ServerConnector@2257fadf{HTTP/1.1}{<IP_ADDRESS>:30000}
2020-09-22 09:50:34,506 INFO  Server - Started @14278ms
2020-09-22 09:50:34,506 INFO  WebServer - Alluxio worker web service started @ /<IP_ADDRESS>:30000
2020-09-22 09:50:34,506 INFO  AlluxioWorkerProcess - Alluxio worker version 2.3.0 started. bindHost=<IP_ADDRESS>, connectHost=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03, rpcPort=29999, webPort=30000
2020-09-22 xx:05:13,373 INFO  GrpcDataServer - Shutting down Alluxio worker gRPC server at <IP_ADDRESS>/<IP_ADDRESS>:29999.
2020-09-22 xx:05:13,375 INFO  AlluxioWorkerProcess - Alluxio worker ended
2020-09-22 xx:05:13,375 INFO  ProcessUtils - Stopping Alluxio worker @xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:29999.
2020-09-22 xx:05:17,390 INFO  ServerConnector - Stopped ServerConnector@2257fadf{HTTP/1.1}{<IP_ADDRESS>:30000}
2020-09-22 xx:05:17,4xx INFO  ContextHandler - Stopped o.e.j.s.ServletContextHandler@46f699d5{/,file:/data/alluxio/webui/worker/build/,UNAVAILABLE}
2020-09-22 xx:05:17,411 INFO  ContextHandler - Stopped o.e.j.s.ServletContextHandler@9bd0fa6{/metrics/json,null,UNAVAILABLE}
2020-09-22 xx:05:17,411 INFO  ContextHandler - Stopped o.e.j.s.ServletContextHandler@58670130{/metrics/prometheus,null,UNAVAILABLE}
2020-09-22 xx:47:25,033 ERROR AlluxioWorker - Fatal error: Cannot run Alluxio worker; Unable to determine master address. Please modify alluxio-site.properties to either set alluxio.master.hostname, configure zookeeper with alluxio.zookeeper.enabled=true and alluxio.zookeeper.address=[comma-separated zookeeper master addresses], or utilize internal HA by setting alluxio.master.embedded.journal.addresses=[comma-separated alluxio master addresses]
<<< EOF
--- Printing the log tail for /data/alluxio/logs/worker.out
--- EMPTY ---
--- [ FAILED ] The job_worker @ xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03 is not serving requests.
--- Printing the log tail for /data/alluxio/logs/job_worker.log
BEGIN
2020-09-22 09:50:28,308 INFO  alluxio.ProcessUtils (ProcessUtils.java:run) - Starting Alluxio job worker.
2020-09-22 09:50:28,309 INFO  web.WebServer (WebServer.java:start) - Alluxio Job Manager Worker Web service starting @ /<IP_ADDRESS>:30003
2020-09-22 09:50:28,3xx INFO  server.Server (Server.java:doStart) - jetty-9.2.z-SNAPSHOT
2020-09-22 09:50:28,332 INFO  handler.ContextHandler (ContextHandler.java:doStart) - Started o.e.j.s.ServletContextHandler@61f05988{/metrics/json,null,AVAILABLE}
2020-09-22 09:50:28,333 WARN  security.SecurityHandler (ConstraintSecurityHandler.java:checkPathsWithUncoveredHttpMethods) - ServletContext@o.e.j.s.ServletContextHandler@613a8ee1{/,null,STARTING} has uncovered http methods for path: /
2020-09-22 09:50:31,567 INFO  handler.ContextHandler (ContextHandler.java:doStart) - Started o.e.j.s.ServletContextHandler@613a8ee1{/,null,AVAILABLE}
2020-09-22 09:50:31,572 INFO  server.ServerConnector (AbstractConnector.java:doStart) - Started ServerConnector@2d66530f{HTTP/1.1}{<IP_ADDRESS>:30003}
2020-09-22 09:50:31,572 INFO  server.Server (Server.java:doStart) - Started @7921ms
2020-09-22 09:50:31,572 INFO  web.WebServer (WebServer.java:start) - Alluxio Job Manager Worker Web service started @ /<IP_ADDRESS>:30003
2020-09-22 09:50:31,574 INFO  metrics.MetricsSystem (MetricsSystem.java:startSinksFromConfig) - Starting sinks with config: {}.
2020-09-22 09:50:31,576 INFO  imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start) - Starting
2020-09-22 09:50:31,576 INFO  zookeeper.ZooKeeper (ZooKeeper.java:) - Initiating client connection, connectString=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-01:2181,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-02:2181,xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@f9b5552
2020-09-22 09:50:31,577 INFO  zookeeper.ClientCnxnSocket (ClientCnxnSocket.java:initProperties) - jute.maxbuffer value is 4194304 Bytes
2020-09-22 09:50:31,577 INFO  zookeeper.ClientCnxn (ClientCnxn.java:initRequestTimeout) - zookeeper.request.timeout value is 0. feature enabled=
2020-09-22 09:50:31,578 INFO  imps.CuratorFrameworkImpl (CuratorFrameworkImpl.java:start) - Default schema
2020-09-22 09:50:31,578 INFO  zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect) - Opening socket connection to server xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03/xx.3.67.73:2181. Will not attempt to authenticate using SASL (unknown error)
2020-09-22 09:50:31,578 INFO  zookeeper.ClientCnxn (ClientCnxn.java:primeConnection) - Socket connection established, initiating session, client: /xx.3.67.73:38xx0, server: xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03/xx.3.67.73:2181
2020-09-22 09:50:31,580 INFO  zookeeper.ClientCnxn (ClientCnxn.java:onConnected) - Session establishment complete on server xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03/xx.3.67.73:2181, sessionid = 0x27451f3xxc3ae8b, negotiated timeout = 60000
2020-09-22 09:50:31,581 INFO  state.ConnectionStateManager (ConnectionStateManager.java:postState) - State change: CONNECTED
2020-09-22 09:51:48,134 WARN  job.JobMasterClient (AbstractClient.java:retryRPC) - RegisterWorker(address=WorkerNetAddress{host=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03, containerHost=, rpcPort=30001, dataPort=30002, webPort=30003, domainSocketPath=, tieredIdentity=null}) returned<PHONE_NUMBER>918 in 76559 ms (>=xx000 ms)
2020-09-22 09:51:48,149 INFO  worker.AlluxioJobWorkerProcess (AlluxioJobWorkerProcess.java:start) - Started Alluxio job worker with id<PHONE_NUMBER>918
2020-09-22 09:51:48,150 INFO  worker.AlluxioJobWorkerProcess (AlluxioJobWorkerProcess.java:start) - Alluxio job worker version 2.3.0 started. bindHost=/<IP_ADDRESS>:30001, connectHost=xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:30001, rpcPort=30001, webPort=30003
2020-09-22 09:51:48,150 INFO  worker.AlluxioJobWorkerProcess (AlluxioJobWorkerProcess.java:startServingRPCServer) - Starting gRPC server on address xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:30001
2020-09-22 09:51:48,182 INFO  worker.AlluxioJobWorkerProcess (AlluxioJobWorkerProcess.java:startServingRPCServer) - Started gRPC server on address xx-dev-cq-ecs-dtpbu-datalake-cdh-work-03:30001