adding 2nd node in cluster: dbase copy hang with "node not online yet"

OrientDB Version, operating system, or hardware.

v2.2.7

Operating System

[x] Linux

I have my database "cdrarch" running on a first server node, which has been started in distributed mode, and is ONLINE:
orientdb {db=cdrarch}> list servers
CONFIGURED SERVERS
+----+-----------------+------+-----------+-----------------------+-----------------+-----------------+---------------+---------------+---------+
|#   |Name             |Status|Connections|StartedOn              |Binary           |HTTP             |UsedMemory     |FreeMemory     |MaxMemory|
+----+-----------------+------+-----------+-----------------------+-----------------+-----------------+---------------+---------------+---------+
|0   |node1473107743544|ONLINE|1          |2016-09-07 10:27:24.808|<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|5.51GB (38.74%)|1.71GB (12.05%)|14.22GB  |
+----+-----------------+------+-----------+-----------------------+-----------------+-----------------+---------------+---------------+---------+

I have copied the "hazelcast.xml" file to a second server, and started that server as well.
The logs on both nodes say that they get a socket connection: extract from the logs on the second node:
2016-09-07 12:18:51:293 WARNI Assigning distributed node name: node1473243531293 [OHazelcastPlugin]

2016-09-07 12:18:52:399 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.3] Creating MulticastJoiner [Node]
2016-09-07 12:18:52:405 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.3] Address[<IP_ADDRESS>]:2434 is STARTING [LifecycleService]
...
2016-09-07 12:18:52:668 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.3] Connecting to /<IP_ADDRESS>:2434, timeout: 0, bind-any: true [InitConnectionTask]
2016-09-07 12:18:52:675 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.3] Established socket connection between /<IP_ADDRESS>:42394 and /<IP_ADDRESS>:2434 [TcpIpConnectionManager]
2016-09-07 12:18:58:692 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.3]

Members [2] {
        Member [<IP_ADDRESS>]:2434
        Member [<IP_ADDRESS>]:2434 this
}

Then the database 'GratefulDeadConcerts' gets replicated:
2016-09-07 12:19:03:593 INFO  [node1473243531293] Current node started as MASTER for database 'GratefulDeadConcerts' [OHazelcastPlugin]
2016-09-07 12:19:03:596 INFO  [node1473243531293] New distributed configuration for database: GratefulDeadConcerts (version=1)

CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+--------+-----------+----------+-----------------+
|        |           |          |     MASTER      |
|        |           |          |     ONLINE      |
+--------+-----------+----------+-----------------+
|CLUSTER |writeQuorum|readQuorum|node1473107743544|
+--------+-----------+----------+-----------------+
|*       |     1     |    1     |        X        |
|internal|     1     |    1     |                 |
+--------+-----------+----------+-----------------+

 [OHazelcastPlugin]
2016-09-07 12:19:03:596 INFO  [node1473243531293] Saving distributed configuration file for database 'GratefulDeadConcerts' to: /opt/orientdb/databases/GratefulDeadConcerts/distribut
ed-config.json [OHazelcastPlugin]
2016-09-07 12:19:03:600 INFO  [node1473243531293] Distributed servers status:

+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+
|Name              |Status  |Databases                           |Conns|StartedOn|Binary           |HTTP             |UsedMemory               |
+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+
|node1473243531293*|STARTING|                                    |0    |12:18:50 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|75.77MB/491.00MB (15.43%)|
|node1473107743544 |ONLINE  |GratefulDeadConcerts=ONLINE (MASTER)|3    |10:27:24 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|5.35GB/14.22GB (37.63%)  |
|                  |        |bvdd-test=ONLINE (MASTER)           |     |         |                 |                 |                         |
+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+
...
2016-09-07 12:19:05:102 INFO  [node1473243531293] Publishing ONLINE status for database node1473243531293.GratefulDeadConcerts... [ODistributedDatabaseImpl]
2016-09-07 12:19:05:109 INFO  [node1473243531293] New distributed configuration for database: GratefulDeadConcerts (version=31)

CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+-------------+----+-----------+----------+-----------------+-----------------+
|             |    |           |          |     MASTER      |     MASTER      |
|             |    |           |          |     ONLINE      |     ONLINE      |
+-------------+----+-----------+----------+-----------------+-----------------+
|CLUSTER      |  id|writeQuorum|readQuorum|node1473107743544|node1473243531293|
+-------------+----+-----------+----------+-----------------+-----------------+
|*            |    |     2     |    1     |        X        |        o        |
|_studio_1    |  50|     2     |    1     |        o        |        X        |
|_studio_5    |  54|     2     |    1     |        o        |        X        |
|_studio_6    |  55|     2     |    1     |        o        |        X        |
|_studio_7    |  56|     2     |    1     |        o        |        X        |
|e_4          |  21|     2     |    1     |        o        |        X        |
|e_5          |  22|     2     |    1     |        o        |        X        |
...
|written_by_7 |  40|     2     |    1     |        o        |        X        |
+-------------+----+-----------+----------+-----------------+-----------------+

 [OHazelcastPlugin]
2016-09-07 12:19:05:109 INFO  [node1473243531293] Saving distributed configuration file for database 'GratefulDeadConcerts' to: /opt/orientdb/databases/GratefulDeadConcerts/distribut
ed-config.json [OHazelcastPlugin]

So far so good, it seems.
Then it starts with my big (300GB) database "cdrarch", but here we get problems:
2016-09-07 12:19:05:119 INFO  [node1473243531293] Current node started as MASTER for database 'cdrarch' [OHazelcastPlugin]
2016-09-07 12:19:05:120 INFO  [node1473243531293] New distributed configuration for database: cdrarch (version=1)

CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+--------+-----------+----------+-----------------+
|        |           |          |     MASTER      |
|        |           |          |     ONLINE      |
+--------+-----------+----------+-----------------+
|CLUSTER |writeQuorum|readQuorum|node1473107743544|
+--------+-----------+----------+-----------------+
|*       |     1     |    1     |        X        |
|internal|     1     |    1     |                 |
+--------+-----------+----------+-----------------+
2016-09-07 12:19:05:121 INFO  [node1473243531293] Saving distributed configuration file for database 'cdrarch' to: /opt/orientdb/databases/cdrarch/distributed-config.json [OHazelcast
Plugin]
2016-09-07 12:19:05:124 INFO  [node1473243531293] Distributed servers status:

+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+
|Name              |Status  |Databases                           |Conns|StartedOn|Binary           |HTTP             |UsedMemory               |
+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+
|node1473243531293*|STARTING|                                    |0    |12:18:50 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|75.77MB/491.00MB (15.43%)|
|node1473107743544 |ONLINE  |GratefulDeadConcerts=ONLINE (MASTER)|3    |10:27:24 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|5.35GB/14.22GB (37.63%)  |
|                  |        |cdrarch=ONLINE (MASTER)             |     |         |                 |                 |                         |
|                  |        |bvdd-test=ONLINE (MASTER)           |     |         |                 |                 |                         |
+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+-------------------------+
 [OHazelcastPlugin]
2016-09-07 12:19:05:131 WARNI [node1473243531293]->[[node1473107743544]] Requesting deploy of database 'cdrarch' on local server... [OHazelcastPlugin]
2016-09-07 12:19:05:444 WARNI [node1473243531293] Moving existent database 'cdrarch' in '/opt/orientdb/databases/cdrarch' to '/opt/orientdb/databases//../backup/databases/cdrarch' an
d get a fresh copy from a remote node... [OHazelcastPlugin]
2016-09-07 12:19:05:445 SEVER [node1473243531293] Error on moving existent database 'cdrarch' located in '/opt/orientdb/databases/cdrarch' to '/opt/orientdb/databases/../backup/datab
ases/cdrarch'. Deleting old database... [OHazelcastPlugin]
2016-09-07 12:19:05:445 INFO  [node1473243531293]<-[node1473107743544] Copying remote database 'cdrarch' to: /tmp/orientdb/install_cdrarch.zip [OHazelcastPlugin]
2016-09-07 12:19:05:446 INFO  [node1473243531293]<-[node1473107743544] Installing database 'cdrarch' to: /opt/orientdb/databases/cdrarch... [OHazelcastPlugin]
2016-09-07 12:19:05:447 INFO  [node1473243531293] - writing chunk #1 offset=0 size=0b [OHazelcastPlugin]
2016-09-07 12:19:05:448 INFO  [node1473243531293] Database copied correctly, size=0b [ODistributedAbstractPlugin$3]
2016-09-07 12:19:07:476 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=1, queue=0 worker=0) [ODistributedWorker]
2016-09-07 12:19:09:476 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=2, queue=0 worker=0) [ODistributedWorker]
...

-> what's happening?

Did another (more patient) attempt:

stopped server entirely (on single node)

started (with "dserver.sh") ODB on first server node, and waited till my 300GB dbase "cdrarch" is accessible via the "console.sh" tool ; in server logs:
2016-09-08 08:34:45:767 INFO  OrientDB Server is active v2.2.7 (build 2.2.x@rdcab5af4dce4b538bdb4b372abba46e3fc9f19b7; 2016-08-11 15:17:33+0000). [OServer]

started (with "dserver.sh") ODB on second server node, and notice the logs on the 1st server which indicate that both nodes are connected and start synchronizing:

2016-09-08 08:45:15:558 INFO  [node1473107743544] Distributed servers status:
+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+------------------------+
|Name              |Status  |Databases                           |Conns|StartedOn|Binary           |HTTP             |UsedMemory              |
+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+------------------------+
|node1473243531293 |STARTING|                                    |0    |08:45:04 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|100.87MB/14.22GB (0.69%)|
|node1473107743544*|ONLINE  |GratefulDeadConcerts=ONLINE (MASTER)|1    |08:30:05 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|342.71MB/14.22GB (2.35%)|
|                  |        |cdrarch=ONLINE (MASTER)             |     |         |                 |                 |                        |
|                  |        |bvdd-test=ONLINE (MASTER)           |     |         |                 |                 |                        |
|                  |        |c5qp=ONLINE (MASTER)                |     |         |                 |                 |                        |
+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+------------------------+

a first (small) dbase "bvdd-test" gets exchanged, it seems:

2016-09-08 08:45:15:597 INFO  [node1473107743544]<-[node1473243531293] Received new status node1473243531293.bvdd-test=SYNCHRONIZING [OHazelcastPlugin]
2016-09-08 08:45:15:702 INFO  [node1473107743544] Received updated status node1473243531293.bvdd-test=SYNCHRONIZING [OHazelcastPlugin]
2016-09-08 08:45:15:704 INFO  [node1473107743544] Received updated status node1473107743544.bvdd-test=SYNCHRONIZING [OHazelcastPlugin]
2016-09-08 08:45:15:704 INFO  [node1473107743544]->[node1473243531293] Deploying database bvdd-test... [OSyncDatabaseTask]
2016-09-08 08:45:15:710 INFO  [node1473107743544]->[node1473243531293] Creating backup of database 'bvdd-test' (compressionRate=7) in directory: /tmp/orientdb/backup_bvdd-t
est.zip. LSN=OLogSequenceNumber{segment=0, position=2723661}... [OSyncDatabaseTask]
2016-09-08 08:45:16:159 INFO  [node1473107743544]->[node1473243531293] Backup of database 'bvdd-test' completed. lastOperationId=1.0... [OSyncDatabaseTask$1]
2016-09-08 08:45:16:314 INFO  [node1473107743544]->[node1473243531293] - transferring chunk #1 offset=0 size=168274... [OSyncDatabaseTask]
2016-09-08 08:45:16:315 INFO  [node1473107743544] Received updated status node1473107743544.bvdd-test=ONLINE [OHazelcastPlugin]
2016-09-08 08:45:16:317 INFO  [node1473107743544]->[node1473243531293] Deploy database task completed [OSyncDatabaseTask]
2016-09-08 08:45:16:318 INFO  [node1473107743544] Distributed servers status:
...
2016-09-08 08:45:17:931 INFO  [node1473107743544]<-[node1473243531293] Received updated status node1473243531293.bvdd-test=ONLINE [OHazelcastPlugin]
2016-09-08 08:45:17:942 INFO  [node1473107743544]<-[node1473243531293] Updated configuration db=bvdd-test [OHazelcastPlugin]
2016-09-08 08:45:17:945 INFO  [node1473107743544] New distributed configuration for database: bvdd-test (version=12)...
2016-09-08 08:45:17:946 INFO  [node1473107743544] Saving distributed configuration file for database 'bvdd-test' to: /opt/orientdb/databases/bvdd-test/distributed-config.js
on [OHazelcastPlugin]

but then this message is disturbing :
2016-09-08 08:45:17:953 INFO  [node1473107743544] Cannot install database 'bvdd-test' on local node, because no servers are ONLINE [OHazelcastPlugin]

for dbase "GratefulDeadConcerts" it seems to go all fine :
...
2016-09-08 08:45:19:568 INFO  [node1473107743544] Saving distributed configuration file for database 'GratefulDeadConcerts' to: /opt/orientdb/databases/GratefulDeadConcerts
/distributed-config.json [OHazelcastPlugin]
2016-09-08 08:45:19:574 INFO  [node1473107743544] Current node started as MASTER for database 'GratefulDeadConcerts' [OHazelcastPlugin]

for the big dbase "cdrarch", it gets stuck :

2016-09-08 08:45:19:577 INFO  [node1473107743544]<-[node1473243531293] Received new status node1473243531293.cdrarch=SYNCHRONIZING [OHazelcastPlugin]
2016-09-08 08:45:19:587 INFO  [node1473107743544] Received updated status node1473243531293.cdrarch=SYNCHRONIZING [OHazelcastPlugin]
2016-09-08 08:45:19:588 INFO  [node1473107743544] Received updated status node1473107743544.cdrarch=SYNCHRONIZING [OHazelcastPlugin]
2016-09-08 08:45:19:589 INFO  [node1473107743544]->[node1473243531293] Deploying database cdrarch... [OSyncDatabaseTask]
...
2016-09-08 08:45:19:591 INFO  [node1473107743544]->[node1473243531293] Creating backup of database 'cdrarch' (compressionRate=7) in directory: /tmp/orientdb/backup_cdrarch.
zip. LSN=OLogSequenceNumber{segment=17575, position=70007445}... [OSyncDatabaseTask]
2016-09-08 08:45:19:893 INFO  [node1473107743544]->[node1473243531293] - transferring chunk #1 offset=0 size=0... [OSyncDatabaseTask]
2016-09-08 08:45:19:894 INFO  [node1473107743544] Received updated status node1473107743544.cdrarch=ONLINE [OHazelcastPlugin]
2016-09-08 08:45:19:897 INFO  [node1473107743544]->[node1473243531293] Deploy database task completed [OSyncDatabaseTask]
2016-09-08 08:45:42:814 WARNI [node1473107743544] Timeout (20002ms) on waiting for synchronous responses from nodes=[node1473243531293] responsesSoFar=[] request=(id=0.0 ta
sk=heartbeat timestamp:<PHONE_NUMBER>803) [ODistributedDatabaseImpl]
2016-09-08 08:45:42:815 WARNI [node1473107743544]->[node1473243531293] Server 'node1473243531293' did not respond to the heartbeat message (db=GratefulDeadConcerts, timeout
=10000ms), but cannot be set OFFLINE by configuration [OClusterHealthChecker]
2016-09-08 08:45:52:818 WARNI [node1473107743544]->[node1473243531293] Server 'node1473243531293' did not respond to the heartbeat message (db=bvdd-test, timeout=10000ms),
but cannot be set OFFLINE by configuration [OClusterHealthChecker]
2016-09-08 08:45:52:828 INFO  [node1473107743544] Distributed servers status:
+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+------------------------+
|Name              |Status  |Databases                           |Conns|StartedOn|Binary           |HTTP             |UsedMemory              |
+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+------------------------+
|node1473243531293 |STARTING|                                    |0    |08:45:04 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|100.87MB/14.22GB (0.69%)|
|node1473107743544*|ONLINE  |GratefulDeadConcerts=ONLINE (MASTER)|3    |08:30:05 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|401.81MB/14.22GB (2.76%)|
|                  |        |cdrarch=ONLINE (MASTER)             |     |         |                 |                 |                        |
|                  |        |bvdd-test=ONLINE (MASTER)           |     |         |                 |                 |                        |
|                  |        |c5qp=ONLINE (MASTER)                |     |         |                 |                 |                        |
+------------------+--------+------------------------------------+-----+---------+-----------------+-----------------+------------------------+
[OHazelcastPlugin]
2016-09-08 08:46:02:828 WARNI [node1473107743544]->[node1473243531293] Server 'node1473243531293' did not respond to the heartbeat message (db=GratefulDeadConcerts, timeout
=10000ms), but cannot be set OFFLINE by configuration [OClusterHealthChecker]
2016-09-08 08:46:12:829 WARNI [node1473107743544]->[node1473243531293] Server 'node1473243531293' did not respond to the heartbeat message (db=bvdd-test, timeout=10000ms),
but cannot be set OFFLINE by configuration [OClusterHealthChecker]
...(and so on)

after about 45 mins, the situation is still the same:
2016-09-08 09:29:44:069 WARNI [node1473107743544] Timeout (10001ms) on waiting for synchronous responses from nodes=[node1473243531293] responsesSoFar=[] request=(id=0.264
task=heartbeat timestamp:<PHONE_NUMBER>067) [ODistributedDatabaseImpl]
2016-09-08 09:29:44:070 WARNI [node1473107743544]->[node1473243531293] Server 'node1473243531293' did not respond to the heartbeat message (db=GratefulDeadConcerts, timeout
=10000ms), but cannot be set OFFLINE by configuration [OClusterHealthChecker]
2016-09-08 09:29:54:070 WARNI [node1473107743544]->[node1473243531293] Server 'node1473243531293' did not respond to the heartbeat message (db=bvdd-test, timeout=10000ms),
but cannot be set OFFLINE by configuration [OClusterHealthChecker]

on that second node, the logs are this:
...
2016-09-08 09:31:33:550 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=1321, queue=1 worker=3) [ODistributedWorker]
2016-09-08 09:31:33:550 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=1361, queue=1 worker=1) [ODistributedWorker]
2016-09-08 09:31:33:550 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=1386, queue=123 worker=0) [ODistributedWorker]
... (and so on)

=> are they stuck, or is something (supposed to be) happening that simply takes an (awfull) lot of time?

Found why :

in the "default-distributed-db-config.json" file, I configured explicitly nodenames, which did not correspond with the random nodenames assigned to each server when I launched "dserver.sh"
so I stopped all servers, trashed the "distributed-db-config.json" files which were modified by the startup, set the environment variable ORIENTDB_NODE_NAME on each server, and then started "dserver.sh" again

Situation is better now (not this "Node is not online yet" loop anymore) but only the first node gets in status ONLINE, the 2nd and 3rd nodes remain in status STARTING ...

Can the logging level be increased so that hopefully any reason becomes visible why a 2nd and a 3rd node remain in status STARTING ?
The docs do not describe anything about setting logging levels ...

Seems the same issue as closed issues #4176 and #4789 ; however these scenarios seem to be a conversion from a standalone server into a distributed setup, where according to the docs the OPs copy their entire database to the new nodes.
In our case, database of 350GB, that is an unrealistic approach, and as such also documented to be an alternative way of working (i.e. not copying first the dbase).
1st node gets ONLINE,
2nd node is STARTING and is copying clusters, then remains in STARTING saying "Node is not online yet",
3rd node does exactly the same thing (remains in STARTING).

Are the 2nd and 3rd nodes starting at the same time?

@lvca  : It seems to make no difference , I did a few different combinations : start 1st, wait till connectivity with dbase is possible; then start 2nd node, see it synchronizing some clusters until it gets into that loop ("Node is not online yet"), the start 3rd node to see if that helps gettings clusters distributed, but to no avail.

Do you have logs about the last attempt with nodes that starts progressively? Could you share the log files in some way? GIST is fine too. Thanks.

@lvca, ok, I managed to upload the logfiles of the first and 2nd server to GIST:
[https://gist.github.com/rdelangh/192a9fd0c759121e305d2c5ab35dc9fa]

the logfile on GIST for the 1st server is named "orient1-orient-server.log.0"
the logfile on GIST for the 2nd server is named "orient2-orient-server.log.0"

Ok, this error should be fixed in the last 2.2.10. Could you please retry?

@lvca
Hi Luca, I tried with this version, but even the "dserver.sh" startup on a 1st server node fails. The logs are clearly indicating that it's not starting as it should be:
https://gist.github.com/rdelangh/4510549d4a2c03027af58fcace82d235

Do you have the logs of the other servers that cannot join?

@lvca the first server cannot even start, so I did not attempt to startup any additional server.

I started the first server with version 2.2.7 again, and this one starts fine. So there seems some issue with version 2.2.10

Hi @rdelangh! Regarding 2016-09-24 01:32:56:927 SEVER OCachePointer.finalize: readers != 0 [OCachePointer] entries in the latest log. Could you please enable leak detector using -Dmemory.directMemory.trackMode=true -Djava.util.logging.manager=com.orientechnologies.common.log.OLogManager$DebugLogManager and do a full start-work-stop cycle on the node, log entries marker with DIRECT-TRACK should appear in the log if a leak will be detected. BTW, are those "OCachePointer.finalize: readers != 0" entries consistently reproducible or they are random?

@taburet
These "OCachePointer.finalize: readers != 0" log entries start appearing at extreme high speed (zillions of entries, continuously) in the logfile some 30 seconds after I started the server, and before any client has been launched to use that server.

Tested locally on a primitive 3-node setup  – no leaks. Lets see what leak detector will uncover on your side.

@taburet

awaiting a timeslot that our current (v2.2.7) can be shut, will do then the startup of v2.2.10 with the leak detector settings
meanwhile I found that the v2.2.7 "server.sh" script contains options for Java, which are not included in the v2.2.10 bundled "server.sh" script, I don't know if they can be relevant for this issue:
In the v2.2.7 "server.sh" script:
JAVA_OPTS_SCRIPT="-Djna.nosys=true -XX:+HeapDumpOnOutOfMemoryError -XX:MaxDirectMemorySize=512g -Djava.awt.headless=true -
Dfile.encoding=UTF8 -Drhino.opt.level=9"

That's really strange, inspected a freshly built 2.2.10 distribution and the distribution downloaded from the OrientDB site. They both contain the same JAVA_OPTS_SCRIPT in server.sh as in 2.2.7. Maybe you just accidentally modified the file?

I see that there is actually 2 lines in the v2.2.7 "server.sh" script where JAVA_OPTS_SCRIPT is set : the first one, on line number 75, is not present in the version 2.2.10 script. It really doesn't matter, because -if not set- the variable gets set anyhow on line number 100 in the v2.2.10 script
-> ignore my comment as not relevant

BTW, just documented the leak detector: http://orientdb.com/docs/2.2.x/Leak-Detector.html , maybe you will find this information useful.

ok, got it: I had tuned the v2.2.7 "server.sh" script with extra heap and storage cache settings, which were not (of course) present in the standard v2.2.10 "server.sh" script, and apparently required to be able to launch the v2.2.10 engine for our large dbase (470GB) :
ORIENTDB_OPTS_MEMORY="-Xms1024m -Xmx16384m"
I added these tuned settings to the v2.2.10 "server.sh" script, and started it. It took quite some time (=30mins) for it to get "online" with our big dbase, but finally it succeeded:
2016-09-27 15:23:39:346 INFO  [orient1] Current node started as MASTER for database 'cdrarch' [OHazelcastPlugin$4]
2016-09-27 15:23:39:350 INFO  [orient1] New distributed configuration for database: cdrarch (version=1)
...
2016-09-27 15:59:37:101 INFO  [orient1] Reassigning cluster ownership for database cdrarch [OHazelcastPlugin]
2016-09-27 15:59:37:105 INFO  [orient1] New distributed configuration for database: cdrarch (version=1)
...
2016-09-27 15:59:37:273 INFO  [orient1] Publishing ONLINE status for database orient1.cdrarch... [ODistributedDatabaseImpl]

ok, v2.2.10 is running on a single node, now trying to add a 2nd node to this...

Hi! How it's going with the second node?

started on the first node, dbases are ONLINE:

2016-09-28 14:38:25:202 WARNI [orient1] Updated node status to 'ONLINE' [OHazelcastPlugin]
2016-09-28 14:38:25:212 INFO  [orient1] Distributed servers status:
+--------+------+------------------------------------+-----+---------+-----------------+-----------------+----------------------+
|Name    |Status|Databases                           |Conns|StartedOn|Binary           |HTTP             |UsedMemory            |
+--------+------+------------------------------------+-----+---------+-----------------+-----------------+----------------------+
|orient1*|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|0    |14:02:49 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|1.19GB/14.22GB (8.38%)|
|        |      |cdrarch=ONLINE (MASTER)             |     |         |                 |                 |                      |
|        |      |bvdd-test=ONLINE (MASTER)           |     |         |                 |                 |                      |
+--------+------+------------------------------------+-----+---------+-----------------+-----------------+----------------------+

started on the second node, but failed :
...

2016-09-28 14:55:58:733 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.5] Established socket connection between /<IP_ADDRESS>:40972 and /<IP_ADDRESS>:2434 [TcpIpConnectionManager]
2016-09-28 14:56:04:756 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.5]
Members [2] {
Member [<IP_ADDRESS>]:2434
Member [<IP_ADDRESS>]:2434 this
}
[ClusterService]
2016-09-28 14:56:06:766 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.5] Address[<IP_ADDRESS>]:2434 is STARTED [LifecycleService]
2016-09-28 14:56:06:767 INFO  Starting distributed server 'orient2' (hzID=e59aa858-a3e5-4c91-a05c-3d4a87f1d7ba)... [OHazelcastPlugin]
2016-09-28 14:56:06:830 WARNI [orient2] Servers in cluster: [orient2, orient1] [OHazelcastPlugin]
2016-09-28 14:56:06:872 INFO  [orient2] Current node started as MASTER for database 'bvdd-test' [OHazelcastPlugin]
2016-09-28 14:56:06:909 INFO  [orient2] New distributed configuration for database: bvdd-test (version=1)
CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+-------------------------+-----------+----------+-------+-------+-------+-------+-------+-------+
|                         |           |          |MASTER |MASTER |MASTER |MASTER |MASTER |MASTER |
|                         |           |          |ONLINE |OFFLINE|OFFLINE|OFFLINE|OFFLINE|OFFLINE|
+-------------------------+-----------+----------+-------+-------+-------+-------+-------+-------+
|CLUSTER                  |writeQuorum|readQuorum|orient1|orient2|orient4|orient5|orient3|orient6|
+-------------------------+-----------+----------+-------+-------+-------+-------+-------+-------+
|*                        |     2     |    1     |   X   |       |       |       |       |       |
|_studio_49               |     2     |    1     |   X   |   o   |       |       |       |       |
|studio_50               |     2     |    1     |   X   |       |       |       |   o   |       |
|clp0                  |     2     |    1     |   X   |   o   |   o   |       |       |       |
|clp1                  |     2     |    1     |       |   X   |   o   |   o   |       |       |
|clp2                  |     2     |    1     |       |       |   X   |   o   |   o   |       |
|clp3                  |     2     |    1     |       |       |       |   X   |   o   |   o   |
|clp4                  |     2     |    1     |   o   |       |       |       |   X   |   o   |
|cl_p5                  |     2     |    1     |   o   |   o   |       |       |       |   X   |
|e_49                     |     2     |    1     |   X   |   o   |       |       |       |       |
|e_50                     |     2     |    1     |   X   |       |       |       |   o   |       |
|internal                 |     2     |    1     |       |       |       |       |       |       |
|navi_idtv_dhcp_tmstb_50  |     2     |    1     |   X   |   o   |       |       |       |       |
|navi_idtv_dhcp_tmstb_51  |     2     |    1     |   X   |       |       |       |   o   |       |
|navi_idtv_transactions_49|     2     |    1     |   X   |   o   |       |       |       |       |
|navi_idtv_transactions_50|     2     |    1     |   X   |       |       |       |   o   |       |
|ofunction_2              |     2     |    1     |   X   |   o   |       |       |       |       |
|ofunction_3              |     2     |    1     |   X   |       |       |       |   o   |       |
|orole_2                  |     2     |    1     |   X   |   o   |       |       |       |       |
|orole_3                  |     2     |    1     |   X   |       |       |       |   o   |       |
|oschedule_2              |     2     |    1     |   X   |   o   |       |       |       |       |
|oschedule_3              |     2     |    1     |   X   |       |       |       |   o   |       |
|osequence_2              |     2     |    1     |   X   |   o   |       |       |       |       |
|osequence_3              |     2     |    1     |   X   |       |       |       |   o   |       |
|ouser_2                  |     2     |    1     |   X   |   o   |       |       |       |       |
|ouser_3                  |     2     |    1     |   X   |       |       |       |   o   |       |
|v_50                     |     2     |    1     |   X   |   o   |       |       |       |       |
|v_51                     |     2     |    1     |   X   |       |       |       |   o   |       |
|vcas_keys_logs_50        |     2     |    1     |   X   |   o   |       |       |       |       |
|vcas_keys_logs_51        |     2     |    1     |   X   |       |       |       |   o   |       |
+-------------------------+-----------+----------+-------+-------+-------+-------+-------+-------+
[OHazelcastPlugin]
2016-09-28 14:56:06:910 INFO  [orient2] Saving distributed configuration file for database 'bvdd-test' to: /opt/orientdb/databases/bvdd-test/distributed-config.json [OHazelcastPlugin]
2016-09-28 14:56:06:922 INFO  [orient2] Distributed servers status:
+--------+--------+-------------------------+-----+---------+-----------------+-----------------+------------------------+
|Name    |Status  |Databases                |Conns|StartedOn|Binary           |HTTP             |UsedMemory              |
+--------+--------+-------------------------+-----+---------+-----------------+-----------------+------------------------+
|orient2*|STARTING|                         |0    |14:55:56 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|107.08MB/14.22GB (0.74%)|
|orient1 |ONLINE  |bvdd-test=ONLINE (MASTER)|0    |14:02:49 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|1.25GB/14.22GB (8.80%)  |
+--------+--------+-------------------------+-----+---------+-----------------+-----------------+------------------------+
[OHazelcastPlugin]
2016-09-28 14:56:06:952 WARNI [orient2]->[[orient1]] Requesting deploy of database 'bvdd-test' on local server... [OHazelcastPlugin]
2016-09-28 14:56:07:632 WARNI [orient2] Moving existent database 'bvdd-test' in '/opt/orientdb/databases/bvdd-test' to '/opt/orientdb/databases//../backup/databases/bvdd-test' and get a fresh copy from a remote node... [OHazelcastPlugin]
2016-09-28 14:56:07:633 SEVER [orient2] Error on moving existent database 'bvdd-test' located in '/opt/orientdb/databases/bvdd-test' to '/opt/orientdb/databases/../backup/databases/bvdd-test'. Deleting old database... [OHazelcastPlugin]
2016-09-28 14:56:07:634 INFO  [orient2]<-[orient1] Copying remote database 'bvdd-test' to: /tmp/orientdb/install_bvdd-test.zip [OHazelcastPlugin]
2016-09-28 14:56:07:638 INFO  [orient2]<-[orient1] Installing database 'bvdd-test' to: /opt/orientdb/databases/bvdd-test... [OHazelcastPlugin]
2016-09-28 14:56:07:639 INFO  [orient2] - writing chunk #1 offset=0 size=184.53KB [OHazelcastPlugin]
2016-09-28 14:56:07:644 INFO  [orient2] Database copied correctly, size=184.53KB [ODistributedAbstractPlugin$3]
2016-09-28 14:56:08:904 WARNI {db=bvdd-test} Storage 'bvdd-test' was not closed properly. Will try to recover from write ahead log [OLocalPaginatedStorage]
2016-09-28 14:56:08:905 SEVER {db=bvdd-test} Restore is not possible because write ahead log is empty. [OLocalPaginatedStorage]
2016-09-28 14:56:08:906 INFO  {db=bvdd-test} Storage data recover was completed [OLocalPaginatedStorage]
2016-09-28 14:56:09:141 INFO  {db=bvdd-test} [orient2] Installed database 'bvdd-test' (LSN=OLogSequenceNumber{segment=0, position=24}) [OHazelcastPlugin]
2016-09-28 14:56:09:251 INFO  [orient2] Publishing ONLINE status for database orient2.bvdd-test... [ODistributedDatabaseImpl]
2016-09-28 14:56:09:254 INFO  [orient2] Reassigning cluster ownership for database bvdd-test [OHazelcastPlugin]
2016-09-28 14:56:09:266 INFO  [orient2] New distributed configuration for database: bvdd-test (version=1)
CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+-------------------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|                         |    |           |          |MASTER |MASTER |MASTER |MASTER |MASTER |MASTER |
|                         |    |           |          |ONLINE |ONLINE |OFFLINE|OFFLINE|OFFLINE|OFFLINE|
+-------------------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|CLUSTER                  |  id|writeQuorum|readQuorum|orient1|orient2|orient4|orient5|orient3|orient6|
+-------------------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|*                        |    |     2     |    1     |   X   |       |       |       |       |       |
|_studio_49               | 324|     2     |    1     |   X   |   o   |       |       |       |       |
|studio_50               | 336|     2     |    1     |   X   |       |       |       |   o   |       |
|clp0                  |    |     2     |    1     |   X   |   o   |   o   |       |       |       |
|clp1                  |    |     2     |    1     |       |   X   |   o   |   o   |       |       |
|clp2                  |    |     2     |    1     |       |       |   X   |   o   |   o   |       |
|clp3                  |    |     2     |    1     |       |       |       |   X   |   o   |   o   |
|clp4                  |    |     2     |    1     |   o   |       |       |       |   X   |   o   |
|cl_p5                  |    |     2     |    1     |   o   |   o   |       |       |       |   X   |
|e_49                     | 326|     2     |    1     |   X   |   o   |       |       |       |       |
|e_50                     | 339|     2     |    1     |   X   |       |       |       |   o   |       |
|internal                 |   0|     2     |    1     |       |       |       |       |       |       |
|navi_idtv_dhcp_tmstb_50  | 322|     2     |    1     |   X   |   o   |       |       |       |       |
|navi_idtv_dhcp_tmstb_51  | 333|     2     |    1     |   X   |       |       |       |   o   |       |
|navi_idtv_transactions_49| 323|     2     |    1     |   X   |   o   |       |       |       |       |
|navi_idtv_transactions_50| 335|     2     |    1     |   X   |       |       |       |   o   |       |
|ofunction_2              | 325|     2     |    1     |   X   |   o   |       |       |       |       |
|ofunction_3              | 338|     2     |    1     |   X   |       |       |       |   o   |       |
|orole_2                  | 332|     2     |    1     |   X   |   o   |       |       |       |       |
|orole_3                  | 334|     2     |    1     |   X   |       |       |       |   o   |       |
|oschedule_2              | 330|     2     |    1     |   X   |   o   |       |       |       |       |
|oschedule_3              | 342|     2     |    1     |   X   |       |       |       |   o   |       |
|osequence_2              | 327|     2     |    1     |   X   |   o   |       |       |       |       |
|osequence_3              | 340|     2     |    1     |   X   |       |       |       |   o   |       |
|ouser_2                  | 328|     2     |    1     |   X   |   o   |       |       |       |       |
|ouser_3                  | 337|     2     |    1     |   X   |       |       |       |   o   |       |
|v_50                     | 331|     2     |    1     |   X   |   o   |       |       |       |       |
|v_51                     | 343|     2     |    1     |   X   |       |       |       |   o   |       |
|vcas_keys_logs_50        | 329|     2     |    1     |   X   |   o   |       |       |       |       |
|vcas_keys_logs_51        | 341|     2     |    1     |   X   |       |       |       |   o   |       |
+-------------------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
[OHazelcastPlugin]
2016-09-28 14:56:09:269 INFO  [orient2] Distributed servers status:
+--------+--------+-------------------------+-----+---------+-----------------+-----------------+------------------------+
|Name    |Status  |Databases                |Conns|StartedOn|Binary           |HTTP             |UsedMemory              |
+--------+--------+-------------------------+-----+---------+-----------------+-----------------+------------------------+
|orient2*|STARTING|bvdd-test=ONLINE (MASTER)|0    |14:55:56 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|107.08MB/14.22GB (0.74%)|
|orient1 |ONLINE  |bvdd-test=ONLINE (MASTER)|0    |14:02:49 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|1.25GB/14.22GB (8.80%)  |
+--------+--------+-------------------------+-----+---------+-----------------+-----------------+------------------------+
[OHazelcastPlugin]
2016-09-28 14:56:09:296 INFO  [orient2] Class 'E', creation of new local cluster 'e_52' (id=-1) [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:380 INFO  [orient2] Class 'E': change mastership of cluster 'e_52' (id=355) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:380 INFO  [orient2] Class 'ORole', creation of new local cluster 'orole_5' (id=-1) [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:434 INFO  [orient2] Class 'ORole': change mastership of cluster 'orole_5' (id=356) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:435 INFO  [orient2] Class 'VCAS_KEYS_LOGS', creation of new local cluster 'vcas_keys_logs_53' (id=-1) [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:480 INFO  [orient2] Class 'VCAS_KEYS_LOGS': change mastership of cluster 'vcas_keys_logs_53' (id=357) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:482 INFO  [orient2] Class 'V', creation of new local cluster 'v_53' (id=-1) [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:525 INFO  [orient2] Class 'V': change mastership of cluster 'v_53' (id=358) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:525 INFO  [orient2] Class 'OFunction', creation of new local cluster 'ofunction_5' (id=-1) [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:569 INFO  [orient2] Class 'OFunction': change mastership of cluster 'ofunction_5' (id=359) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:570 INFO  [orient2] Class 'navi_idtv_dhcp_tmstb', creation of new local cluster 'navi_idtv_dhcp_tmstb_53' (id=-1) [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:606 INFO  [orient2] Class 'navi_idtv_dhcp_tmstb': change mastership of cluster 'navi_idtv_dhcp_tmstb_53' (id=360) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:607 INFO  [orient2] Class '_studio': change mastership of cluster '_studio_49' (id=324) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:608 INFO  [orient2] Class 'OUser': change mastership of cluster 'ouser_2' (id=328) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:609 INFO  [orient2] Class 'OSchedule', creation of new local cluster 'oschedule_5' (id=-1) [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:654 INFO  [orient2] Class 'OSchedule': change mastership of cluster 'oschedule_5' (id=361) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:655 INFO  [orient2] Class 'navi_idtv_transactions', creation of new local cluster 'navi_idtv_transactions_52' (id=-1) [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:697 INFO  [orient2] Class 'navi_idtv_transactions': change mastership of cluster 'navi_idtv_transactions_52' (id=362) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:698 INFO  [orient2] Class 'OSequence', creation of new local cluster 'osequence_5' (id=-1) [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:739 INFO  [orient2] Class 'OSequence': change mastership of cluster 'osequence_5' (id=363) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
2016-09-28 14:56:09:748 INFO  [orient2] New distributed configuration for database: bvdd-test (version=12)
CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+-------------------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|                         |    |           |          |MASTER |MASTER |MASTER |MASTER |MASTER |MASTER |
|                         |    |           |          |ONLINE |ONLINE |OFFLINE|OFFLINE|OFFLINE|OFFLINE|
+-------------------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|CLUSTER                  |  id|writeQuorum|readQuorum|orient1|orient2|orient4|orient5|orient3|orient6|
+-------------------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|*                        |    |     2     |    1     |   X   |       |       |       |       |       |
|_studio_49               | 324|     2     |    1     |   o   |   X   |       |       |       |       |
|studio_50               | 336|     2     |    1     |   X   |       |       |       |   o   |       |
|clp0                  |    |     2     |    1     |   X   |   o   |   o   |       |       |       |
|clp1                  |    |     2     |    1     |       |   X   |   o   |   o   |       |       |
|clp2                  |    |     2     |    1     |       |       |   X   |   o   |   o   |       |
|clp3                  |    |     2     |    1     |       |       |       |   X   |   o   |   o   |
|clp4                  |    |     2     |    1     |   o   |       |       |       |   X   |   o   |
|cl_p5                  |    |     2     |    1     |   o   |   o   |       |       |       |   X   |
|e_49                     | 326|     2     |    1     |   X   |   o   |       |       |       |       |
|e_50                     | 339|     2     |    1     |   X   |       |       |       |   o   |       |
|e_52                     | 355|     2     |    1     |   o   |   X   |       |       |       |       |
|internal                 |   0|     2     |    1     |       |       |       |       |       |       |
|navi_idtv_dhcp_tmstb_50  | 322|     2     |    1     |   X   |   o   |       |       |       |       |
|navi_idtv_dhcp_tmstb_51  | 333|     2     |    1     |   X   |       |       |       |   o   |       |
|navi_idtv_dhcp_tmstb_53  | 360|     2     |    1     |   o   |   X   |       |       |       |       |
|navi_idtv_transactions_49| 323|     2     |    1     |   X   |   o   |       |       |       |       |
|navi_idtv_transactions_50| 335|     2     |    1     |   X   |       |       |       |   o   |       |
|navi_idtv_transactions_52| 362|     2     |    1     |   o   |   X   |       |       |       |       |
|ofunction_2              | 325|     2     |    1     |   X   |   o   |       |       |       |       |
|ofunction_3              | 338|     2     |    1     |   X   |       |       |       |   o   |       |
|ofunction_5              | 359|     2     |    1     |   o   |   X   |       |       |       |       |
|orole_2                  | 332|     2     |    1     |   X   |   o   |       |       |       |       |
|orole_3                  | 334|     2     |    1     |   X   |       |       |       |   o   |       |
|orole_5                  | 356|     2     |    1     |   o   |   X   |       |       |       |       |
|oschedule_2              | 330|     2     |    1     |   X   |   o   |       |       |       |       |
|oschedule_3              | 342|     2     |    1     |   X   |       |       |       |   o   |       |
|oschedule_5              | 361|     2     |    1     |   o   |   X   |       |       |       |       |
|osequence_2              | 327|     2     |    1     |   X   |   o   |       |       |       |       |
|osequence_3              | 340|     2     |    1     |   X   |       |       |       |   o   |       |
|osequence_5              | 363|     2     |    1     |   o   |   X   |       |       |       |       |
|ouser_2                  | 328|     2     |    1     |   o   |   X   |       |       |       |       |
|ouser_3                  | 337|     2     |    1     |   X   |       |       |       |   o   |       |
|v_50                     | 331|     2     |    1     |   X   |   o   |       |       |       |       |
|v_51                     | 343|     2     |    1     |   X   |       |       |       |   o   |       |
|v_53                     | 358|     2     |    1     |   o   |   X   |       |       |       |       |
|vcas_keys_logs_50        | 329|     2     |    1     |   X   |   o   |       |       |       |       |
|vcas_keys_logs_51        | 341|     2     |    1     |   X   |       |       |       |   o   |       |
|vcas_keys_logs_53        | 357|     2     |    1     |   o   |   X   |       |       |       |       |
+-------------------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
[OHazelcastPlugin]
2016-09-28 14:56:09:749 INFO  [orient2] Saving distributed configuration file for database 'bvdd-test' to: /opt/orientdb/databases/bvdd-test/distributed-config.json [OHazelcastPlugin][orient2] Error on starting distributed plugin
com.orientechnologies.orient.server.distributed.ODistributedException: Quorum (1) cannot be reached because it is major than available nodes (0)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.calculateQuorum(ODistributedDatabaseImpl.java:785)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.send2Nodes(ODistributedDatabaseImpl.java:350)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.sendRequest(ODistributedAbstractPlugin.java:619)
at com.orientechnologies.orient.server.distributed.sql.OCommandExecutorSQLHASyncCluster.replaceCluster(OCommandExecutorSQLHASyncCluster.java:160)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseFromNetwork(ODistributedAbstractPlugin.java:1376)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestDatabaseFullSync(ODistributedAbstractPlugin.java:1108)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestFullDatabase(ODistributedAbstractPlugin.java:932)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:903)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabasesFromCluster(OHazelcastPlugin.java:1066)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:207)
at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:1208)
at com.orientechnologies.orient.server.OServer.activate(OServer.java:391)
at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41)
2016-09-28 14:56:14:919 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=1, queue=0 worker=0) [ODistributedWorker]
2016-09-28 14:56:16:234 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.5] Ignoring join check from Address[<IP_ADDRESS>]:2434, because this node is not master... [JoinCheckOperation]
2016-09-28 14:56:16:920 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=2, queue=0 worker=0) [ODistributedWorker]
2016-09-28 14:56:18:920 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=3, queue=0 worker=0) [ODistributedWorker]
2016-09-28 14:56:20:921 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=4, queue=0 worker=0) [ODistributedWorker]
2016-09-28 14:56:22:922 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=5, queue=0 worker=0) [ODistributedWorker]
2016-09-28 14:56:24:922 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=6, queue=0 worker=0) [ODistributedWorker]^C

Created a separate issue for the cache pointers problem. Could you please share the database schema in it, so we will be able to generate a database of the similar structure and size.
Handling you back to Luca.

The problem here is simple: you have 6 servers, so the quorum is 4, but you have only 2 servers running and the commands of creating additional clusters won't succeed. Could you please remove the other servers with:
HA REMOVE SERVER orient3
HA REMOVE SERVER orient4
HA REMOVE SERVER orient5
HA REMOVE SERVER orient6
And retry? Once server 2 is connected you can start the other servers with no problem. At that point, you can try killing servers to check everything is self-recovering.

@lvca
where do I run these "HA REMOVE..." commands? Not in "console.sh" apparently.

sorry, found it: need to run "console.sh" as "root" user. Is this documented ? I searched the online docs but could not find anything about this HA command or that it would need to be executed from "console.sh", and/or with "root" UNIX user

@lvca
I did this removal of servers, for each database (GratefulDeadConcerts, bvdd-test, cdrarch), which helped because now the 'sync' is attempting only between the only 2 servers in the picture, i.e. "orient1" and "orient2".
Started the first server "orient1", and had to wait for 30min before the databases were ONLINE...
Then started "orient2", saw succesfull 'handshakes' up till the point where (extract from logs of "orient1"):
2016-09-30 00:09:02:578 INFO  [orient1]<-[orient2] Received new status orient2.cdrarch=SYNCHRONIZING [OHazelcastPlugin]
2016-09-30 00:09:02:607 INFO  [orient1]->[orient2] Deploying database cdrarch... [OSyncDatabaseTask]
2016-09-30 00:09:02:614 INFO  [orient1]->[orient2] Creating backup of database 'cdrarch' (compressionRate=7) in directory: /tmp/orientdb/backup_cdrarch.zip. LSN=OLogSeq
uenceNumber{segment=20518, position=122323395}...  [OSyncDatabaseTask]
2016-09-30 00:09:02:917 INFO  [orient1]->[orient2] Deploy database task completed [OSyncDatabaseTask]
However, the server "orient2" remains still in STARTING status, even after waiting 8 hours after this startup:
orientdb {db=GratefulDeadConcerts}> list servers
CONFIGURED SERVERS
+----+-------+--------+-----------+-------------------+-----------------+-----------------+----------------+----------------+---------+
|#   |Name   |Status  |Connections|StartedOn          |Binary           |HTTP             |UsedMemory      |FreeMemory      |MaxMemory|
+----+-------+--------+-----------+-------------------+-----------------+-----------------+----------------+----------------+---------+
|0   |orient2|STARTING|0          |2016-09-29 22:08:52|<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|108.54MB (0.75%)|991.96MB (6.81%)|14.22GB  |
|1   |orient1|ONLINE  |5          |2016-09-29 20:02:33|<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|1.46GB (10.26%) |1.27GB (8.94%)  |14.22GB  |
+----+-------+--------+-----------+-------------------+-----------------+-----------------+----------------+----------------+---------+
The logs of "orient2" show this:
...
2016-09-30 00:09:02:231 WARNI [orient2]->[[orient1]] Requesting deploy of database 'cdrarch' on local server... [OHazelcastPlugin]
2016-09-30 00:09:02:565 WARNI [orient2] Saving clusters [cl_*_p1] to directory '/opt/orientdb/orientdb-community-2.2.10/temp/db_cdrarch' to be replaced after distributed full backup... [OHazelcastPlugin]
2016-09-30 00:09:02:566 WARNI [orient2] Moving existent database 'cdrarch' in '/opt/orientdb/databases/cdrarch' to '/opt/orientdb/databases//../backup/databases/cdrarch' and get a fresh copy from a remote node... [OHazelcastPlugin]
2016-09-30 00:09:02:567 SEVER [orient2] Error on moving existent database 'cdrarch' located in '/opt/orientdb/databases/cdrarch' to '/opt/orientdb/databases/../backup/databases/cdrarch'. Deleting old database... [OHazelcastPlugin]
2016-09-30 00:09:02:569 INFO  [orient2]<-[orient1] Copying remote database 'cdrarch' to: /tmp/orientdb/install_cdrarch.zip [OHazelcastPlugin]
2016-09-30 00:09:02:573 INFO  [orient2]<-[orient1] Installing database 'cdrarch' to: /opt/orientdb/databases/cdrarch... [OHazelcastPlugin]
2016-09-30 00:09:02:573 INFO  [orient2] - writing chunk #1 offset=0 size=0b [OHazelcastPlugin]
2016-09-30 00:09:02:577 INFO  [orient2] Database copied correctly, size=0b [ODistributedAbstractPlugin$3]
Maybe this is not correct: "Database copied correctly, size=0b" ?

HA commands need server's root user or database admin. It's a super privilege. You're right, in our docs we missed to report which permission is required. I'm opening a new issue with that.
About the 0kb is suspect. Could you please check in the origin server if the database folder is empty?

@lvca
Luca, the database folder in the origin server is far from empty ;-)
$ cd /opt/orientdb/databases/
root@orient1:/opt/orientdb/databases$ du -sh *
7.5M    bvdd-test
469G    cdrarch
4.9M    GratefulDeadConcerts
1.8M    OSystem

@lvca
Hi Luca, any idea about the "0kb" update message during the 'sync' of 2nd server from 1st server in cluster?
Is this a relevant issue for the fact that this sync is not happening and the 2nd server remains in status STARTING ?

It could be a problem of the delta backup. Could you please try to remove the file distributed-sync.json in all the servers under the database directory?

@lvca
That was already done before, no copies of that file exist on either server ("orient1", "orient2") in their "databases" directory.
Maybe there is something wrong with my assumptions of what should be in the config files?
This is what I changed in the "hazelcast.xml" file that I have in the "config" dir of both servers:

cdrcontrols
... (something random) ...

<IP_ADDRESS>
2434

                  <tcp-ip enabled="true">
                          <member><IP_ADDRESS></member>
                          <member><IP_ADDRESS></member>
                          <member><IP_ADDRESS></member>
                          <member><IP_ADDRESS></member>
                          <member><IP_ADDRESS></member>
                          <member><IP_ADDRESS></member>
                  </tcp-ip>

          </join>

This is what I have in the "default-distributed-db-config.json" files:
{
"autoDeploy": true,
"readQuorum": 1,
"writeQuorum": 2,
"executionMode": "undefined",
"readYourWrites": true,
"servers": {
"": "master"
},
"clusters": {
"internal": {
},
"cl_p0": { "servers": [ "orient1", "orient2", "orient4" ] },
"clp1": { "servers": [ "orient2", "orient4", "orient5" ] },
"clp2": { "servers": [ "orient4", "orient5", "orient3" ] },
"clp3": { "servers": [ "orient5", "orient3", "orient6" ] },
"clp4": { "servers": [ "orient3", "orient6", "orient1" ] },
"cl_p5": { "servers": [ "orient6", "orient1", "orient2" ] },
"": { "servers": ["orient1"]}
}
}
For example, is the value that I put for "writeQuorum" (=2) correct? Since we define groups of 3 servers for each pattern of clusternames, we want majority of such group (i.e. 2) to confirm that a write was completed.
-> will that block the startup of a 2nd server node?

As a test, I changed my "default-distributed-db-config.json" file, setting
"writeQuorum": "majority",
instead of 2
This seems to work :
in logs of 2nd server :
2016-10-06 16:29:34:780 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.5] Address[<IP_ADDRESS>]:2434 is STARTED [LifecycleService]
2016-10-06 16:29:34:781 INFO  Starting distributed server 'orient2' (hzID=9faa1068-c189-4d94-8e7f-f3611ea68ff1)... [OHazelcastPlugin]
2016-10-06 16:29:34:869 WARNI [orient2] Servers in cluster: [orient2, orient1] [OHazelcastPlugin]
2016-10-06 16:29:34:910 INFO  [orient2] Current node started as MASTER for database 'navi' [OHazelcastPlugin]
2016-10-06 16:29:34:937 INFO  [orient2] New distributed configuration for database: navi (version=1)
CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+--------+-----------+----------+-------+-------+-------+-------+-------+-------+
|        |           |          |MASTER |MASTER |MASTER |MASTER |MASTER |MASTER |
|        |           |          |ONLINE |OFFLINE|OFFLINE|OFFLINE|OFFLINE|OFFLINE|
+--------+-----------+----------+-------+-------+-------+-------+-------+-------+
|CLUSTER |writeQuorum|readQuorum|orient1|orient2|orient4|orient5|orient3|orient6|
+--------+-----------+----------+-------+-------+-------+-------+-------+-------+
|*       |     1     |    1     |   X   |       |       |       |       |       |
|cl_p0 |     1     |    1     |   X   |   o   |   o   |       |       |       |
|clp1 |     1     |    1     |       |   X   |   o   |   o   |       |       |
|clp2 |     1     |    1     |       |       |   X   |   o   |   o   |       |
|clp3 |     1     |    1     |       |       |       |   X   |   o   |   o   |
|clp4 |     1     |    1     |   o   |       |       |       |   X   |   o   |
|cl_p5 |     1     |    1     |   o   |   o   |       |       |       |   X   |
|internal|     1     |    1     |       |       |       |       |       |       |
+--------+-----------+----------+-------+-------+-------+-------+-------+-------+
2016-10-06 16:29:34:937 INFO  [orient2] Saving distributed configuration file for database 'navi' to: /opt/orientdb/orientdb-community-2.2.10/databases/navi/distributed-config.json [OHazelcastPlugin]
2016-10-06 16:29:34:943 INFO  [orient2] Distributed servers status:
+--------+--------+--------------------+-----+---------+-----------------+-----------------+------------------------+
|Name    |Status  |Databases           |Conns|StartedOn|Binary           |HTTP             |UsedMemory              |
+--------+--------+--------------------+-----+---------+-----------------+-----------------+------------------------+
|orient2*|STARTING|                    |0    |16:29:24 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|113.76MB/14.22GB (0.78%)|
|orient1 |ONLINE  |navi=ONLINE (MASTER)|1    |15:52:11 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|1.20GB/14.22GB (8.43%)  |
+--------+--------+--------------------+-----+---------+-----------------+-----------------+------------------------+
[OHazelcastPlugin]
2016-10-06 16:29:34:968 WARNI [orient2]->[[orient1]] Requesting deploy of database 'navi' on local server... [OHazelcastPlugin]
2016-10-06 16:29:37:247 WARNI [orient2] Moving existent database 'navi' in '/opt/orientdb/orientdb-community-2.2.10/databases/navi' to '/opt/orientdb/orientdb-community-2.2.10/databases//../backup/databases/navi' and get a fresh copy from a remote node... [OHazelcastPlugin]
2016-10-06 16:29:37:248 INFO  [orient2]<-[orient1] Copying remote database 'navi' to: /tmp/orientdb/install_navi.zip [OHazelcastPlugin]
2016-10-06 16:29:37:251 INFO  [orient2]<-[orient1] Installing database 'navi' to: /opt/orientdb/orientdb-community-2.2.10/databases/navi... [OHazelcastPlugin]
2016-10-06 16:29:37:252 INFO  [orient2] - writing chunk #1 offset=0 size=8.00MB [OHazelcastPlugin]
2016-10-06 16:29:38:254 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.5] Ignoring join check from Address[<IP_ADDRESS>]:2434, because this node is not master... [JoinCheckOperation]
2016-10-06 16:29:39:751 INFO  [orient2] - writing chunk #2 offset=8388406 size=8.00MB [OHazelcastPlugin]
2016-10-06 16:29:41:748 INFO  [orient2] - writing chunk #3 offset=16776904 size=8.00MB [OHazelcastPlugin]
2016-10-06 16:29:43:659 INFO  [orient2] - writing chunk #4 offset=25165427 size=8.00MB [OHazelcastPlugin]
...
So finally there is exchange happening of data...
Was is my incorrect interpretation of the (sorry, my complaint: limited) documentation about the "writeQuorum" parameter effect on the server behaviour ?
I want this "writeQuorum" to change finally back to 2, not "majority", when running on 6 servers.
"majority" would mean 3 or more servers out of the 6 servers. I want only 2 servers to acknowledge a write operation.

Still issues!
(logs of 2nd server):
...
2016-10-06 16:36:15:877 INFO  [orient2] - writing chunk #152 offset=1266639481 size=1.60MB [OHazelcastPlugin]
2016-10-06 16:36:15:888 INFO  [orient2] - writing chunk #153 offset=1268313781 size=0b [OHazelcastPlugin]
2016-10-06 16:36:15:903 INFO  [orient2] Database copied correctly, size=1.18GB [ODistributedAbstractPlugin$3]
2016-10-06 16:36:16:459 WARNI {db=navi} Storage 'navi' was not closed properly. Will try to recover from write ahead log [OLocalPaginatedStorage]
2016-10-06 16:36:16:459 SEVER {db=navi} Restore is not possible because write ahead log is empty. [OLocalPaginatedStorage]
2016-10-06 16:36:16:460 INFO  {db=navi} Storage data recover was completed [OLocalPaginatedStorage]
2016-10-06 16:38:16:569 INFO  {db=navi} [orient2] Installed database 'navi' (LSN=OLogSequenceNumber{segment=0, position=24}) [OHazelcastPlugin]
2016-10-06 16:38:16:702 INFO  [orient2] Publishing ONLINE status for database orient2.navi... [ODistributedDatabaseImpl]
2016-10-06 16:38:16:705 INFO  [orient2] Reassigning cluster ownership for database navi [OHazelcastPlugin]
2016-10-06 16:38:16:712 INFO  [orient2] New distributed configuration for database: navi (version=1)
CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+--------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|        |    |           |          |MASTER |MASTER |MASTER |MASTER |MASTER |MASTER |
|        |    |           |          |ONLINE |ONLINE |OFFLINE|OFFLINE|OFFLINE|OFFLINE|
+--------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|CLUSTER |  id|writeQuorum|readQuorum|orient1|orient2|orient4|orient5|orient3|orient6|
+--------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|*       |    |     2     |    1     |   X   |       |       |       |       |       |
|cl_p0 |    |     2     |    1     |   X   |   o   |   o   |       |       |       |
|clp1 |    |     2     |    1     |       |   X   |   o   |   o   |       |       |
|clp2 |    |     2     |    1     |       |       |   X   |   o   |   o   |       |
|clp3 |    |     2     |    1     |       |       |       |   X   |   o   |   o   |
|clp4 |    |     2     |    1     |   o   |       |       |       |   X   |   o   |
|cl_p5 |    |     2     |    1     |   o   |   o   |       |       |       |   X   |
|internal|   0|     2     |    1     |       |       |       |       |       |       |
+--------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
[OHazelcastPlugin]
2016-10-06 16:38:16:714 INFO  [orient2] Distributed servers status:
+--------+--------+--------------------+-----+---------+-----------------+-----------------+------------------------+
|Name    |Status  |Databases           |Conns|StartedOn|Binary           |HTTP             |UsedMemory              |
+--------+--------+--------------------+-----+---------+-----------------+-----------------+------------------------+
|orient2*|STARTING|navi=ONLINE (MASTER)|0    |16:29:24 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|113.76MB/14.22GB (0.78%)|
|orient1 |ONLINE  |navi=ONLINE (MASTER)|3    |15:52:11 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|2.53GB/14.22GB (17.80%) |
+--------+--------+--------------------+-----+---------+-----------------+-----------------+------------------------+
[OHazelcastPlugin]
2016-10-06 16:38:16:727 INFO  [orient2] Class 'OSequence', creation of new local cluster 'osequence_0' (id=-1) [ODefaultClusterOwnershipAssignmentStrat                               egy]
2016-10-06 16:38:16:840 INFO  [orient2] Class 'OSequence': change mastership of cluster 'osequence_0' (id=249) to node 'orient2' [ODefaultClusterOwners                               hipAssignmentStrategy]
...
2016-10-06 16:38:17:209 INFO  [orient2] Class 'V', creation of new local cluster 'v_0' (id=-1) [ODefaultClusterOwnershipAssignmentStrategy]
2016-10-06 16:38:17:241 INFO  [orient2] Class 'V': change mastership of cluster 'v_0' (id=258) to node 'orient2' [ODefaultClusterOwnershipAssignmentStr                               ategy]
2016-10-06 16:38:17:247 INFO  [orient2] New distributed configuration for database: navi (version=11)
CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+-----------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|                 |    |           |          |MASTER |MASTER |MASTER |MASTER |MASTER |MASTER |
|                 |    |           |          |ONLINE |ONLINE |OFFLINE|OFFLINE|OFFLINE|OFFLINE|
+-----------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|CLUSTER          |  id|writeQuorum|readQuorum|orient1|orient2|orient4|orient5|orient3|orient6|
+-----------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|*                |    |     2     |    1     |   X   |       |       |       |       |       |
|studio_0        | 253|     2     |    1     |   o   |   X   |       |       |       |       |
|clp0          |    |     2     |    1     |   X   |   o   |   o   |       |       |       |
|clp1          |    |     2     |    1     |       |   X   |   o   |   o   |       |       |
|clp2          |    |     2     |    1     |       |       |   X   |   o   |   o   |       |
|clp3          |    |     2     |    1     |       |       |       |   X   |   o   |   o   |
|clp4          |    |     2     |    1     |   o   |       |       |       |   X   |   o   |
|cl_p5          |    |     2     |    1     |   o   |   o   |       |       |       |   X   |
|e_0              | 252|     2     |    1     |   o   |   X   |       |       |       |       |
|idtv_bannerlog_0 | 250|     2     |    1     |   o   |   X   |       |       |       |       |
|idtv_dhcp_tmstb_0| 251|     2     |    1     |   o   |   X   |       |       |       |       |
|internal         |   0|     2     |    1     |       |       |       |       |       |       |
|ofunction_0      | 255|     2     |    1     |   o   |   X   |       |       |       |       |
|orole_0          | 257|     2     |    1     |   o   |   X   |       |       |       |       |
|oschedule_0      | 254|     2     |    1     |   o   |   X   |       |       |       |       |
|osequence_0      | 249|     2     |    1     |   o   |   X   |       |       |       |       |
|ouser_0          | 256|     2     |    1     |   o   |   X   |       |       |       |       |
|v_0              | 258|     2     |    1     |   o   |   X   |       |       |       |       |
+-----------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
[OHazelcastPlugin]
2016-10-06 16:38:17:248 INFO  [orient2] Saving distributed configuration file for database 'navi' to: /opt/orientdb/orientdb-community-2.2.10/databases/navi/distributed-config.json [OHazelcastPlugin][orient2] Error on starting distributed plugin
com.orientechnologies.orient.server.distributed.ODistributedException: Quorum (1) cannot be reached because it is major than available nodes (0)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.calculateQuorum(ODistributedDatabaseImpl.java:785)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.send2Nodes(ODistributedDatabaseImpl.java:350)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.sendRequest(ODistributedAbstractPlugin.java:619)
at com.orientechnologies.orient.server.distributed.sql.OCommandExecutorSQLHASyncCluster.replaceCluster(OCommandExecutorSQLHASyncCluster.java:160)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseFromNetwork(ODistributedAbstractPlugin.java:1376)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestDatabaseFullSync(ODistributedAbstractPlugin.java:1108)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestFullDatabase(ODistributedAbstractPlugin.java:932)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:903)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabasesFromCluster(OHazelcastPlugin.java:1066)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:207)
at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:1208)
at com.orientechnologies.orient.server.OServer.activate(OServer.java:391)
at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41)
2016-10-06 16:38:22:595 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=1, queue=0 worker=0) [ODistributedWorker]
2016-10-06 16:38:24:596 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=2, queue=0 worker=0) [ODistributedWorker]
...
(again in the endless waiting loop that blocks the joining of this 2nd server)

let's try with
"writeQuorum" : 1
...
(shutting down, then restart, will take again 40 mins of consistency check, see #6762 )

Same effect: first dbase starts being replicated from 1st server to 2nd server, but then:
(logs on 2nd server)
...
[OHazelcastPlugin]
2016-10-06 17:03:37:701 INFO  [orient2] Saving distributed configuration file for database 'navi' to: /opt/orientdb/orientdb-community-2.2.10/databases/navi/distributed-config.json [OHazelcastPlugin][orient2] Error on starting distributed plugin
com.orientechnologies.orient.server.distributed.ODistributedException: Quorum (1) cannot be reached because it is major than available nodes (0)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.calculateQuorum(ODistributedDatabaseImpl.java:785)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.send2Nodes(ODistributedDatabaseImpl.java:350)
...
at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41)
2016-10-06 17:03:39:036 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=1, queue=0 worker=0) [ODistributedWorker]
2016-10-06 17:03:41:037 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=2, queue=0 worker=0) [ODistributedWorker]
2016-10-06 17:03:43:037 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=3, queue=0 worker=0) [ODistributedWorker]
...

@lvca
hi Luca, about this message:
com.orientechnologies.orient.server.distributed.ODistributedException: Quorum (1) cannot be reached because it is major than available nodes (0)
is this a bug, that the ONLINE 1st server node is not recognized as such by the STARTING 2nd node ?

Hi @rdelangh I think I found the problem. Look at your table:
|* | | 2 | 1 | X | | | | | |

The default * is present only on the first server.  This means in your distributed-config.json configuration file you have something like:
servers: ["orient1"]

But you should have:
servers: ["orient1", "<NEW_NODE>"]

I realize this is not clear in the documentation, I'm working to explain it better or to bypass checking the * for the server deployment. Could you please test my theory first?

@lvca ok, I will give that a try with only the demo databases, because effectively this behaviour is NOT what we would want to achieve : any new node which joins the group would get a copy of the ENTIRE , huge database. Not only once, but would be kept in sync with the other servers. This implies massive network traffic between all the servers.
That is why I would like to configure the distribution of the clusters with the groups of 3 servers (and not all of them) for each pattern of cluster names.
Will do that test somewhere today...

@lvca no, this does not change the behaviour:

stopped all nodes, changed their "config/default-distributed-db-config.json"
on 1st node, removed the "databases/distributed*" files and moved the subdirs "databases/cdrarch" (540GB) and "databases/navi" (6GB) out of the way, and started "bin/dserver.sh" the 1st node
when 1st node alive and ONLINE, started the 2nd node

Logs on 2nd node:
...
2016-10-09 10:39:17:304 INFO  {db=GratefulDeadConcerts} [orient2] Installed database 'GratefulDeadConcerts' (LSN=OLogSequenceNumber{segment=0, position=24}) [OHazelcastPlugin]
2016-10-09 10:39:17:331 INFO  [orient2] Publishing ONLINE status for database orient2.GratefulDeadConcerts... [ODistributedDatabaseImpl]
2016-10-09 10:39:17:333 INFO  [orient2] Reassigning cluster ownership for database GratefulDeadConcerts [OHazelcastPlugin]
2016-10-09 10:39:17:340 INFO  [orient2] New distributed configuration for database: GratefulDeadConcerts (version=89)
CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+-------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|             |    |           |          |MASTER |MASTER |MASTER |MASTER |MASTER |MASTER |
|             |    |           |          |ONLINE |ONLINE |OFFLINE|OFFLINE|OFFLINE|OFFLINE|
+-------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|CLUSTER      |  id|writeQuorum|readQuorum|orient1|orient2|orient4|orient5|orient3|orient6|
+-------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
|*            |    |     1     |    1     |   X   |   o   |       |       |       |       |
|_studio_0    |  62|     1     |    1     |   o   |   X   |       |       |       |       |
|_studio_1    |  50|     1     |    1     |   o   |   X   |       |       |       |       |
|_studio_6    |  55|     1     |    1     |   o   |   X   |       |       |       |       |
|studio_7    |  56|     1     |    1     |   o   |   X   |       |       |       |       |
|clp0      |    |     1     |    1     |   X   |   o   |   o   |       |       |       |
|clp1      |    |     1     |    1     |       |   X   |   o   |   o   |       |       |
|clp2      |    |     1     |    1     |       |       |   X   |   o   |   o   |       |
|clp3      |    |     1     |    1     |       |       |       |   X   |   o   |   o   |
|clp4      |    |     1     |    1     |   o   |       |       |       |   X   |   o   |
|cl_p5      |    |     1     |    1     |   o   |   o   |       |       |       |   X   |
|e_4          |  21|     1     |    1     |   o   |   X   |       |       |       |       |
|e_5          |  22|     1     |    1     |   o   |   X   |       |       |       |       |
|e_6          |  23|     1     |    1     |   o   |   X   |       |       |       |       |
|e_7          |  24|     1     |    1     |   o   |   X   |       |       |       |       |
|followed_by_0|  67|     1     |    1     |   o   |   X   |       |       |       |       |
|followed_by_1|  26|     1     |    1     |   o   |   X   |       |       |       |       |
|followed_by_3|  28|     1     |    1     |   o   |   X   |       |       |       |       |
|followed_by_4|  29|     1     |    1     |   o   |   X   |       |       |       |       |
|internal     |   0|     1     |    1     |       |       |       |       |       |       |
|ofunction_0  |  60|     1     |    1     |   o   |   X   |       |       |       |       |
|orole_1      |  71|     1     |    1     |   o   |   X   |       |       |       |       |
|oschedule_0  |  58|     1     |    1     |   o   |   X   |       |       |       |       |
|osequence_0  |  59|     1     |    1     |   o   |   X   |       |       |       |       |
|ouser_1      |  68|     1     |    1     |   o   |   X   |       |       |       |       |
|sung_by_0    |  69|     1     |    1     |   o   |   X   |       |       |       |       |
|sung_by_1    |  42|     1     |    1     |   o   |   X   |       |       |       |       |
|sung_by_2    |  43|     1     |    1     |   o   |   X   |       |       |       |       |
|sung_by_3    |  44|     1     |    1     |   o   |   X   |       |       |       |       |
|v_3          |  12|     1     |    1     |   o   |   X   |       |       |       |       |
|v_5          |  14|     1     |    1     |   o   |   X   |       |       |       |       |
|v_6          |  15|     1     |    1     |   o   |   X   |       |       |       |       |
|v_7          |  16|     1     |    1     |   o   |   X   |       |       |       |       |
|written_by   |  33|     1     |    1     |   o   |   X   |       |       |       |       |
|written_by_4 |  37|     1     |    1     |   o   |   X   |       |       |       |       |
|written_by_6 |  39|     1     |    1     |   o   |   X   |       |       |       |       |
|written_by_7 |  40|     1     |    1     |   o   |   X   |       |       |       |       |
+-------------+----+-----------+----------+-------+-------+-------+-------+-------+-------+
[OHazelcastPlugin][orient2] Error on starting distributed plugin
com.orientechnologies.orient.server.distributed.ODistributedException: Quorum (1) cannot be reached because it is major than available nodes (0)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.calculateQuorum(ODistributedDatabaseImpl.java:785)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.send2Nodes(ODistributedDatabaseImpl.java:350)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.sendRequest(ODistributedAbstractPlugin.java:619)
at com.orientechnologies.orient.server.distributed.sql.OCommandExecutorSQLHASyncCluster.replaceCluster(OCommandExecutorSQLHASyncCluster.java:160)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabaseFromNetwork(ODistributedAbstractPlugin.java:1376)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestDatabaseFullSync(ODistributedAbstractPlugin.java:1108)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.requestFullDatabase(ODistributedAbstractPlugin.java:932)
at com.orientechnologies.orient.server.distributed.impl.ODistributedAbstractPlugin.installDatabase(ODistributedAbstractPlugin.java:921)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.installNewDatabasesFromCluster(OHazelcastPlugin.java:1066)
at com.orientechnologies.orient.server.hazelcast.OHazelcastPlugin.startup(OHazelcastPlugin.java:207)
at com.orientechnologies.orient.server.OServer.registerPlugins(OServer.java:1208)
at com.orientechnologies.orient.server.OServer.activate(OServer.java:391)
at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41)
2016-10-09 10:39:19:105 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=1, queue=0 worker=0) [ODistributedWorker]
2016-10-09 10:39:21:106 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=2, queue=0 worker=0) [ODistributedWorker]
2016-10-09 10:39:23:107 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=3, queue=0 worker=0) [ODistributedWorker]
2016-10-09 10:39:25:107 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=4, queue=0 worker=0) [ODistributedWorker]
...
Logs on 1st node:
...
2016-10-09 10:39:17:153 INFO  [orient1] Current node started as MASTER for database 'GratefulDeadConcerts' [OHazelcastPlugin]
2016-10-09 10:39:17:218 INFO  [orient1]->[orient2] Deploying database GratefulDeadConcerts... [OSyncDatabaseTask]
2016-10-09 10:39:17:219 INFO  [orient1] Received updated status orient2.GratefulDeadConcerts=SYNCHRONIZING [OHazelcastPlugin]
2016-10-09 10:39:17:219 INFO  [orient1]->[orient2] Reusing last backup of database 'GratefulDeadConcerts' in directory: /tmp/orientdb/backup_GratefulDeadConcerts.zip... [OSyncDatabaseTask]
2016-10-09 10:39:17:220 INFO  [orient1]->[orient2] - transferring chunk #1 offset=0 size=278904... [OSyncDatabaseTask]
2016-10-09 10:39:17:222 INFO  [orient1]->[orient2] Deploy database task completed [OSyncDatabaseTask]
2016-10-09 10:39:17:763 INFO  [orient1]<-[orient2] Received updated status orient2.GratefulDeadConcerts=ONLINE [OHazelcastPlugin]
2016-10-09 10:39:39:534 WARNI [orient1] Timeout (20000ms) on waiting for synchronous responses from nodes=[orient2] responsesSoFar=[] request=(id=0.4 task=heartbeat timestamp:<PHONE_NUMBER>531) [ODistributedDatabaseImpl]
2016-10-09 10:39:39:535 WARNI [orient1]->[orient2] Server 'orient2' did not respond to the heartbeat message (db=GratefulDeadConcerts, timeout=10000ms), but cannot be set OFFLINE by configuration [OClusterHealthChecker]
...

Hi, the fact you still have |* | | 1 | 1 | X | o | | | | | means your cfg is without the <NEW_NODE> tag. Could you please double check it? You can also shutdown the cluster, open the file under the databases of the first node, add that tag, save and start the 1st node, then all the others.
However, what are you asking makes sense. I'm fixing this issue by asking the database (at startup) to any server configured in the cfg file, not only to the servers in * cluster.

I've just pushed in 2.2.x branch a fix. Could you please try last 2.2.x?

@lvca
hey Luca, many thanks for your swift feedback, awfully appreciated !

stopped v2.2.10 instances on both nodes "orient1" and "orient2" ; removed "databases//distributed" files on 1st node "orient1", and removed entire "databases/" contents on 2nd node "orient2"
installed v2.2.11 , and applied my customisations to "bin/server.sh" ("-Xms1024m -Xmx6000m" and "-Dstorage.diskCache.bufferSize=32000"), to "config/default-distributed-db-config.json" (see below), "config/orientdb-server-config.xml" (only setting the "nodeName") and "config/hazelcast.xml" ('group' name and password, multicast disabled, "tcp-ip" enabled with a list of fixed IP addrs)
started 1st node via "bin/server.sh", no issues
connected with a client, inserting some 100,000's of records into a class, no issues (like before)
stopped 1st node again, moved the big database subdirs out of "databases/" so that they cannot be impacted by the next steps
started 1st node again now via "bin/dserver.sh", no issues
started 2nd node via "bin/dserver.sh" (initial situation is without anything under "databases/", and identical "hazelcast.xml", "default-distributed-db-config.json" , but adjusting only the parameter "nodeName" in "orientdb-server-config.xml"

This failed with the same quorum-related message:
logs of 2nd node:
...
2016-10-12 15:05:19:357 INFO  {db=GratefulDeadConcerts} [orient2] Installed database 'GratefulDeadConcerts' (LSN=OLogSequenceNumber{segment=0, position=24}) [OHazelcastPlugin]
2016-10-12 15:05:19:453 INFO  [orient2] Publishing ONLINE status for database orient2.GratefulDeadConcerts... [ODistributedDatabaseImpl]
2016-10-12 15:05:19:456 INFO  [orient2] Reassigning cluster ownership for database GratefulDeadConcerts [OHazelcastPlugin]
2016-10-12 15:05:19:465 INFO  [orient2] New distributed configuration for database: GratefulDeadConcerts (version=1)
CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+--------+----+-----------+----------+-------+-------+-------------+-------------+-------------+-------------+
|        |    |           |          |MASTER |MASTER |   MASTER    |   MASTER    |   MASTER    |   MASTER    |
|        |    |           |          |ONLINE |ONLINE |NOT_AVAILABLE|NOT_AVAILABLE|NOT_AVAILABLE|NOT_AVAILABLE|
+--------+----+-----------+----------+-------+-------+-------------+-------------+-------------+-------------+
|CLUSTER |  id|writeQuorum|readQuorum|orient1|orient2|   orient4   |   orient5   |   orient3   |   orient6   |
+--------+----+-----------+----------+-------+-------+-------------+-------------+-------------+-------------+
|*       |    |     2     |    1     |   X   |       |             |             |             |             |
|cl_p0 |    |     2     |    1     |   X   |   o   |      o      |             |             |             |
|clp1 |    |     2     |    1     |       |   X   |      o      |      o      |             |             |
|clp2 |    |     2     |    1     |       |       |      X      |      o      |      o      |             |
|clp3 |    |     2     |    1     |       |       |             |      X      |      o      |      o      |
|clp4 |    |     2     |    1     |   o   |       |             |             |      X      |      o      |
|cl_p5 |    |     2     |    1     |   o   |   o   |             |             |             |      X      |
|internal|   0|     2     |    1     |       |       |             |             |             |             |
+--------+----+-----------+----------+-------+-------+-------------+-------------+-------------+-------------+
[OHazelcastPlugin]
2016-10-12 15:05:19:473 INFO  [orient2] Class '_studio', creation of new local cluster '_studio_8' (id=-1) [ODefaultClusterOwnershipAssignmentStrategy]
2016-10-12 15:05:19:533 INFO  [orient2] Class '_studio': change mastership of cluster '_studio_8' (id=73) to node 'orient2' [ODefaultClusterOwnershipAssignmentStrategy]
...
2016-10-12 15:05:19:902 INFO  [orient2] New distributed configuration for database: GratefulDeadConcerts (version=12)
CLUSTER CONFIGURATION (LEGEND: X = Owner, o = Copy)
+-------------+----+-----------+----------+-------+-------+-------------+-------------+-------------+-------------+
|             |    |           |          |MASTER |MASTER |   MASTER    |   MASTER    |   MASTER    |   MASTER    |
|             |    |           |          |ONLINE |ONLINE |NOT_AVAILABLE|NOT_AVAILABLE|NOT_AVAILABLE|NOT_AVAILABLE|
+-------------+----+-----------+----------+-------+-------+-------------+-------------+-------------+-------------+
|CLUSTER      |  id|writeQuorum|readQuorum|orient1|orient2|   orient4   |   orient5   |   orient3   |   orient6   |
+-------------+----+-----------+----------+-------+-------+-------------+-------------+-------------+-------------+
|*            |    |     2     |    1     |   X   |       |             |             |             |             |
|studio_8    |  73|     2     |    1     |   o   |   X   |             |             |             |             |
|clp0      |    |     2     |    1     |   X   |   o   |      o      |             |             |             |
|cl_p1      |    |     2     |    1     |       |   X   |      o      |      o      |             |             |
|v_8          |  83|     2     |    1     |   o   |   X   |             |             |             |             |
|written_by_8 |  74|     2     |    1     |   o   |   X   |             |             |             |             |
+-------------+----+-----------+----------+-------+-------+-------------+-------------+-------------+-------------+
[OHazelcastPlugin]
2016-10-12 15:05:19:903 INFO  [orient2] Saving distributed configuration file for database 'GratefulDeadConcerts' to: /opt/orientdb/orientdb-community-2.2.11/databases/GratefulDeadConcerts/distributed-config.json [OHazelcastPlugin][orient2] Error on starting distributed plugin
com.orientechnologies.orient.server.distributed.ODistributedException: Quorum (1) cannot be reached because it is major than available nodes (0)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.calculateQuorum(ODistributedDatabaseImpl.java:785)
at com.orientechnologies.orient.server.distributed.impl.ODistributedDatabaseImpl.send2Nodes(ODistributedDatabaseImpl.java:350)
...
at com.orientechnologies.orient.server.OServerMain.main(OServerMain.java:41)
2016-10-12 15:05:24:014 INFO  Node is not online yet (status=STARTING), blocking the command until it is online (retry=1, queue=0 worker=0) [ODistributedWorker]
The file "config/default-distributed-db-config.json" contains :
{
"autoDeploy": true,
"readQuorum": 1,
"writeQuorum": 2,
"executionMode": "undefined",
"readYourWrites": true,
"servers": {
"": "master"
},
"clusters": {
"internal": {
},
"cl_p0": { "servers": [ "orient1", "orient2", "orient4" ] },
"clp1": { "servers": [ "orient2", "orient4", "orient5" ] },
"clp2": { "servers": [ "orient4", "orient5", "orient3" ] },
"clp3": { "servers": [ "orient5", "orient3", "orient6" ] },
"clp4": { "servers": [ "orient3", "orient6", "orient1" ] },
"cl_p5": { "servers": [ "orient6", "orient1", "orient2" ] },
"": { "servers": ["orient1" ]}
}
}
Dunnow if it might be relevant, but I did another attempt with "writeQuorum" equals to 1 (iso 2 in previous attempt).

stopped both nodes
removed on 1st node all files "databases/*/distributed-config.json"
removed on 2nd node everyting under "databases/"
modified "config/default-distributed-db-config.json" on both nodes, setting "writeQuorum" to 1
started 1st node with "bin/dserver.sh", all ok
started 2nd node with "bin/dserver.sh", again exactly the same problem:

...
2016-10-12 15:15:48:507 INFO  [orient2] Saving distributed configuration file for database 'GratefulDeadConcerts' to: /opt/orientdb/orientdb-community-2.2.11/databases/GratefulDeadConcerts/distributed-config.json [OHazelcastPlugin][orient2] Error on starting distributed plugin
com.orientechnologies.orient.server.distributed.ODistributedException: Quorum (1) cannot be reached because it is major than available nodes (0)

@lvca
is your fix included in v2.2.11 ? I see on the download page that this version dates from October-3...
If not, can you please explain me what I must do to obtain your fix ?
many thx
Rob

The fix is in the 2.2.x branch. You can compile it by yourself:
git clone https://github.com/orientechnologies/orientdb.git
cd orientdb
git checkout 2.2.x
mvn clean install -DskipTests

At the end you will find your 2.2.x (2.2.12-SNAPSHOT) in the folder 'distribution/target'.

@lvca sorry but I have no direct access from my ODB servers accross our company firewalls with "git"
Can you please transfer me the file(s) via some other medium?

Sure, get last published SNAPSHOT at:  https://oss.sonatype.org/content/repositories/snapshots/com/orientechnologies/orientdb-community/2.2.12-SNAPSHOT/orientdb-community-2.2.12-20161012.150151-25.zip

@lvca
thx Luca !

got your v2.2.12 SNAPSHOT bundle, installed it on both nodes, applied my customisations (see above), and moved my big databases out of the "databases/" directory so they cannot be impacted
started 1st node, all ok

+--------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|Name    |Status|Databases                           |Conns|StartedOn|Binary           |HTTP             |UsedMemory             |
+--------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|orient1*|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|1    |12:38:56 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|104.41MB/5.21GB (1.96%)|
+--------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+

launched 2nd node , seems ok !

2016-10-13 12:50:17:907 WARNI [orient2] Updated node status to 'ONLINE' [OHazelcastPlugin]
2016-10-13 12:50:17:914 INFO  [orient2] Distributed servers status:
+--------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|Name    |Status|Databases                           |Conns|StartedOn|Binary           |HTTP             |UsedMemory             |
+--------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|orient2*|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|2    |12:50:06 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|148.07MB/5.21GB (2.78%)|
|orient1 |ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|1    |12:38:56 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|176.19MB/5.21GB (3.30%)|
+--------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
-> finally have a group of 2 nodes , hurray
I stop 2nd node again :
$ bin/shutdown.sh -h orient2.bc -P 2424-2430 -u root -p $ourPwd
pid file not present or params detected
allowed parameters
-h | --host hostname : name or ip of the host where OrientDB is running. Deafult to localhost
-P | --ports  ports : ports in the form of single value or range. Default to 2424-2430
-p | --password password : the super user password
-u | --user username: the super user name. Default to root
example: shutdown.sh -h orientserver.mydomain -P 2424-2430 -u root -p securePassword
Sending shutdown command to remote OrientDB Server instance...
Shutdown executed correctly
logs on 1st node:
...
2016-10-13 12:54:33:407 WARNI [orient2] Node removed id=Member [<IP_ADDRESS>]:2434 name=orient2 [OHazelcastPlugin]
2016-10-13 12:54:33:410 INFO  [orient1] Reassigning cluster ownership for database GratefulDeadConcerts [OHazelcastPlugin]
...
2016-10-13 12:54:33:418 INFO  [orient1] Distributed servers status:
+--------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|Name    |Status|Databases                           |Conns|StartedOn|Binary           |HTTP             |UsedMemory             |
+--------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
|orient1*|ONLINE|GratefulDeadConcerts=ONLINE (MASTER)|3    |12:38:56 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|219.77MB/5.21GB (4.12%)|
+--------+------+------------------------------------+-----+---------+-----------------+-----------------+-----------------------+
...
2016-10-13 12:54:48:382 INFO  [<IP_ADDRESS>]:2434 [cdrcontrols] [3.6.5] Partition balance is ok, no need to re-partition cluster data...  [InternalPartitionService]
So stopped 1st node as well :
$ bin/shutdown.sh -h orient1.bc -P 2424-2430 -u root -p $ourPwd
-> ok
Placing now our big databases back into the "databases/" directory so that they are opened.
-> restart 1st node : ok
-> restart 2nd node : ok, starts slurping dbase "navi" (7GB) from "orient1", but this fails after a while :
logs on 2nd node "orient2":
...2016-10-13 13:31:40:199 INFO  [orient2] - writing chunk #404 offset=3376340669 size=0b [OHazelcast
Plugin]
2016-10-13 13:31:40:212 INFO  [orient2] Database copied correctly, size=3.14GB [ODistributedAbstra
ctPlugin$3]
2016-10-13 13:31:41:619 WARNI {db=navi} Storage 'navi' was not closed properly. Will try to recove
r from write ahead log [OLocalPaginatedStorage]
2016-10-13 13:31:41:619 SEVER {db=navi} Restore is not possible because write ahead log is empty.
[OLocalPaginatedStorage]
2016-10-13 13:31:41:620 INFO  {db=navi} Storage data recover was completed [OLocalPaginatedStorage
]
2016-10-13 13:31:44:573 INFO  {db=navi} [orient2] Installed database 'navi' (LSN=OLogSequenceNumbe
r{segment=0, position=24}) [OHazelcastPlugin]
2016-10-13 13:31:44:714 INFO  [orient2] Publishing ONLINE status for database orient2.navi... [ODi
stributedDatabaseImpl]
2016-10-13 13:31:44:721 INFO  [orient2] Reassigning cluster ownership for database navi [OHazelcas
tPlugin]
2016-10-13 13:31:44:726 INFO  [orient2] New distributed configuration for database: navi (version=
1)
... (all OK)
(then starts copying GratefulDeadConcerts:)
2016-10-13 13:31:45:495 WARNI [orient2]->[[orient1]] requesting delta database sync for 'GratefulD
eadConcerts' on local server... [OHazelcastPlugin]
2016-10-13 13:31:45:498 INFO  [orient2]->[orient1] Requesting database delta sync for 'GratefulDea
dConcerts' LSN=OLogSequenceNumber{segment=0, position=8376169}... [OHazelcastPlugin]
2016-10-13 13:31:45:533 WARNI [orient2]<-[orient1] Error on installing database delta for 'Gratefu
lDeadConcerts' (Requested delta sync with LSN=OLogSequenceNumber{segment=0, position=8376169} but
found the following error: Cannot execute delta-sync because a new file has been added. Filename:
_studio_10.cpm(id=9068728281913622739)
DB name="GratefulDeadConcerts") [OHazelcastPlugin]
2016-10-13 13:31:45:534 WARNI [orient2]<-[orient1] Requesting full database 'GratefulDeadConcerts'
sync... [OHazelcastPlugin]
2016-10-13 13:31:45:535 SEVER [orient2]->[orient1] Error on asking delta backup of database 'Grate
fulDeadConcerts'... [OHazelcastPlugin]
2016-10-13 13:31:45:535 INFO  [orient2] Loaded database configuration from the active cluster [OHa
zelcastPlugin]
...2016-10-13 13:31:45:546 INFO  [orient2] Distributed servers status:
+--------+--------+-------------------------------------------+-----+---------+-----------------+-
----------------+----------------------+
|Name    |Status  |Databases                                  |Conns|StartedOn|Binary           |H
TTP             |UsedMemory            |
+--------+--------+-------------------------------------------+-----+---------+-----------------+-
----------------+----------------------+
|orient2*|STARTING|navi=ONLINE (MASTER)                       |0    |13:14:40 |<IP_ADDRESS>:2424|1
<IP_ADDRESS>:2480|94.58MB/5.21GB (1.77%)|
|        |        |GratefulDeadConcerts=SYNCHRONIZING (MASTER)|     |         |                 |
|                      |
|orient1 |ONLINE  |navi=ONLINE (MASTER)                       |3    |12:58:36 |<IP_ADDRESS>:2424|1
<IP_ADDRESS>:2480|1.38GB/5.21GB (26.44%)|
|        |        |GratefulDeadConcerts=ONLINE (MASTER)       |     |         |                 |
|                      |
+--------+--------+-------------------------------------------+-----+---------+-----------------+-
----------------+----------------------+
(this dbase finally was synchronized ok, and in ONLINE status)
(then the big dbase "cdrarch", 580GB, starts syncing:)
...
2016-10-13 13:31:47:545 WARNI [orient2]->[[orient1]] Requesting deploy of database 'cdrarch' on lo
cal server... [OHazelcastPlugin]
2016-10-13 13:31:47:857 WARNI [orient2] Moving existent database 'cdrarch' in '/opt/orientdb/orien
tdb-community-2.2.12-SNAPSHOT/databases/cdrarch' to '/opt/orientdb/orientdb-community-2.2.12-SNAPS
HOT/databases//../backup/databases/cdrarch' and get a fresh copy from a remote node... [OHazelcast
Plugin]
2016-10-13 13:31:47:858 INFO  [orient2]<-[orient1] Copying remote database 'cdrarch' to: /tmp/orie
ntdb/install_cdrarch.zip [OHazelcastPlugin]
2016-10-13 13:31:47:860 INFO  [orient2]<-[orient1] Installing database 'cdrarch' to: /opt/orientdb
/orientdb-community-2.2.12-SNAPSHOT/databases/cdrarch... [OHazelcastPlugin]
2016-10-13 13:31:47:860 INFO  [orient2] - writing chunk #1 offset=0 size=0b [OHazelcastPlugin]
2016-10-13 13:31:47:862 INFO  [orient2] Database copied correctly, size=0b [ODistributedAbstractPl
ugin$3]
2016-10-13 13:31:51:597 INFO  Node is not online yet (status=STARTING), blocking the command until
it is online (retry=1, queue=0 worker=0) [ODistributedWorker]
2016-10-13 13:31:53:598 INFO  Node is not online yet (status=STARTING), blocking the command until
it is online (retry=2, queue=0 worker=0) [ODistributedWorker]
=> failure ...

@rdelangh it looks like there is a problem with your database, maybe it's in the backup phase in the source node. Could you please try to backup the database by connecting with the console?

@lvca sorry but I don't entirely understand what you are asking me to do:

keep only the 1st node active ? If I start 2nd node, both nodes hang after a while
from the console -> ok
connecting -> to the database "cdrarch" which failed to synchronize between the two nodes? Connecting to 1st server
"try to backup" -> which command is that, pls ?

@lvca
I did a clean procedure again, starting 1st node, then 2nd node. Same failure.
Attached are the logfiles of each server.
orient-serverlog0-orient1.zip
orient-serverlog0-orient2.zip

@rdelangh it could be the backup procedure that fails before to send the backup. So could you please try t execute a regular full backup of your database? No server, just connect with the console using plocal:<database-path>.

@lvca
hi Luca, thx a lot for your patience to help me out on this issue !
I am waiting for a timeslot to stop the database , will be in one hour or so.
Meanwhile, could you please elaborate a little on the backgrounds of your reasoning regarding backups? I mean, why would the 1st server (where the database currently resides, no other server has a copy yet) suddenly get involved in a backup at the moment that the 2nd node joins the group ?

@lvca
Launched a backup command from the 1st node :
$ bin/backup.sh plocal:/opt/orientdb/databases/navi admin $ourPwd /opt/orientdb/backups/navi.zip
Error: com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/opt/orientdb/databases/navi' with mode=rw
DB name="navi"
Error: com.orientechnologies.orient.core.exception.OStorageException: Can not open storage it is acquired by other process
DB name="navi"
Error: com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/opt/orientdb/databases/navi' with mode=rw
DB name="navi"
Error: com.orientechnologies.orient.core.exception.OStorageException: Can not open storage it is acquired by other process
DB name="navi"
There is of course a server process active, but the ODB documentation regarding backups says that "Backup executes a complete backup against the currently opened database." So no need to stop the server process beforehand.
Also in the "console.sh", I get the same error while trying to open this database:
$ orientdb-community-2.2.11/bin/console.sh
orientdb> connect plocal:/opt/orientdb/databases/navi admin $ourPwd
Connecting to database [plocal:/opt/orientdb/databases/navi] with user 'admin'...
Error: com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage '/opt/orientdb/databases/navi' with mode=rw
DB name="navi"
Error: com.orientechnologies.orient.core.exception.OStorageException: Can not open storage it is acquired by other process
DB name="navi"
-> please advise me

@lvca
hi Luca, any update regarding this issue, please ?

@rdelangh About "Backup executes a complete backup against the currently opened database." it's correct: you weren't able to open the database, so the backup is not possible. In your case, you have to stop one of the server and try to connect to it via plocal.

@lvca
"Stop one of the server" ;-) this whole issue is about not being able to launch a 2nd server...
So the only running server has been stopped, and a backup has been taken, the server has been restarted afterwards. The resulting backup zip file is nearly 20GB size, the original database subdir contains approx 31GB.
Can you please explain what the point was of this action (the backup)? What does this provide you as added information to help troubleshoot the fact that a 2nd node cannot join the 'group' (where 'group' equals 1 ONLINE server) ?

@lvca
Running the same backup command of our large dbase (580GB) aborted with OOM errors:
Error: java.lang.OutOfMemoryError: Java heap space
and produced no backup (zip) file.

@lvca
hi Luca, any suggestions for getting further with this issue, please ?

@rdelangh you've got an OOM exception, so this could be the cause of everything. Do you have the full stack trace for the exception? Your db size is too big to be shared. Could you profile the console while you're running the backup? You could use YourKit for example.

@lvca
I was thinking that you would reason in that direction, based on the fact that launching the "backup.sh" script generated an OOM situation after minutes of apparently scanning through the database files.
BTW, there was no stack trace on the output, only the messages
Error: java.lang.OutOfMemoryError: Java heap space
-> is there a possibility to retry this backup with another method, so that a stack trace would be dumped when the OOM occurs?
I assume that this OOM error comes from the fact that with our large database, we explicitly need to increase the max heap size parameter for the server engine also, otherwise it also dies with an OOM situation before being able to startup.
The "backup.sh" script does not allow to set any such max heap size parameter, but launches the "console.sh" script to perform the backup, which therefor eventually dies with the OOM.
Also, during the attempts to get a 2nd server node added to the group, there was no such long delay between the timestamp in the logs saying that the database would become copied from 1st node to 2nd node:
2016-10-13 13:31:47:858 INFO [orient2]<-[orient1] Copying remote database 'cdrarch' to: /tmp/orie ntdb/install_cdrarch.zip [OHazelcastPlugin] 2016-10-13 13:31:47:860 INFO [orient2]<-[orient1] Installing database 'cdrarch' to: /opt/orientdb /orientdb-community-2.2.12-SNAPSHOT/databases/cdrarch... [OHazelcastPlugin] 2016-10-13 13:31:47:860 INFO [orient2] - writing chunk #1 offset=0 size=0b [OHazelcastPlugin]
The message saying that the "size=0b" following immediately after the message that says the copying would start ("Installing database..").
-> so I think this is not because of an OOM situation, which occurs only after several minutes of growing memory consumption. But of course, you are the expert of OrientDB : can any extra logging be forced by the server, while it is attempting to 'copying remote database' , from where we could find proof of the OOM situation that you assume is happening?

To run the console with more heap, please try: oconsole.sh -Xmx3g. About the logs you should rather see the logs on orient1 from the first message when he's preparing the backup, to the final sending message. Could you write that snippet of logs here?

@lvca
hi Luca, I did provide already these full logs in attach to my previous comment of 7 days before this one (=Oct-13).
Can you access them? If not, I will attach again, because including these logs in a comment box would be way too many lines.

@lvca
I have redone the test, using v2.2.112 snapshot version "orientdb-community-2.2.12-20161019.051435-38"
Attached are the logs of both servers. The 2nd node has started around 2016-10-20 10:20
orient-serverlog0-orient1.zip
orient-serverlog0-orient2.zip

@lvca
Hi Luca, any update on this issue, please ? Did you find any indications in the logs of the cause of this problem?

@lvca
hi Luca, any news about this issue, please ?
Do you have the necessary information from my logs to investigate this problem?
Could the failure to synchronise to a 2nd node be related to the size of my classes/database ?
thx for your help!

I'm experiencing the same issue OCachePointer.finalize: readers != 0 [OCachePointer] on some node randomly after a stop and a restart
2016-11-02 13:16:59:120 INFO  [node1478089006265] Saving distributed configuration file for database 'test' to: /home/orientdb/databases/test/distributed-config.json [OHazelcastPlugin]
2016-11-02 13:16:59:158 WARNI [node1478089006265] Updated node status to 'ONLINE' [OHazelcastPlugin]
2016-11-02 13:16:59:162 INFO  [node1478089006265] Distributed servers status:
+------------------+------+--------------------+-----+---------+-------------------+-------------------+------------------------+
|Name              |Status|Databases           |Conns|StartedOn|Binary             |HTTP               |UsedMemory              |
+------------------+------+--------------------+-----+---------+-------------------+-------------------+------------------------+
|node1478089006265*|ONLINE|test=ONLINE (MASTER)|5    |13:16:44 |<IP_ADDRESS>:2424|<IP_ADDRESS>:2480|282.34MB/1.78GB (15.51%)|
|node1478088912767 |ONLINE|test=ONLINE (MASTER)|5    |13:15:11 |<IP_ADDRESS>:2424 |<IP_ADDRESS>:2480 |76.07MB/1.78GB (4.18%)  |
|node1478088814652 |ONLINE|test=ONLINE (MASTER)|5    |13:13:32 |<IP_ADDRESS>:2424 |<IP_ADDRESS>:2480 |177.13MB/1.78GB (9.73%) |
+------------------+------+--------------------+-----+---------+-------------------+-------------------+------------------------+
[OHazelcastPlugin]
2016-11-02 13:17:02:161 WARNI Authenticated clients can execute any kind of code into the server by using the following allowed languages: [sql] [OServerSideScriptInterpreter]
2016-11-02 13:17:02:162 INFO  OrientDB Studio available at http://<IP_ADDRESS>:2480/studio/index.html [OServer]
2016-11-02 13:17:02:162 INFO  OrientDB Server is active v2.2.12-SNAPSHOT (build 2.2.x@r2a76ee1efc4487d9dc5f43e1ddc2fba1c5a0167c; 2016-10-20 13:55:08+0000). [OServer]
2016-11-02 13:30:38:308 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:315 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:316 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:317 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:318 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
2016-11-02 13:30:38:319 SEVER OCachePointer.finalize: readers != 0 [OCachePointer]
`

Hi @lucafrosini, could you please enable the leak detector as described here http://orientdb.com/docs/2.2.x/Leak-Detector.html ?

@taburet sorry for late answer. Sure, I'm going to do it.

I have 3 nodes. After some attempt of start and stop on different nodes when I stopped node with IP  <IP_ADDRESS> I got on node with IP <IP_ADDRESS> the following warning.
`2016-11-04 14:57:23:168 WARNI [<IP_ADDRESS>]:2434 [d4science_dev] [3.6.5] Removing connection to endpoint Address[<IP_ADDRESS>]:2434 Cause => java.net.SocketException {Connection refused to address /<IP_ADDRESS>:2434}, Error-Count: 242 [TcpIpConnectionMonitor]
Please note that it was written 242 times. I don't know if this is related to the same problem.
My orientdb-server-config.xml has the following property:

Please note while i was stopping the node <IP_ADDRESS> I was starting the third node.
Hope this helps

@lvca
hi Luca, any update, please ?

meanwhile, database cannot be opened anymore after running with 2.2.13* snapshots, see #6884
Waiting for that issue to be resolved first ...

@lvca

database is enabled again, issue #6884 is resolved
Luca, do you have any update regarding this issue with the distributed config, adding a 2nd node ?

@rdelangh I see tons of comments and I'm not sure what's the open problem. Could you please update me on this? Thanks in advance.

@lvca
The last relevant comment from me was from 26 days ago (20/Oct/2016); I repeat the text here:
I have redone the test, using v2.2.12 snapshot version "orientdb-community-2.2.12-20161019.051435-38"
Attached are the logs of both servers. The 2nd node has started around 2016-10-20 10:20
orient-serverlog0-orient1.zip
orient-serverlog0-orient2.zip

So the problem is "node not online yet", right? That should be solved by the last 2.2.13. Please can you try it?

@lvca
hi Luca,

running now with version 2.2.14-20161118.121331-4 on the 1st node, started with "dserver.sh"
started same on 2nd node, which seems now to be retrieving a copy of the 1st database "navi "(34GB size): ```
...
2016-11-18 23:43:32:691 INFO  [orient3] - writing chunk #658 offset=5511143934 size=8.00MB [OHazelcastPlugin]
2016-11-18 23:43:37:504 INFO  [orient3] - writing chunk #659 offset=5519532136 size=8.00MB [OHazelcastPlugin]
...

I notice that this is first retrieved into a ".zip" file /tmp/orientdb/install_navi.zip"
- altough I understand the mechanism, this way of working with a temp ".zip" file in /tmp, will not remain feasible for very big databases :
- our 2nd database, "cdrarch", now occupies about 850GB storage on the 1st server node ...

-> is this step with a zip file necessary ? Can you please provide more detailed documentation about the underlying methods to get a database replicated to an extra node which joins a group?

@lvca

The sync of database "navi" (35GB) has completed