Orient DB not behaving properly in Distributed Model On

[Ashwani DU-MCA]

Hi Team, I am facing issue while running the Orient DB with DServer.sh ( Distributed Mode in 3 node cluster - Node1, Node2, Node3 ), mainly the issue I am facing is related to Orient DB Studio and other Distributed API commands which gives the updates about Distributed Nodes


Below are the few things to show case, how it is running currently and its behaviour:

Node 1:
root@stg-toolsgraph001:/home/sre# ps -auxwww | grep orientdb
root   3108668 0.8 17.9 8432964 2933688 ?   Ssl Aug21 61:25 /bin/java -server -Xms4G -Xmx4G -Djna.nosys=true -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9 -Ddistributed=true -Djava.util.logging.manager=com.orientechnologies.common.log.ShutdownLogManager -Djava.util.logging.config.file=/opt/orientdb/config/orientdb-server-log.properties -Dorientdb.config.file=/opt/orientdb/config/orientdb-server-config.xml -Dorientdb.www.path=/opt/orientdb/www -Dorientdb.build.number=UNKNOWN@r; 2024-06-18 15:16:52+0000 -cp /opt/orientdb/lib/orientdb-server-3.2.31.jar:/opt/orientdb/lib/*:/opt/orientdb/plugins/* com.orientechnologies.orient.server.OServerMain
root   4136743 0.0 0.0  7008 2260 pts/1  S+  13:57  0:00 grep --color=auto orientdb
root@stg-toolsgraph001:/home/sre# netstat -plnt | grep 3108668
tcp    0   0 node_1_ip:2424    0.0.0.0:*        LISTEN   3108668/java
tcp    0   0 node_1_ip:2480    0.0.0.0:*        LISTEN   3108668/java
tcp    0   0 0.0.0.0:2434      0.0.0.0:*        LISTEN   3108668/java
root@stg-toolsgraph001:/home/sre#


Node 2:
root@stg-toolsgraph002:/home/sre# ps -auxwww | grep orientdb
root   2867953 0.9 17.8 8432968 2929764 ?   Ssl Aug21 70:58 /bin/java -server -Xms4G -Xmx4G -Djna.nosys=true -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9 -Ddistributed=true -Djava.util.logging.manager=com.orientechnologies.common.log.ShutdownLogManager -Djava.util.logging.config.file=/opt/orientdb/config/orientdb-server-log.properties -Dorientdb.config.file=/opt/orientdb/config/orientdb-server-config.xml -Dorientdb.www.path=/opt/orientdb/www -Dorientdb.build.number=UNKNOWN@r; 2024-06-18 15:16:52+0000 -cp /opt/orientdb/lib/orientdb-server-3.2.31.jar:/opt/orientdb/lib/*:/opt/orientdb/plugins/* com.orientechnologies.orient.server.OServerMain
root   3879265 0.0 0.0  7008 2172 pts/1  S+  13:57  0:00 grep --color=auto orientdb
root@stg-toolsgraph002:/home/sre# netstat -plnt | grep 3108668
root@stg-toolsgraph002:/home/sre# netstat -plnt | grep 2867953
tcp    0   0 0.0.0.0:2434      0.0.0.0:*        LISTEN   2867953/java
tcp    0   0 node_2_ip:2424    0.0.0.0:*        LISTEN   2867953/java
tcp    0   0 node_2_ip:2480    0.0.0.0:*        LISTEN   2867953/java
root@stg-toolsgraph002:/home/sre#



Node 3:
root@stg-toolsgraph003:/home/sre# ps -auxwww | grep orientdb
root   2831954 0.7 17.8 8431940 2929652 ?   Ssl Aug21 53:41 /bin/java -server -Xms4G -Xmx4G -Djna.nosys=true -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9 -Ddistributed=true -Djava.util.logging.manager=com.orientechnologies.common.log.ShutdownLogManager -Djava.util.logging.config.file=/opt/orientdb/config/orientdb-server-log.properties -Dorientdb.config.file=/opt/orientdb/config/orientdb-server-config.xml -Dorientdb.www.path=/opt/orientdb/www -Dorientdb.build.number=UNKNOWN@r; 2024-06-18 15:16:52+0000 -cp /opt/orientdb/lib/orientdb-server-3.2.31.jar:/opt/orientdb/lib/*:/opt/orientdb/plugins/* com.orientechnologies.orient.server.OServerMain
root   3840494 0.0 0.0  7008 2252 pts/1  S+  13:57  0:00 grep --color=auto orientdb
root@stg-toolsgraph003:/home/sre# netstat -plnt | grep 2831954
tcp    0   0 node_3_ip:2424    0.0.0.0:*        LISTEN   2831954/java
tcp    0   0 node_3_ip:2480    0.0.0.0:*        LISTEN   2831954/java
tcp    0   0 0.0.0.0:2434      0.0.0.0:*        LISTEN   2831954/java



Logs are saying this: /opt/orientdb/logs/*
2024-08-21 13:59:31:127 INFO [toolsgraph002] Distributed servers status (*=current):
+----------------+------+---------------------------+-----+---------+-----------------+-----------------+-----------------------+
|Name      |Status|Databases         |Conns|StartedOn|Binary      |HTTP       |UsedMemory       |
+----------------+------+---------------------------+-----+---------+-----------------+-----------------+-----------------------+
|toolsgraph001  |ONLINE|ipdb=ONLINE (MASTER)    |0  |13:59:09 |node_1_ip:2424|node_1_ip:2480|123.58MB/4.00GB (3.02%)|
|        |   |20Aug-test=ONLINE (MASTER) |   |     |         |         |            |
|toolsgraph002(*)|ONLINE|ipdb=ONLINE (MASTER)    |1  |13:59:21 |node_2_ip:2424|node_2_ip:2480|120.52MB/4.00GB (2.94%)|
|        |   |20Aug-test=ONLINE (MASTER) |   |     |         |         |            |
|toolsgraph003  |ONLINE|ipdb=NOT_AVAILABLE (MASTER)|2  |13:59:17 |node_3_ip:2424|node_3_ip:2480|118.83MB/4.00GB (2.90%)|
|        |   |20Aug-test=ONLINE (MASTER) |   |     |         |         |            |
+----------------+------+---------------------------+-----+---------+-----------------+-----------------+-----------------------+
 [OHazelcastPlugin]
2024-08-21 13:59:31:191 INFO [toolsgraph002]<-[toolsgraph003] Received new status toolsgraph003.ipdb=ONLINE [OHazelcastClusterMetadataManager]
2024-08-21 13:59:31:196 INFO [toolsgraph002] Distributed servers status (*=current):



Daemon Service Script I am using:
root@stg-toolsgraph001:/opt/orientdb/config# cat /etc/systemd/system/orientdb.service
[Unit]
Description=OrientDB Server
After=network.target

[Service]
User=root
Group=root
ExecStart=/opt/orientdb/bin/dserver.sh
ExecStop=/opt/orientdb/bin/shutdown.sh
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

root@stg-toolsgraph001:/opt/orientdb/config#




Now If I am trying to run any of the Distributed Commands, it is giving me error on the logs:

Commands I am trying to:

curl -u root:root "http://node_2_ip:2480/distributed/node"
curl -u root:root "http://node_2_ip:2480/distributed/database/GratefulDeadConcerts"
curl -u root:root "http://node_2_ip:2480/distributed/queue/*"



Error In the Logs:

[OHazelcastPlugin]
2024-08-21 13:44:41:903 WARNI ->node_2_ip: Command not found: GET.distributed/node [ONetworkProtocolHttpDb]
2024-08-21 13:45:41:704 WARNI ->node_2_ip: Command not found: GET.distributed/database/GratefulDeadConcerts [ONetworkProtocolHttpDb]
2024-08-21 13:46:36:890 WARNI ->node_2_ip: Command not found: GET.distributed/queue/* [ONetworkProtocolHttpDb]

Version of OrientDB:

Console:
orientdb {server=remote:node_1_ip,node_2_ip,node_3_ip}> LIST SERVERS;


Distributed configuration is not active, cannot retrieve server list
orientdb {server=remote:node_1_ip,node_2_ip,node_3_ip}>

PS:
1. Have change the IP of the Hosts to there individual name in the mail
2. orientdb-community-3.2.32.tar.gz : Version of Orient DB I am working with
3. Orient DB Studio on the URL is also not behaving properly, Images are attached in the email:

Please let me know if any further details are needed to help understand and fix this issue. 

Regards

Ashwani