distributed architecture : number of replica

159 views
Skip to first unread message

yooo

unread,
Jun 30, 2015, 10:42:47 AM6/30/15
to orient-...@googlegroups.com
Hello,

in a distributed architecture, what is the default number of replicas ? 2 ?

is it possible to configure it, for exemple 3, or is it hard-coded ?

Thanks

scott molinari

unread,
Jul 1, 2015, 2:16:07 AM7/1/15
to orient-...@googlegroups.com
You can certainly have more than 2 nodes. The docs only mention, if you do have more than two, you should always have an odd number, so that you can avoid an issue of split brain with network partitioning.

Also, OrientDB has a multi-master replication system. There are no replicas, but rather replicated clusters. Clusters have a different meaning in OrientDB and you should get to know their meaning, to understand how the multi-master replication works. 

This slide presentation sort of shows the meaning.


And here is more information.


Scott

yooo

unread,
Jul 1, 2015, 3:39:30 AM7/1/15
to orient-...@googlegroups.com
Hello,

thanks for the answer. But I think I didn't express me well

if you look at this slide (which is well written) :
    http://fr.slideshare.net/orientdb/orientdb-distributed-architecture-v20?related=1

in slide 15 it is written that replication + sharding works like RAID.

in slide 16 there is an example of cluster replication with a replica factor is 2 (equals RAID10).

in slide 17  there is another example with replica factor 3.

But I can't find in the documentation where we can set this replica factor. Do you know where to set it ?


Thanks

yooo

unread,
Jul 1, 2015, 3:44:38 AM7/1/15
to orient-...@googlegroups.com
One more precision :

in doc http://orientdb.com/docs/last/Distributed-Configuration.html, about writeQuorum, it is stated :

      On "write" operation (any write on database) is the number of responses to be coherent before to send the response to the client.
      Set to 1 if you don't want this check at write time. Suggested value is N/2+1 where N is the number of replicas. In this way the quorum
      is reached only if the majority of nodes are coherent

But I can't see in the doc how to set the number of replicas (N)


Regards

scott molinari

unread,
Jul 1, 2015, 4:59:48 AM7/1/15
to orient-...@googlegroups.com
From what I understand, the OrientDB distributed system uses Hazelcast and automatic discovery and binding of nodes into the cluster. 


You can also manually add the nodes in the Hazelcast config file. Take a look at the example under "Manual IP".

Scott

scott molinari

unread,
Jul 1, 2015, 5:00:32 AM7/1/15
to orient-...@googlegroups.com
Oh, and the answer about where to set the quorum is also in the link above.

Scott

yooo

unread,
Jul 1, 2015, 5:20:10 AM7/1/15
to orient-...@googlegroups.com

OK thanks for your patience. Unfortuantely, I only undertsand myself ;-) I am not speaking about automatic discovery or setting quorum, but about setting the replication factor.

Thanks again. Apparently my wording is not clear.

Regards

yooo

unread,
Jul 1, 2015, 5:57:31 AM7/1/15
to orient-...@googlegroups.com

I'm trying to explain me better :

Please take the example here :

http://orientdb.com/docs/last/Distributed-Sharding.html

The configuration is as follows :

{
  "autoDeploy": true,
  "hotAlignment": false,
  "readQuorum": 1,
  "writeQuorum": 2,
  "failureAvailableNodesLessQuorum": false,
  "readYourWrites": true,
  "clusters": {
    "internal": {
    },
    "index": {
    },
    "client_usa": {
      "servers" : [ "usa", "europe" ]
    },
    "client_europe": {
      "servers" : [ "europe" ]
    },
    "client_china": {
      "servers" : [ "china", "usa", "europe" ]
    },
    "*": {
      "servers" : [ "<NEW_NODE>" ]
    }
  }
}

My understanding is that :
- the replication factor of cluster
client_usa is 2 ;
-
the replication factor of cluster client_europe is 1 ;
- the replication factor of cluster client_china is 3 ;

I find it heavy to have to set the replication nodes of each cluster.
I would have found it easier to be able to say : I want a replication factor of 2 by default,
which means by default, the cluster is stored on its normal node and replicated on (any)one of
the other available (this is really RAID10)

My understanding about the following part :
"*": { "servers" : [ "<NEW_NODE>" ] }
it means that by default all the cluster will be replicated to all the nodes : for me this is not write-scalable.

Regards

scott molinari

unread,
Jul 1, 2015, 11:12:25 AM7/1/15
to orient-...@googlegroups.com
I understand what you want to know now better, but it is a realm where I myself am not very experienced. Sorry. 

Scott

James Wang

unread,
Jul 1, 2015, 11:33:19 AM7/1/15
to orient-...@googlegroups.com
Interested in this as well.

Is not it the other-way around?  ie
China   -  1
US      -  2
Europe - 3

yooo

unread,
Jul 1, 2015, 1:56:05 PM7/1/15
to orient-...@googlegroups.com
Yes you are right. I've been misleaded by the fact that, from my point of view, there is a bug in the documentation :

The configuration
shows as you write :
China   -  1
US      -  2
Europe - 3

And the picture in "Multiple servers per cluster" part shows  what I wrote :
China   -  3
US      -  2
Europe - 1

Regards

yooo

unread,
Jul 1, 2015, 6:46:44 PM7/1/15
to orient-...@googlegroups.com
Distributed configuration is quite mysterious for me. It's not clear how to update it. I don't know if it's in distributed-config.json file that I have to do the modifications or
in default-distributed-db-config.json. In both cases that's not clear ho to update it.

ANyway, I tried the follwoing with a 4 node setup on default config :

{
    "autoDeploy": true,
    "hotAlignment": false,
    "executionMode": "undefined",
    "readQuorum": 1,
    "writeQuorum": 2,
    "failureAvailableNodesLessQuorum": false,
    "readYourWrites": true,
    "clusters": {
        "internal": {
        },
        "index": {
        },
"t_node0": {"servers" : [ "node0","node1"]},
"t_node1": {"servers" : [ "node1","node2"]},
"t_node2": {"servers" : [ "node2","node3"]},
"t_node3": {"servers" : [ "node3","node0"]},
        "*": {
            "servers" : [ "<NEW_NODE>" ]
        }
    }
}

Then created a distibuted db on the 4 nodes. I issued "create cluster t_node0" on node0 and it was replicated everywhere.
I don't understand.

Regards


James Wang

unread,
Jul 2, 2015, 4:16:02 AM7/2/15
to orient-...@googlegroups.com
Please refer to RAID 10:

Clusters == striping ?  (so China is stripped to 3)
Replica  == mirror   ?  (US has a mirror and Europe has 2 mirrors)

I might be wrong :D

James Wang

unread,
Jul 2, 2015, 4:40:33 AM7/2/15
to orient-...@googlegroups.com
If above is correct, then I guess cluster numbers have to be a multiple of factor numbers.

I.e. if replica factor number is 3, cluster numbers can then be:
3 (at least), 6, 9, 12 ....

Am I correct please?

yooo

unread,
Jul 2, 2015, 9:26:38 AM7/2/15
to orient-...@googlegroups.com
orientdb seems be able to define a fine grained configuration for distributing and replicating the clusters.

For exemple :

China   -  no replica
US      -  one replica
Europe - two replicas

This is smarter than basic RAID. but in my tests the only thing I am able to do is replicate all the clusters on all the nodes,
which is not so good for my purpose (write scalability).


Regards




yooo

unread,
Jul 3, 2015, 7:43:41 AM7/3/15
to orient-...@googlegroups.com
OK I've browsed the source code (the configuration parser only)

The fact is the * is not interpreted. It does not mean "anything". It means default. Then if you write :

"*_node0": {
            "servers" : [ "node0", "node1" ]
        },

It will not be interpreted as "any cluster ending by _node0". Then I made the following test with 4 nodes (node0 to node4) to implement RAID10 for the t class :

{
    "autoDeploy": true,
    "hotAlignment": false,
    "executionMode": "undefined",
    "readQuorum": 1,
    "writeQuorum": 2,
    "failureAvailableNodesLessQuorum": false,
    "readYourWrites": true,
    "clusters": {
        "internal": {
        },
        "index": {
        },
        "*": {
            "servers" : [ "<NEW_NODE>" ]
        },
"t_node0": {
            "servers" : [ "node0", "node1" ]
        },
"t_node1": {
            "servers" : [ "node1", "node2" ]
        },
"t_node2": {
            "servers" : [ "node2", "node3" ]
        },
"t_node3": {
            "servers" : [ "node3", "node0" ]
        }
    }

then create cluster t_node0. When running the "clusters" command, t_node0 is seen on each node, but this is only metadata.
when connecting on node3 and I try to insert :

 orientdb {db=db01}> INSERT INTO t(name, surname) VALUES ('Jay', 'Miner' ) ;

Error: com.orientechnologies.common.exception.OException: Cannot find best cluster for class 't' on server 'node3'. ClusterStrategy=local

wihich is expected behaviour.

Conclusion : it would be nice if orientdb could implement pattern matching for the distributed configuration to avoid too much reload/config complexity.

Regards

Luca Garulli

unread,
Jul 3, 2015, 8:37:32 AM7/3/15
to orient-...@googlegroups.com
Hi,
The command "INSERT INTO t(name, surname) VALUES ('Jay', 'Miner' )" execute on node3 with that configuration, should insert the record into cluster t_node3, because the master server is node3.




Best Regards,

Luca Garulli
CEO at Orient Technologies LTD
the Company behind OrientDB

Conclusion : it would be nice if orientdb could implement pattern matching for the distributed configuration to avoid too much reload.

Regards

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

yooo

unread,
Jul 3, 2015, 9:04:32 AM7/3/15
to orient-...@googlegroups.com
Hi, 

yes, thanks for reply. my test was wrong. i didn't had created t_node3 and with local policy it has generated an error. 

yooo

unread,
Jul 3, 2015, 9:32:07 AM7/3/15
to orient-...@googlegroups.com
Ok I've done more testing with the following configuration which seems the be easier to add node (2 at one time) when dealing with pure RAID10 sharding :

{
    "autoDeploy": true,
    "hotAlignment": false,
    "executionMode": "undefined",
    "readQuorum": 1,
    "writeQuorum": 2,
    "failureAvailableNodesLessQuorum": false,
    "readYourWrites": true,
    "clusters": {
        "internal": {
        },
        "index": {
        },
        "*": {
            "servers" : [ "<NEW_NODE>" ]
        },
"t_node0": {
            "servers" : [ "node0", "node1" ]
        },
"t_node1": {
            "servers" : [ "node1", "node0" ]
        },
"t_node2": {
            "servers" : [ "node2", "node3" ]
        },
"t_node3": {
            "servers" : [ "node3", "node2" ]
        }
    }
}

when adding two more nodes, would be :

{
    "autoDeploy": true,
    "hotAlignment": false,
    "executionMode": "undefined",
    "readQuorum": 1,
    "writeQuorum": 2,
    "failureAvailableNodesLessQuorum": false,
    "readYourWrites": true,
    "clusters": {
        "internal": {
        },
        "index": {
        },
        "*": {
            "servers" : [ "" ]
        },
        "t_node0": {
            "servers" : [ "node0", "node1" ]
        },
        "t_node1": {
            "servers" : [ "node1", "node0" ]
        },
        "t_node2": {
            "servers" : [ "node2", "node3" ]
        },
        "t_node3": {
            "servers" : [ "node3", "node2" ]
        },
        "t_node4": {
            "servers" : [ "node4", "node5" ]
        },
        "t_node5": {
            "servers" : [ "node5", "node4" ]
        }
    }
}



Regards


Reply all
Reply to author
Forward
0 new messages