[project-voldemort] <replication-factor xx> for in memory store not working

[yanming Li]

Hi,

 

I'm testing voldemort in memory store with replication factors in a cluster of three, but found out that the data replication never happened. Could someone please let me know what is the problem?

 

I have tried different combination of replication factors and number of nodes, but nothing worked.

 

thanks

Store configuration:

 

<stores>
    <name>XXX</name>
    <persistence>memory</persistence>
    <routing>client</routing>
    <replication-factor>3</replication-factor>
    <required-reads>2</required-reads>
    <required-writes>2</required-writes>
    <key-serializer>
      <type>string</type>
    </key-serializer>
    <value-serializer>
      <type>identity</type>
    </value-serializer>
  </store>
</stores>

Cluster.xml:

 

<cluster>
  <name>EipDataGrid</name>
  <server>
    <id>0</id>
    <host>localhost</host>
    <http-port>8081</http-port>
    <socket-port>6665</socket-port>
    <partitions>0,1,2,3</partitions>
  </server>
  <server>
    <id>1</id>
    <host>localhost</host>
    <http-port>8083</http-port>
    <socket-port>6669</socket-port>
    <partitions>4,5,6,7</partitions>
  </server>
  <server>
    <id>2</id>
    <host>localhost</host>
    <http-port>8087</http-port>
    <socket-port>6673</socket-port>
    <partitions>8,9,10,11</partitions>
  </server>
</cluster>

 

On Fri, Feb 8, 2013 at 9:32 AM, Brendan Harris <vold...@stotch.com> wrote:

Samyuthka,

In addition to what Carlos said, you also need a large enough bdb cache size, so that the cleaners can fit enough of the index into the cache to do their cleaning. If your cache is too small, your cleaner threads can fall behind and cause your jdb structure to grow in an unbounded fashion (rarely reclaiming enough space). Your max logfile size (jdb size) can also impact the performance of the cleaners. And I'd definitely recommend upgrading to a much more recent release.

Also no transaction data is stored in the structure, just key, versions, timestamps and values. It's log structured (append-only), so all modifications to existing keys get written to the end of the current jdb file and the old files are deleted over time as the keys in them are invalidated by the more current jdb files. That deleting of the old jdb files can be improved or degraded depending on your configuration, how many keys you have and how many updates to existing keys you do.

Brendan

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Yanming
(Home)650-520-2673
(cell)  650-278-3837

[Vinoth Chandar]

With client side routing, replication happens from the client itself, so this has nothing to do with what type of persistence is being used on the server..
How did you conclude that it was not replicated? given its in memory?

[yanming Li]

I tried two ways to verify the replication:

1) Using voldemort admin client, looped through nodes and partitions. The resulted keys from all partitions presented no duplicates.

2) Shut down node 1 ( one of the three nodes )and queryed the all keys through voldemort admin client or my own client. All the keys originally belonging to node 1 were lost.

with the above two tests, I am pretty sure there was no replication happening among the three nodes. However, according to your comments, what could be wrong with my client when it was writing?

 

thanks

You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Yanming
(Home)650-520-2673
(cell)  650-278-3837

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Brendan Harris]

Hello Yanming,

Answers inline ...

On Friday, February 8, 2013 10:44:51 PM UTC-8, ajmap wrote:

I tried two ways to verify the replication:

1) Using voldemort admin client, looped through nodes and partitions. The resulted keys from all partitions presented no duplicates.

Which fetchKeys (or fetchEntries) implementation did you use? Certain implementations will only give you back the primary (not secondary or tertiary) keys those partitions own.

 

2) Shut down node 1 ( one of the three nodes )and queryed the all keys through voldemort admin client or my own client. All the keys originally belonging to node 1 were lost.

with the above two tests, I am pretty sure there was no replication happening among the three nodes. However, according to your comments, what could be wrong with my client when it was writing?

If you did the same strategy as above with the admin client, you'll have the same problem depending on the method called. But, with a normal storeClient (or the voldemort-shell) you should be able to do a get() and get the secondary and tertiary keys.

Since you have required-reads and required-writes set to 2. If you do not get an InsufficientOperationalNodesException back from your get/getAll/put call, then you have successfully written or read both the primary and secondary keys at the bare minimum (and have probably written all 3 on a put() if you did not see any other exceptions).

Brendan

[yanming Li]

Hi Brendan,

The replication issue got resolved by specifying prefered-reads greater than required-reads. Now, I got another issue for re-balancing. I have three nodes with 12 partitions. I was trying to add the 4th node and perform re-balancing and got the following error:

[2013-02-12 17:15:52,034 voldemort.client.rebalance.RebalanceCLI] ERROR Unsuccessfully terminated rebalance operation - Rebalance does not support rebalancing of stores of type memory - sdp 

voldemort.VoldemortException: Rebalance does not support rebalancing of stores of type memory - sdp

        at voldemort.utils.RebalanceUtils.validateRebalanceStore(RebalanceUtils.java:1071)

        at voldemort.client.rebalance.RebalanceController.rebalance(RebalanceController.java:114)

        at voldemort.client.rebalance.RebalanceController.rebalance(RebalanceController.java:95)

        at voldemort.client.rebalance.RebalanceController.rebalance(RebalanceController.java:76)

        at voldemort.client.rebalance.RebalanceCLI.main(RebalanceCLI.java:134)

This error indicated that voldemort does not support rebalancing of stores of type memory. Is it ture? If that's the case, is there anyway to rebalancing in-memory stores?

thanks
Yanming

Brendan

--

You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Brendan Harris]

Hi Yanming,

I, myself, do not have any hands-on experience with the InMemoryStorageConfiguration, but indeed it is not supported in the rebalancing code. There is a filter used by the RebalanceController which comes from RebalanceUtils that has a static list of two storage engines in it that it supports: ReadOnlyStorageConfiguration and BdbStorageConfiguration

I am not sure how you would rebalance it because I do not have any experience with the mechanics of that storage engine. The two people I know who do are Roshan Sumbaly and Alex Feinberg. They watch this forum and respond occasionally. I'll see if I can get one of them to comment.

Brendan