Replication doesn't seem to work in 0.90.1
90 views
Skip to first unread message
Steve
Jan 17, 2012, 5:16:42 AM1/17/12
to project-voldemort
I have a test case with 2 nodes using BDB, replication factor 2,
required writes 2.
I a key/value into a node 1 StoreClient, then pull down node 1.
I read the key from a node 2 StoreClient, and assert that it's there,
but it's not.
The same test worked in 0.80.1.
Anyone got any clues as to why this is?
required writes 2.
I a key/value into a node 1 StoreClient, then pull down node 1.
I read the key from a node 2 StoreClient, and assert that it's there,
but it's not.
The same test worked in 0.80.1.
Anyone got any clues as to why this is?
ctasada
Feb 2, 2012, 4:09:29 AM2/2/12
to project-voldemort
I'm having a similar problem using 0.90. In my case I have 3 nodes
using BDB with replication factor 2, preferred writes 2, reads 1.
When I pull down any of the nodes I get an
InsuficientOperationalNodesException.
Regards,
using BDB with replication factor 2, preferred writes 2, reads 1.
When I pull down any of the nodes I get an
InsuficientOperationalNodesException.
Regards,
Carlos Tasada
Feb 1, 2012, 3:42:41 AM2/1/12
to project-...@googlegroups.com
I'm having a similar problem using 0.90. In my case I have 3 nodes using BDB with replication factor 2, preferred writes 2, reads 1.
When I pull down any of the nodes I get an InsuficientOperationalNodesException.
Regards,
--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.
Alex Feinberg
Feb 7, 2012, 3:50:00 PM2/7/12
to project-...@googlegroups.com
Can you paste your cluster.xml and stores.xml?
If you want read-repair to occur, you should set preferred-reads to 2 as well.
Thanks,
- Alex
ctasada
Feb 8, 2012, 4:57:04 AM2/8/12
to project-voldemort
Hi Alex,
If I'm understanding it properly, the preferred-reads should be set as
the replication factor, is that right?
Here are the files:
cluster.xml
<cluster>
<name>cluster-caronte</name>
<server>
<id>0</id>
<host>vserver1</host>
<http-port>38189</http-port>
<socket-port>6688</socket-port>
<partitions>1,5,9,10,11,13,16,19,24,29,31,32,33,35,39,46,47,60,61,65,74,77,79,80,84,85,86,87,91,92,93,97,103,107,115,119,120,123,126,134,136,138,140,141,143,144,152,157,158,160,164,165,171,172,177,179,183,184,185,188,189,192,193,194,195,201,210,211,214,215,223,224,230,235,237,250,253,256,258,264,267,273,278,287,292,293,294,295,300,301,302,306,308,309,311,314,315,317,318,325,327,328,330,336,337,338,345,346,347,349,350,352,354,356,361,362,365,366,371,373,374,375,377,381,384,389,395,397,400,401,402,406,408,409,412,414,415,417,420,421,422,423,426,429,430,432,443,451,452,456,457,459,463,466,469,470,471,473,481,482,484,486,488,491,496,503,504,507,508,509,512</
partitions>
</server>
<server>
<id>1</id>
<host>vserver2</host>
<http-port>38189</http-port>
<socket-port>6688</socket-port>
<partitions>0,3,6,7,18,21,22,23,26,28,41,42,45,48,49,50,54,56,57,58,59,66,67,68,69,70,75,76,82,83,94,95,98,102,108,109,110,111,113,114,116,121,124,125,128,130,131,133,142,147,148,151,153,154,159,161,169,170,173,175,178,182,186,187,196,197,200,204,205,207,209,217,219,220,225,227,228,229,233,236,240,241,242,243,245,246,247,248,254,255,257,260,263,265,268,270,271,272,274,275,283,285,289,290,291,296,298,312,313,316,322,324,326,331,333,334,335,353,355,358,360,363,364,367,368,369,370,372,376,379,380,388,392,393,398,399,403,404,405,416,419,424,428,431,436,437,439,444,445,446,449,458,461,462,465,467,474,480,483,487,489,492,493,494,495,497,498,500,501,502,510</
partitions>
</server>
<server>
<id>2</id>
<host>vserver3</host>
<http-port>38189</http-port>
<socket-port>6688</socket-port>
<partitions>2,4,8,12,14,15,17,20,25,27,30,34,36,37,38,40,43,44,51,52,53,55,62,63,64,71,72,73,78,81,88,89,90,96,99,100,101,104,105,106,112,117,118,122,127,129,132,135,137,139,145,146,149,150,155,156,162,163,166,167,168,174,176,180,181,190,191,198,199,202,203,206,208,212,213,216,218,221,222,226,231,232,234,238,239,244,249,251,252,259,261,262,266,269,276,277,279,280,281,282,284,286,288,297,299,303,304,305,307,310,319,320,321,323,329,332,339,340,341,342,343,344,348,351,357,359,378,382,383,385,386,387,390,391,394,396,407,410,411,413,418,425,427,433,434,435,438,440,441,442,447,448,450,453,454,455,460,464,468,472,475,476,477,478,479,485,490,499,505,506,511</
partitions>
</server>
</cluster>
stores.xml
<stores>
<store>
<name>protobufAvail</name>
<persistence>bdb</persistence>
<routing>server</routing>
<routing-strategy>consistent-routing</routing-strategy>
<replication-factor>1</replication-factor>
<required-reads>1</required-reads>
<preferred-reads>1</preferred-reads>
<required-writes>1</required-writes>
<preferred-writes>1</preferred-writes>
<key-serializer>
<type>string</type>
</key-serializer>
<value-serializer>
<type>protobuf</type>
<schema-
info>java=com.xxxxxxx.protobuf.ProtobufHotelRoomRateAmountHolder
$ProtobufHotelRoomRateAmountArray</schema-info>
</value-serializer>
</store>
<store>
<name>protobufTax</name>
<persistence>bdb</persistence>
<routing>server</routing>
<routing-strategy>consistent-routing</routing-strategy>
<replication-factor>1</replication-factor>
<required-reads>1</required-reads>
<preferred-reads>1</preferred-reads>
<required-writes>1</required-writes>
<key-serializer>
<type>string</type>
</key-serializer>
<value-serializer>
<type>protobuf</type>
<schema-
info>java=com.xxxxxxx.protobuf.ProtobufHotelRoomRateTaxHolder
$ProtobufHotelRoomRateTaxArray</schema-info>
</value-serializer>
</store>
<store>
<name>protobufInv</name>
<persistence>bdb</persistence>
<routing>server</routing>
<routing-strategy>consistent-routing</routing-strategy>
<replication-factor>1</replication-factor>
<required-reads>1</required-reads>
<preferred-reads>1</preferred-reads>
<required-writes>1</required-writes>
<preferred-writes>1</preferred-writes>
<key-serializer>
<type>string</type>
</key-serializer>
<value-serializer>
<type>protobuf</type>
<schema-info>java=com.xxxxxxx.protobuf.ProtobufInventoryHolder
$ProtobufInventoryArray</schema-info>
</value-serializer>
</store>
</stores>
Thanks,
Carlos
On Feb 7, 9:50 pm, Alex Feinberg <feinb...@gmail.com> wrote:
> Can you paste your cluster.xml and stores.xml?
>
> If you want read-repair to occur, you should set preferred-reads to 2 as well.
>
> Thanks,
> - Alex
>
>
>
>
>
>
>
> On Wed, Feb 1, 2012 at 12:42 AM, Carlos Tasada <ctas...@gmail.com> wrote:
> > I'm having a similar problem using 0.90. In my case I have 3 nodes using BDB
> > with replication factor 2, preferred writes 2, reads 1.
>
> > When I pull down any of the nodes I get an
> > InsuficientOperationalNodesException.
>
> > Regards,
>
If I'm understanding it properly, the preferred-reads should be set as
the replication factor, is that right?
Here are the files:
cluster.xml
<cluster>
<name>cluster-caronte</name>
<server>
<id>0</id>
<host>vserver1</host>
<http-port>38189</http-port>
<socket-port>6688</socket-port>
<partitions>1,5,9,10,11,13,16,19,24,29,31,32,33,35,39,46,47,60,61,65,74,77,79,80,84,85,86,87,91,92,93,97,103,107,115,119,120,123,126,134,136,138,140,141,143,144,152,157,158,160,164,165,171,172,177,179,183,184,185,188,189,192,193,194,195,201,210,211,214,215,223,224,230,235,237,250,253,256,258,264,267,273,278,287,292,293,294,295,300,301,302,306,308,309,311,314,315,317,318,325,327,328,330,336,337,338,345,346,347,349,350,352,354,356,361,362,365,366,371,373,374,375,377,381,384,389,395,397,400,401,402,406,408,409,412,414,415,417,420,421,422,423,426,429,430,432,443,451,452,456,457,459,463,466,469,470,471,473,481,482,484,486,488,491,496,503,504,507,508,509,512</
partitions>
</server>
<server>
<id>1</id>
<host>vserver2</host>
<http-port>38189</http-port>
<socket-port>6688</socket-port>
<partitions>0,3,6,7,18,21,22,23,26,28,41,42,45,48,49,50,54,56,57,58,59,66,67,68,69,70,75,76,82,83,94,95,98,102,108,109,110,111,113,114,116,121,124,125,128,130,131,133,142,147,148,151,153,154,159,161,169,170,173,175,178,182,186,187,196,197,200,204,205,207,209,217,219,220,225,227,228,229,233,236,240,241,242,243,245,246,247,248,254,255,257,260,263,265,268,270,271,272,274,275,283,285,289,290,291,296,298,312,313,316,322,324,326,331,333,334,335,353,355,358,360,363,364,367,368,369,370,372,376,379,380,388,392,393,398,399,403,404,405,416,419,424,428,431,436,437,439,444,445,446,449,458,461,462,465,467,474,480,483,487,489,492,493,494,495,497,498,500,501,502,510</
partitions>
</server>
<server>
<id>2</id>
<host>vserver3</host>
<http-port>38189</http-port>
<socket-port>6688</socket-port>
<partitions>2,4,8,12,14,15,17,20,25,27,30,34,36,37,38,40,43,44,51,52,53,55,62,63,64,71,72,73,78,81,88,89,90,96,99,100,101,104,105,106,112,117,118,122,127,129,132,135,137,139,145,146,149,150,155,156,162,163,166,167,168,174,176,180,181,190,191,198,199,202,203,206,208,212,213,216,218,221,222,226,231,232,234,238,239,244,249,251,252,259,261,262,266,269,276,277,279,280,281,282,284,286,288,297,299,303,304,305,307,310,319,320,321,323,329,332,339,340,341,342,343,344,348,351,357,359,378,382,383,385,386,387,390,391,394,396,407,410,411,413,418,425,427,433,434,435,438,440,441,442,447,448,450,453,454,455,460,464,468,472,475,476,477,478,479,485,490,499,505,506,511</
partitions>
</server>
</cluster>
stores.xml
<stores>
<store>
<name>protobufAvail</name>
<persistence>bdb</persistence>
<routing>server</routing>
<routing-strategy>consistent-routing</routing-strategy>
<replication-factor>1</replication-factor>
<required-reads>1</required-reads>
<preferred-reads>1</preferred-reads>
<required-writes>1</required-writes>
<preferred-writes>1</preferred-writes>
<key-serializer>
<type>string</type>
</key-serializer>
<value-serializer>
<type>protobuf</type>
<schema-
info>java=com.xxxxxxx.protobuf.ProtobufHotelRoomRateAmountHolder
$ProtobufHotelRoomRateAmountArray</schema-info>
</value-serializer>
</store>
<store>
<name>protobufTax</name>
<persistence>bdb</persistence>
<routing>server</routing>
<routing-strategy>consistent-routing</routing-strategy>
<replication-factor>1</replication-factor>
<required-reads>1</required-reads>
<preferred-reads>1</preferred-reads>
<required-writes>1</required-writes>
<key-serializer>
<type>string</type>
</key-serializer>
<value-serializer>
<type>protobuf</type>
<schema-
info>java=com.xxxxxxx.protobuf.ProtobufHotelRoomRateTaxHolder
$ProtobufHotelRoomRateTaxArray</schema-info>
</value-serializer>
</store>
<store>
<name>protobufInv</name>
<persistence>bdb</persistence>
<routing>server</routing>
<routing-strategy>consistent-routing</routing-strategy>
<replication-factor>1</replication-factor>
<required-reads>1</required-reads>
<preferred-reads>1</preferred-reads>
<required-writes>1</required-writes>
<preferred-writes>1</preferred-writes>
<key-serializer>
<type>string</type>
</key-serializer>
<value-serializer>
<type>protobuf</type>
<schema-info>java=com.xxxxxxx.protobuf.ProtobufInventoryHolder
$ProtobufInventoryArray</schema-info>
</value-serializer>
</store>
</stores>
Thanks,
Carlos
On Feb 7, 9:50 pm, Alex Feinberg <feinb...@gmail.com> wrote:
> Can you paste your cluster.xml and stores.xml?
>
> If you want read-repair to occur, you should set preferred-reads to 2 as well.
>
> Thanks,
> - Alex
>
>
>
>
>
>
>
> On Wed, Feb 1, 2012 at 12:42 AM, Carlos Tasada <ctas...@gmail.com> wrote:
> > I'm having a similar problem using 0.90. In my case I have 3 nodes using BDB
> > with replication factor 2, preferred writes 2, reads 1.
>
> > When I pull down any of the nodes I get an
> > InsuficientOperationalNodesException.
>
> > Regards,
>
Chinmay Soman
Feb 22, 2012, 1:59:33 AM2/22/12
to project-voldemort
@ ctasada : Is this the right stores.xml ?
All the stores listed have replication factor 1 not 2 as you mentioned
above.
All the stores listed have replication factor 1 not 2 as you mentioned
above.
Carlos Tasada
Feb 22, 2012, 3:30:15 AM2/22/12
to project-...@googlegroups.com
Sorry, those are the stores of my test enviorment. I'm generating them with and ant script and I copied to wrong ones.
<replication-factor>2</replication-factor>
<preferred-reads>2</preferred-reads>
<required-writes>2</required-writes>
Thanks
Brendan Harris
Feb 22, 2012, 3:54:38 AM2/22/12
to project-voldemort
Carlos,
In your case, you have required writes equal to the replication
factor. So, yes, if you pull down a node, you will fail the operation
(an InsufficientOperationalNodesException will be thrown). That is
expected behavior. Also, if you timeout on a single node you'll have
the same problem. Your required reads/writes should generally be at
least 1 less than your replication factor, otherwise you can never
bounce a node without failing your operations. I suggest bringing
required down to 1 and leaving preferred equal to replication factor.
Brendan
On Feb 22, 12:30 am, Carlos Tasada <ctas...@gmail.com> wrote:
> Sorry, those are the stores of my test enviorment. I'm generating them with
> and ant script and I copied to wrong ones.
>
> <replication-factor>2</replication-factor>
> <preferred-reads>2</preferred-reads>
> <required-writes>2</required-writes>
>
> Thanks
>
In your case, you have required writes equal to the replication
factor. So, yes, if you pull down a node, you will fail the operation
(an InsufficientOperationalNodesException will be thrown). That is
expected behavior. Also, if you timeout on a single node you'll have
the same problem. Your required reads/writes should generally be at
least 1 less than your replication factor, otherwise you can never
bounce a node without failing your operations. I suggest bringing
required down to 1 and leaving preferred equal to replication factor.
Brendan
On Feb 22, 12:30 am, Carlos Tasada <ctas...@gmail.com> wrote:
> Sorry, those are the stores of my test enviorment. I'm generating them with
> and ant script and I copied to wrong ones.
>
> <replication-factor>2</replication-factor>
> <preferred-reads>2</preferred-reads>
> <required-writes>2</required-writes>
>
> Thanks
>
> On Wed, Feb 22, 2012 at 7:59 AM, Chinmay Soman <chinmay.cere...@gmail.com>wrote:
>
>
>
>
>
>
>
> > @ ctasada : Is this the right stores.xml ?
>
> > All the stores listed have replication factor 1 not 2 as you mentioned
> > above.
>
> > On Feb 8, 1:57 am, ctasada <ctas...@gmail.com> wrote:
> > > Hi Alex,
>
> > > If I'm understanding it properly, the preferred-reads should be set as
> > > the replication factor, is that right?
>
> > > Here are the files:
>
> > > cluster.xml
>
> > > <cluster>
> > > <name>cluster-caronte</name>
> > > <server>
> > > <id>0</id>
> > > <host>vserver1</host>
> > > <http-port>38189</http-port>
> > > <socket-port>6688</socket-port>
>
> > <partitions>1,5,9,10,11,13,16,19,24,29,31,32,33,35,39,46,47,60,61,65,74,77, 79,80,84,85,86,87,91,92,93,97,103,107,115,119,120,123,126,134,136,138,140,1 41,143,144,152,157,158,160,164,165,171,172,177,179,183,184,185,188,189,192, 193,194,195,201,210,211,214,215,223,224,230,235,237,250,253,256,258,264,267 ,273,278,287,292,293,294,295,300,301,302,306,308,309,311,314,315,317,318,32 5,327,328,330,336,337,338,345,346,347,349,350,352,354,356,361,362,365,366,3 71,373,374,375,377,381,384,389,395,397,400,401,402,406,408,409,412,414,415, 417,420,421,422,423,426,429,430,432,443,451,452,456,457,459,463,466,469,470 ,471,473,481,482,484,486,488,491,496,503,504,507,508,509,512</
>
>
>
>
>
>
>
> > @ ctasada : Is this the right stores.xml ?
>
> > All the stores listed have replication factor 1 not 2 as you mentioned
> > above.
>
> > On Feb 8, 1:57 am, ctasada <ctas...@gmail.com> wrote:
> > > Hi Alex,
>
> > > If I'm understanding it properly, the preferred-reads should be set as
> > > the replication factor, is that right?
>
> > > Here are the files:
>
> > > cluster.xml
>
> > > <cluster>
> > > <name>cluster-caronte</name>
> > > <server>
> > > <id>0</id>
> > > <host>vserver1</host>
> > > <http-port>38189</http-port>
> > > <socket-port>6688</socket-port>
>
> > > partitions>
> > > </server>
> > > <server>
> > > <id>1</id>
> > > <host>vserver2</host>
> > > <http-port>38189</http-port>
> > > <socket-port>6688</socket-port>
>
> > <partitions>0,3,6,7,18,21,22,23,26,28,41,42,45,48,49,50,54,56,57,58,59,66,6 7,68,69,70,75,76,82,83,94,95,98,102,108,109,110,111,113,114,116,121,124,125 ,128,130,131,133,142,147,148,151,153,154,159,161,169,170,173,175,178,182,18 6,187,196,197,200,204,205,207,209,217,219,220,225,227,228,229,233,236,240,2 41,242,243,245,246,247,248,254,255,257,260,263,265,268,270,271,272,274,275, 283,285,289,290,291,296,298,312,313,316,322,324,326,331,333,334,335,353,355 ,358,360,363,364,367,368,369,370,372,376,379,380,388,392,393,398,399,403,40 4,405,416,419,424,428,431,436,437,439,444,445,446,449,458,461,462,465,467,4 74,480,483,487,489,492,493,494,495,497,498,500,501,502,510</
> > > </server>
> > > <server>
> > > <id>1</id>
> > > <host>vserver2</host>
> > > <http-port>38189</http-port>
> > > <socket-port>6688</socket-port>
>
> > > partitions>
> > > </server>
> > > <server>
> > > <id>2</id>
> > > <host>vserver3</host>
> > > <http-port>38189</http-port>
> > > <socket-port>6688</socket-port>
>
> > <partitions>2,4,8,12,14,15,17,20,25,27,30,34,36,37,38,40,43,44,51,52,53,55, 62,63,64,71,72,73,78,81,88,89,90,96,99,100,101,104,105,106,112,117,118,122, 127,129,132,135,137,139,145,146,149,150,155,156,162,163,166,167,168,174,176 ,180,181,190,191,198,199,202,203,206,208,212,213,216,218,221,222,226,231,23 2,234,238,239,244,249,251,252,259,261,262,266,269,276,277,279,280,281,282,2 84,286,288,297,299,303,304,305,307,310,319,320,321,323,329,332,339,340,341, 342,343,344,348,351,357,359,378,382,383,385,386,387,390,391,394,396,407,410 ,411,413,418,425,427,433,434,435,438,440,441,442,447,448,450,453,454,455,46 0,464,468,472,475,476,477,478,479,485,490,499,505,506,511</
> > > </server>
> > > <server>
> > > <id>2</id>
> > > <host>vserver3</host>
> > > <http-port>38189</http-port>
> > > <socket-port>6688</socket-port>
>
Brendan Harris
Feb 22, 2012, 3:57:32 AM2/22/12
to project-voldemort
Steve,
As Alex asked earlier, can we please see your stores.xml and
cluster.xml? We cannot really help you without first seeing that.
cluster.xml? We cannot really help you without first seeing that.
Carlos Tasada
Feb 22, 2012, 5:22:09 AM2/22/12
to project-...@googlegroups.com
So if I'm getting this right, if one node is down, the replication-factor=2 and the preferred-reads=2 will force the key sync when the node is up again?
Jeff Clites
Feb 29, 2012, 10:43:31 PM2/29/12
to project-...@googlegroups.com
If you are configured to use hinted handoff, then the data will be pushed to the down node from a peer node automatically when it recovers--that's independent of preferred-reads. Also, with preferred-reads equal to the replication factor as you have it, then read repair would sync up the data upon read of the key, independent of hinted handoff. So there are two independent mechanisms working to fix up the data.
ctasada
Mar 1, 2012, 10:53:24 AM3/1/12
to project-...@googlegroups.com
Thanks a lot Jeff.
Steve
Mar 6, 2012, 9:53:18 PM3/6/12
to project-voldemort
Brendan,
We reworked the upgrade and the problem did not repeat. I didn't keep
the patch for the original failed upgrade, so I'm unsure what I did
wrong.
Thanks to you and Alex for getting back to me though.
Steve
We reworked the upgrade and the problem did not repeat. I didn't keep
the patch for the original failed upgrade, so I'm unsure what I did
wrong.
Thanks to you and Alex for getting back to me though.
Steve
Reply all
Reply to author
Forward
0 new messages