Getting started...

[redboy1972]

I am just getting started evaluating AP based key value storage and am having trouble wrapping my head around how to configure my evaluation lab with Voldemort.  The sample configs are for VERY small setups and I'm unsure how to scale them.

I need GR and HA.  I must be able to lose an entire data center without loss of data AND in remaining up to one server per zone for outage/maintenance.

Need to support 5 million keys likely with both key and value under 20 bytes.

Need fast read/write times.

Data center in US-West, US-East to start.  Soon EU and Asia Pacific.

Looking for optimal and not minimal approach and am not scared off if this takes 20+ servers.

QUESTIONS:

How many zones, nodes should I have per data center?

How many partitions and how to spread them so that a write will not encur a huge delay.

Partitions, reads, writes and replication factor are a little confusing.  Not sure what to set them to.

I need a sample setup to get me started in the right direction.  I would hate to eval based on a config error and dismiss this as a potential solution.

I figure 2-3 servers per zone during eval.

My first attempt for a evaluation lab was going to be AWS using 4 servers with 2 in CA, other two in VA.  I may in my final config I may put one in Europe and another in Asia and increase the nodes per zone to 3-5.

<cluster>
  <name>VZWCluster</name>
 
  <!------------------------------------------------------------------------------->
  <!--       These are the datacenters and which are closest to each other.      -->
  <!------------------------------------------------------------------------------->
  <!-- CA  -->
  <zone>
    <zone-id>0</zone-id>
    <proximity-list>1</proximity-list>
  </zone>
  <!-- Virginia -->
  <zone>
    <zone-id>1</zone-id>
    <proximity-list>0</proximity-list>
  </zone>
 
  <!------------------------------------------------------------------------------->
  <!--   These are the servers in the datacenter and which zone they belong to.  -->
  <!------------------------------------------------------------------------------->
 
  <!-- CA  -->
  <server>
    <id>0</id>
    <host>1.1.1.1</host>
    <zone-id>0</zone-id>
    <partitions>1,5,9,13,17,21,25,29,33,37</partitions>
  </server>
  <server>
    <id>1</id>
    <host>2.2.2.2</host>
    <zone-id>0</zone-id>
    <partitions>0,4,8,12,16,20,24,28,32,36</partitions>
  </server>  
 
  <!-- Virginia -->
  <server>
    <id>2</id>
    <host>3.3.3.3</host>
    <zone-id>1</zone-id>
    <partitions>3,7,11,15,19,23,27,31,35,39</partitions>
  </server>
  <server>
    <id>3</id>
    <host>4.4.4.4</host>
    <zone-id>2</zone-id>
    <partitions>2,6,10,14,18,22,26,30,34,38</partitions>
  </server>
</cluster>

<stores>
  <store>
    <name>test</name>
    <persistence>bdb</persistence>
    <routing>client</routing>
    <routing-strategy>zone-routing</routing-strategy>
    <replication-factor>1</replication-factor>
    <required-reads>1</required-reads>
    <required-writes>1</required-writes>
    <zone-replication-factor>
        <replication-factor zone-id="0">1</replication-factor>
        <replication-factor zone-id="1">1</replication-factor>
    </zone-replication-factor>
    <zone-count-reads>1</zone-count-reads>
    <zone-count-writes>1</zone-count-writes>
    <key-serializer>
      <type>string</type>
    </key-serializer>
    <value-serializer>
      <type>string</type>
    </value-serializer>
  </store>
</stores>

[Brendan Harris (a.k.a. stotch on irc.oftc.net)]

Hello,

First and foremost, I'd say that Voldemort is not the right product for your needs. A/P is not what you want, based on your description of your needs. You want a C/P system. But, if you want to try to make it work with Voldemort, I can give you a few pointers.

On Tuesday, April 28, 2015 at 11:00:14 AM UTC-7, redboy1972 wrote:

I am just getting started evaluating AP based key value storage and am having trouble wrapping my head around how to configure my evaluation lab with Voldemort.  The sample configs are for VERY small setups and I'm unsure how to scale them.

I need GR and HA.  I must be able to lose an entire data center without loss of data AND in remaining up to one server per zone for outage/maintenance.

That's stretching the limits of distributed systems in general, but a few systems can do this. Voldemort is not one of them, though. So, you're going to have to settle for either blocking for all writes to succeed in all datacenters/zones up to n-1 replicas and potentially interfering with writes during maintenance or risking consistency yet being able to not interrupt activity during maintenance.

 

Need to support 5 million keys likely with both key and value under 20 bytes.

That's easily done. We have clusters with billions of keys.

 

Need fast read/write times.

You won't get "fast" write times if you need guaranteed consistency, but it may depend on what your definition of "fast" is..

 

Data center in US-West, US-East to start.  Soon EU and Asia Pacific.

 

Looking for optimal and not minimal approach and am not scared off if this takes 20+ servers.

With such a small number of keys, you could probably pull this off with 5 servers per datacenter. But I'd suggest preparing 10 per initially. It also depends on your machine specs, though. We use 48+gB RAM physical hosts with SSD storage. So, we can pack a lot of records into one store.

QUESTIONS:

How many zones, nodes should I have per data center?

You should generally consider a zone to be a datacenter, but it's up to you. I'd do 1:1 datacenter to zone.

 

How many partitions and how to spread them so that a write will not encur a huge delay.

You should have more partitions than you plan to have host (you can have no fewer than 1 partition per host and you probably want to have many partitions per host even a couple years down the road after expanding the number of hosts in the cluster multiple times) and you should use our "rebalance-new-zoned-cluster.sh" script in bin/ in our github repo to shuffle the partitions. The number of partitions depends on how many keys you think you might have. I just always set it to 2000 partitions myself. That works well with billions of keys. Bear in mind, though, that you cannot change the number of partitions once you have data on the cluster.

Partitions, reads, writes and replication factor are a little confusing.  Not sure what to set them to.

Here's where things get complicated. Voldemort uses one hash ring in a zoned cluster. So, if you have 1000 partitions in your cluster, those partitions will be randomly distributed across all hosts in all zones. So, partition 0 might be on node 3 in zone 0, while partition 1 might be on node 0 in zone 1, for example.

When a key is hashed to the ring, whatever partition it initially maps to, that is replica 0. If you have a replication-factor of 3, then there would be a replica 1 and 2. And if you have two zones, you'd multiple the replication factor (set it to 6), then there would be an additional replica 3, 4 and 5. And so, much like the previous example, replica 0 might map to partition 30 on node 2 in zone 1, while replica 1 might map to partition 31 on node 4 in zone 0.

You seem to be familiar already with required reads and writes, so I won't dig into that, but zone-count reads and writes are not what you think. Let's say that your replication factor is 6 and you have 2 zones. That would mean that there are 6 total replicas of keys across the hash ring, 3 of them in one zone and 3 of them in the other. The zone-count reads and writes is simply a blocking count of read/write actions to all replicas across the hash ring. So, if you have required writes 2 for your in-zone consistency, but you want to guarantee that consistency across zones and you have 6 replicas, then you need to set your zone-count writes to 5 to ensure that you block for writing to 5 out of 6 of the replicas (and regardless of where that one inconsistent replica may reside, the required reads 2 will locally resolve the inconsistency at read time).

I need a sample setup to get me started in the right direction.  I would hate to eval based on a config error and dismiss this as a potential solution.

I'd start out with:

zones 2

replication-factor 6

replication-factor per zone 3

required-reads 2

required-writes 1

zone-count-writes 5

Should be zone-id 1, here.

This config gives you no resiliency at all, just one instance of each key. Take a look at my suggested settings. Keep in mind that replication-factor _includes_ the first instances of a key (replica 0).

Based on your explanation of needs, though, you should probably ask yourself what you need more: Availability or consistency? And, based on your answer, start evaluating products that specialize in the feature that is more important to you.

Brendan

[Brendan Harris (a.k.a. stotch on irc.oftc.net)]

Typo fix ...

On Tuesday, April 28, 2015 at 12:27:56 PM UTC-7, Brendan Harris (a.k.a. stotch on irc.oftc.net) wrote:

... So, if you have required writes 2 for your in-zone consistency,

Should have been:

"So, if you have required reads 2 for your in-zone consistency,"

[redboy1972]

First of all, thanks so much for trying to answer completely.

I would like each server who wishes to read/write the cache to not have to contact a remote cache server to do so.  Kinda defeats a cache.   I thought this implies a AP solution of which Voldemort is one.  I do not need strong/guaranteed consistency.  Out of order is fine, multiple values is fine as this is unlikely in my case and likely easily resolved with timestamp.   I have a desire to return reads/writes in <10ms so if I have to write to another data center this isn't possible.  I am trying to get around this using a P-P type architecture as M-S implies latency.  Dynomite was able to do this with 99% <=1ms but had other issues that I'm concerned with.

I would like it to take a write request and return to the client after writing it at least once "zone-count-writes=1" but then have the server continue to replicate it "replication-factor=6" times.  Does Voldemort do that?

http://www.project-voldemort.com/voldemort/configuration.html

I'd start out with:

zones 2                                                  <- Data centers

2000 partitions                                        <- How many buckets each hash will fall in.  Is there a downside to making this huge?  I think I saw replication of route tables but not sure what that means to performance.  Is there an ideal keys:partition ratio?

5-10 nodes/zone                                     <- Will likely depend on read/write speeds as not listed above we have a potential of needing to do 30-50k writes/s and 350k+ reads/s  I figure that a high partition count with fewer per server will do this.  Kinda like sharding.

replication-factor 6                                  <- Write to at least 6 nodes.  Can have up to 5 failures.  One zone=3 + 2 servers from operational zone.  If I set this to 5 would this be better albiet less safe or is there a reason to do more.

replication-factor per zone 3                     <- Write it at least 3 times per zone

required-reads 2                                      <- Block until you read twice and throw an error if that count can't be reached.  Why would I want 2?  Is this to cover for the case of partition divergence?

required-writes 1                                     <- Block until you write at least to one place and throw an error if that count can't be reached.  Will the server continue to replicate it until it writes it 6 times?

zone-count-writes 5                                 <- Block while you attempt to write to 5 zones.  Should this be min=0, minSafe=# of zones

My POC config is bad.  Agreed.

[Arunachalam]

>>  I have a desire to return reads/writes in <10ms so if I have to write to another data center this isn't possible. 

The other data centers are written asynchronously. Your writes never wait for the other data centers to acknowledge, they are handled in the background . Your writes will only wait for the local data center in most cases ( unless you have multiple failures in the local data center).

Voldemort recently had some bugs in the cross data center connection creation, they are fixed in https://github.com/voldemort/voldemort/tree/nioFixes  but not yet merged to the master.  This patch will be merged to the master within a month, after this your writes should return within 10 ms.

Thanks,

Arun.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Brendan Harris (a.k.a. stotch on irc.oftc.net)]

OK, well that changes things a bit.

First, let me call out a few clarifications:

- Voldemort is not sharded and there are no "buckets." There is only the hash ring, which is built against your partition map and each key is hashed to the ring individually. Hence, no bucketing or sharding. The hash is a FNV hash, which is pretty reliable so long as you have enough partitions. Don't go small, but don't go too big either. Like I said, use the aforementioned script to shuffle the partitions across the hosts and use about 2000 partitions in total. More than 2000 tends to be needlessly excessive, fewer than 500 tends to create skew and hot spots that are not easy to redistribute/balance across hosts.

- The voldemort client does all of the replication, even to the hosts in other zones. Calls to all keys hit the first replica in the local zone every time unless the node hosting it is down. So, the traffic balance is also determined by the aforementioned FNV hash.

- zone-count-writes 5 = block while you attempt to write to 5 out of your 6 total replicas (not zones) across all zones

Voldemort is a very different animal from most other distributed databases, so the way it treats hosts, regions/datacenters and keys can definitely be confusing to people who are used to more traditional systems like (Mem|Couch)base and MongoDB.

You should be able to satisfy your latency preference with ease, but bear in mind that this app is in Java and if you do not know JVM tuning well you might run into performance problems later on down the road as your use case evolves. One thing I learned very well after taking over support for voldemort at LinkedIn was JVM internals and JVM tuning. ;-) But, just so you know, we have clusters satisfying 300-500k reads per second and 20-50k writes per second with average read response times of 0.5ms and 95th response times of 1ms and write response times from 0.9ms - 5ms. So, this very well may work for you. But, once again, be prepared to tweak the JVM.

Anyway, now that I have more info from you, you should just remove the zone-count-writes config from my suggested config and that will probably be good enough for you. Alternately you can substitute "preferred-reads 2" in place of "required-reads 2" in order to give you greater availability. With required and preferred both block for n replicas, but a failure to satisfy "preferred-reads" is still a success, whereas a failure to satisfy "required-reads" is a failed operation.

Brendan

[Brendan Harris (a.k.a. stotch on irc.oftc.net)]

One more clarification on my last post:

"The voldemort client does all of the replication, even to the hosts in other zones."

As Arun pointed out, replica writes to the other zones are non-blocking by default and reads will only go to the local zone by default. Configuring zone-count-(reads|writes) changes that behavior and causes additional calls being made to replicas in other zones now be blocking.

Brendan

[redboy1972]

Good to hear.

 

Thanks for the help so far.  I have enough info and have sorted enough out to at least start experimenting.

I have noticed that with my current test that if I take out any single server in 4 server ring, that things continue to work.  This is also true for all servers within a whole zone.  However when I take out a single server from each zone, then I experience failures.  Is this due to a low server count (2 servers in 2 zones = 4 total) which will be remided when I increase servers or will that just make it more frequent and I need to increase some read/write counter?

[root@ip-1.1.1.1 voldemort]# more config/cluster.xml
<cluster>
  <name>MyCluster</name>
  <zone>
    <zone-id>0</zone-id>
    <proximity-list>1</proximity-list>
  </zone>

  <zone>
    <zone-id>1</zone-id>
    <proximity-list>0</proximity-list>
  <

/zone>
  <server>
    <id>0</id>
    <host>1.1.1.1</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>
    <partitions>3, 4, 5, 7, 9, 17, 20, 23, 28, 35, 40, 47, 51, 52, 56, 71, 77, 79, 80, 84, 87,
 88, 91, 92, 95, 99, 101, 108, 109, 112, 119, 120, 123, 132, 137, 145, 147, 148, 155, 157, 159
, 175, 177, 180, 181, 189, 191, 193, 197, 201, 203, 211, 212, 216, 222, 224, 229, 233, 244, 24
7, 252, 255, 256, 257, 265, 268, 271, 273, 281, 283, 288, 292, 311, 316, 321, 327, 329, 339, 3
42, 344, 353, 355, 357, 359, 361, 363, 367, 368, 369, 371, 373, 379, 384, 389, 397, 400, 405,
407, 411, 413, 415, 420, 421, 424, 428, 436, 437, 444, 445, 448, 453, 468, 469, 471, 476, 483,
 485, 492, 497, 499, 504, 509, 511, 515, 525, 527, 531, 532, 533, 552, 553, 557, 568, 571, 575
, 578, 580, 581, 583, 584, 591, 603, 604, 607, 608, 614, 623, 624, 635, 639, 640, 643, 653, 65
6, 663, 665, 668, 673, 675, 685, 687, 692, 693, 697, 700, 704, 705, 707, 708, 712, 721, 723, 7
24, 748, 752, 763, 767, 768, 769, 771, 775, 777, 781, 783, 793, 796, 803, 806, 812, 817, 820,
825, 828, 829, 831, 832, 835, 848, 849, 853, 867, 871, 872, 877, 879, 884, 888, 891, 893, 901,
 919, 921, 931, 936, 947, 955, 956, 960, 961, 964, 968, 973, 975, 981, 983, 985, 987, 995, 100
1, 1003, 1004, 1009, 1012, 1016, 1032, 1035, 1039, 1055, 1057, 1061, 1064, 1068, 1071, 1074, 1
077, 1087, 1088, 1092, 1093, 1096, 1099, 1103, 1104, 1108, 1111, 1112, 1115, 1116, 1120, 1123,
 1127, 1128, 1133, 1135, 1137, 1143, 1144, 1153, 1159, 1160, 1163, 1164, 1167, 1169, 1172, 117
3, 1176, 1184, 1185, 1192, 1193, 1195, 1196, 1197, 1199, 1203, 1208, 1212, 1213, 1217, 1219, 1
220, 1224, 1225, 1228, 1229, 1237, 1239, 1241, 1245, 1248, 1253, 1256, 1257, 1259, 1261, 1264,
 1265, 1267, 1279, 1288, 1289, 1291, 1292, 1297, 1299, 1300, 1303, 1307, 1319, 1320, 1321, 132
3, 1327, 1329, 1332, 1335, 1336, 1337, 1340, 1352, 1359, 1361, 1364, 1367, 1369, 1375, 1380, 1
381, 1383, 1389, 1397, 1404, 1408, 1409, 1419, 1420, 1423, 1424, 1427, 1431, 1440, 1441, 1444,
 1448, 1451, 1452, 1453, 1456, 1457, 1464, 1468, 1472, 1475, 1476, 1480, 1484, 1488, 1492, 149
7, 1501, 1505, 1508, 1513, 1521, 1525, 1527, 1535, 1548, 1557, 1560, 1561, 1563, 1568, 1569, 1
572, 1576, 1580, 1583, 1588, 1589, 1593, 1595, 1597, 1605, 1611, 1612, 1617, 1620, 1625, 1629,
 1632, 1639, 1640, 1644, 1651, 1656, 1657, 1663, 1667, 1669, 1679, 1688, 1697, 1700, 1704, 170
8, 1709, 1715, 1716, 1717, 1720, 1721, 1725, 1729, 1731, 1735, 1748, 1749, 1751, 1752, 1756, 1
759, 1760, 1767, 1768, 1777, 1781, 1783, 1785, 1793, 1797, 1803, 1804, 1807, 1809, 1811, 1816,
 1823, 1824, 1825, 1829, 1835, 1839, 1843, 1845, 1848, 1849, 1852, 1857, 1861, 1863, 1864, 186
5, 1868, 1871, 1873, 1880, 1888, 1895, 1897, 1900, 1901, 1905, 1907, 1909, 1912, 1915, 1916, 1
920, 1926, 1932, 1933, 1939, 1940, 1941, 1944, 1945, 1949, 1957, 1959, 1963, 1965, 1968, 1971,
 1972, 1980, 1984, 1987, 1992, 1996</partitions>
    <zone-id>0</zone-id>

  </server>
  <server>
    <id>1</id>
    <host>2.2.2.2<

/host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>
    <partitions>8, 11, 13, 15, 16, 19, 25, 31, 33, 36, 37, 43, 45, 60, 64, 73, 75, 81, 97, 104
, 107, 111, 115, 116, 117, 121, 124, 125, 128, 129, 133, 141, 149, 152, 163, 165, 169, 171, 17
3, 176, 179, 184, 185, 192, 195, 196, 199, 200, 204, 208, 213, 215, 219, 236, 239, 241, 243, 2
48, 253, 259, 260, 261, 267, 272, 276, 277, 279, 284, 285, 291, 293, 295, 299, 300, 301, 303,
307, 312, 315, 317, 325, 331, 332, 335, 336, 347, 350, 352, 356, 360, 365, 377, 383, 391, 393,
 395, 399, 403, 409, 412, 419, 427, 431, 432, 433, 439, 440, 452, 456, 459, 460, 464, 475, 479
, 484, 487, 491, 493, 495, 501, 503, 507, 512, 513, 517, 521, 523, 528, 529, 537, 540, 544, 54
9, 551, 555, 559, 560, 561, 563, 565, 569, 572, 573, 587, 589, 593, 596, 599, 611, 616, 617, 6
19, 621, 625, 627, 628, 629, 631, 633, 636, 641, 645, 648, 651, 660, 664, 669, 679, 681, 684,
689, 691, 696, 701, 715, 716, 719, 728, 729, 731, 732, 733, 737, 740, 743, 744, 745, 747, 749,
 755, 757, 760, 764, 765, 772, 787, 789, 792, 795, 800, 801, 808, 809, 815, 816, 819, 824, 833
, 836, 837, 839, 841, 843, 845, 847, 851, 852, 855, 859, 863, 865, 873, 880, 881, 885, 887, 88
9, 892, 895, 897, 899, 903, 904, 907, 909, 915, 920, 924, 925, 927, 929, 932, 933, 937, 939, 9
43, 944, 948, 949, 953, 959, 971, 978, 984, 988, 991, 993, 996, 999, 1000, 1007, 1013, 1015, 1
017, 1021, 1023, 1031, 1036, 1040, 1041, 1044, 1047, 1048, 1051, 1052, 1060, 1065, 1067, 1080,
 1081, 1083, 1085, 1089, 1097, 1101, 1105, 1109, 1117, 1129, 1131, 1132, 1140, 1141, 1145, 114
7, 1148, 1151, 1152, 1156, 1161, 1165, 1168, 1171, 1177, 1180, 1181, 1188, 1189, 1204, 1205, 1
207, 1209, 1216, 1223, 1227, 1232, 1233, 1235, 1243, 1244, 1247, 1252, 1260, 1263, 1268, 1269,
 1272, 1276, 1281, 1284, 1285, 1296, 1308, 1309, 1311, 1313, 1317, 1324, 1331, 1333, 1343, 134
4, 1345, 1349, 1353, 1355, 1356, 1357, 1360, 1363, 1365, 1371, 1373, 1376, 1377, 1388, 1391, 1
392, 1395, 1400, 1401, 1405, 1407, 1412, 1415, 1435, 1443, 1445, 1449, 1455, 1459, 1463, 1471,
 1473, 1477, 1479, 1483, 1487, 1496, 1499, 1500, 1504, 1509, 1515, 1516, 1520, 1523, 1528, 152
9, 1531, 1532, 1533, 1538, 1540, 1541, 1545, 1547, 1549, 1552, 1566, 1577, 1585, 1587, 1592, 1
616, 1621, 1627, 1628, 1631, 1633, 1635, 1636, 1641, 1643, 1647, 1649, 1652, 1655, 1659, 1665,
 1668, 1675, 1676, 1677, 1681, 1685, 1691, 1693, 1695, 1696, 1699, 1703, 1705, 1711, 1712, 171
3, 1723, 1724, 1736, 1740, 1743, 1744, 1753, 1763, 1764, 1765, 1769, 1771, 1772, 1775, 1779, 1
787, 1788, 1789, 1791, 1795, 1796, 1799, 1808, 1812, 1813, 1815, 1819, 1821, 1831, 1837, 1853,
 1859, 1860, 1867, 1869, 1872, 1875, 1877, 1879, 1883, 1884, 1891, 1893, 1899, 1904, 1908, 192
3, 1929, 1936, 1943, 1952, 1953, 1960, 1961, 1964, 1967, 1977, 1981, 1983, 1985, 1988, 1991, 1
993, 2000</partitions>
    <zone-id>0</zone-id>
  </server>
  <server>
    <id>2</id>
    <host>3.3.3.3</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>
    <partitions>1, 10, 12, 14, 21, 24, 27, 29, 32, 38, 41, 49, 50, 53, 55, 59, 62, 66, 68, 69,
 70, 72, 74, 82, 83, 85, 86, 89, 93, 96, 98, 102, 110, 113, 126, 127, 131, 134, 135, 136, 138,
 139, 143, 146, 150, 153, 158, 161, 162, 166, 170, 178, 182, 183, 190, 206, 209, 214, 217, 218
, 223, 225, 226, 227, 230, 232, 235, 237, 240, 242, 246, 250, 251, 258, 262, 263, 266, 270, 27
8, 286, 287, 289, 290, 305, 306, 308, 313, 314, 319, 320, 323, 333, 334, 337, 338, 348, 349, 3
51, 354, 362, 366, 370, 374, 375, 376, 378, 380, 385, 392, 394, 396, 398, 401, 402, 406, 410,
416, 417, 418, 423, 429, 430, 435, 438, 442, 447, 449, 454, 461, 467, 474, 478, 480, 489, 490,
 494, 500, 502, 505, 506, 520, 522, 526, 530, 534, 536, 539, 541, 550, 562, 576, 579, 586, 588
, 592, 594, 595, 597, 601, 602, 605, 606, 610, 618, 626, 630, 634, 647, 649, 650, 652, 655, 65
7, 658, 667, 671, 674, 676, 680, 682, 686, 688, 690, 694, 695, 706, 709, 711, 718, 726, 735, 7
41, 746, 756, 758, 759, 761, 773, 778, 780, 784, 788, 794, 798, 804, 810, 811, 818, 821, 822,
827, 834, 838, 840, 842, 860, 864, 866, 868, 869, 878, 883, 886, 890, 896, 902, 905, 906, 908,
 910, 911, 912, 913, 914, 916, 918, 926, 928, 930, 934, 941, 945, 951, 952, 954, 958, 962, 963
, 965, 966, 972, 974, 977, 980, 982, 989, 990, 992, 994, 998, 1002, 1010, 1011, 1014, 1019, 10
20, 1026, 1027, 1028, 1029, 1037, 1038, 1043, 1049, 1054, 1056, 1058, 1062, 1066, 1069, 1072,
1075, 1078, 1082, 1090, 1091, 1094, 1110, 1114, 1119, 1124, 1126, 1130, 1134, 1142, 1146, 1154
, 1155, 1162, 1170, 1174, 1175, 1183, 1186, 1187, 1190, 1200, 1210, 1211, 1214, 1215, 1218, 12
21, 1226, 1230, 1231, 1234, 1236, 1238, 1240, 1242, 1249, 1250, 1262, 1266, 1271, 1273, 1278,
1283, 1286, 1287, 1293, 1295, 1304, 1305, 1315, 1316, 1318, 1322, 1339, 1342, 1346, 1347, 1350
, 1351, 1362, 1366, 1368, 1370, 1379, 1382, 1385, 1386, 1387, 1390, 1393, 1398, 1402, 1406, 14
10, 1411, 1418, 1421, 1422, 1426, 1428, 1429, 1433, 1434, 1436, 1439, 1454, 1458, 1460, 1461,
1462, 1466, 1469, 1470, 1474, 1481, 1482, 1491, 1494, 1495, 1498, 1506, 1507, 1512, 1517, 1518
, 1537, 1544, 1551, 1556, 1559, 1562, 1564, 1567, 1571, 1573, 1575, 1579, 1581, 1582, 1586, 16
00, 1603, 1607, 1610, 1613, 1614, 1615, 1618, 1619, 1623, 1626, 1634, 1642, 1653, 1654, 1658,
1660, 1664, 1670, 1672, 1673, 1682, 1686, 1698, 1701, 1702, 1706, 1710, 1718, 1719, 1722, 1733
, 1734, 1739, 1741, 1746, 1747, 1754, 1755, 1758, 1762, 1766, 1770, 1774, 1784, 1786, 1794, 17
98, 1806, 1817, 1818, 1822, 1827, 1828, 1830, 1832, 1833, 1838, 1840, 1844, 1855, 1858, 1878,
1887, 1890, 1892, 1894, 1898, 1902, 1903, 1911, 1913, 1922, 1925, 1927, 1930, 1934, 1937, 1942
, 1946, 1951, 1956, 1958, 1966, 1970, 1973, 1975, 1979, 1982, 1986, 1994, 1997, 1998, 1999</pa
rtitions>
    <zone-id>1</zone-id>

  </server>
  <server>
    <id>3</id>
    <host>4.4.4.4<

/host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>
    <partitions>0, 2, 6, 18, 22, 26, 30, 34, 39, 42, 44, 46, 48, 54, 57, 58, 61, 63, 65, 67, 7
6, 78, 90, 94, 100, 103, 105, 106, 114, 118, 122, 130, 140, 142, 144, 151, 154, 156, 160, 164,
 167, 168, 172, 174, 186, 187, 188, 194, 198, 202, 205, 207, 210, 220, 221, 228, 231, 234, 238
, 245, 249, 254, 264, 269, 274, 275, 280, 282, 294, 296, 297, 298, 302, 304, 309, 310, 318, 32
2, 324, 326, 328, 330, 340, 341, 343, 345, 346, 358, 364, 372, 381, 382, 386, 387, 388, 390, 4
04, 408, 414, 422, 425, 426, 434, 441, 443, 446, 450, 451, 455, 457, 458, 462, 463, 465, 466,
470, 472, 473, 477, 481, 482, 486, 488, 496, 498, 508, 510, 514, 516, 518, 519, 524, 535, 538,
 542, 543, 545, 546, 547, 548, 554, 556, 558, 564, 566, 567, 570, 574, 577, 582, 585, 590, 598
, 600, 609, 612, 613, 615, 620, 622, 632, 637, 638, 642, 644, 646, 654, 659, 661, 662, 666, 67
0, 672, 677, 678, 683, 698, 699, 702, 703, 710, 713, 714, 717, 720, 722, 725, 727, 730, 734, 7
36, 738, 739, 742, 750, 751, 753, 754, 762, 766, 770, 774, 776, 779, 782, 785, 786, 790, 791,
797, 799, 802, 805, 807, 813, 814, 823, 826, 830, 844, 846, 850, 854, 856, 857, 858, 861, 862,
 870, 874, 875, 876, 882, 894, 898, 900, 917, 922, 923, 935, 938, 940, 942, 946, 950, 957, 967
, 969, 970, 976, 979, 986, 997, 1005, 1006, 1008, 1018, 1022, 1024, 1025, 1030, 1033, 1034, 10
42, 1045, 1046, 1050, 1053, 1059, 1063, 1070, 1073, 1076, 1079, 1084, 1086, 1095, 1098, 1100,
1102, 1106, 1107, 1113, 1118, 1121, 1122, 1125, 1136, 1138, 1139, 1149, 1150, 1157, 1158, 1166
, 1178, 1179, 1182, 1191, 1194, 1198, 1201, 1202, 1206, 1222, 1246, 1251, 1254, 1255, 1258, 12
70, 1274, 1275, 1277, 1280, 1282, 1290, 1294, 1298, 1301, 1302, 1306, 1310, 1312, 1314, 1325,
1326, 1328, 1330, 1334, 1338, 1341, 1348, 1354, 1358, 1372, 1374, 1378, 1384, 1394, 1396, 1399
, 1403, 1413, 1414, 1416, 1417, 1425, 1430, 1432, 1437, 1438, 1442, 1446, 1447, 1450, 1465, 14
67, 1478, 1485, 1486, 1489, 1490, 1493, 1502, 1503, 1510, 1511, 1514, 1519, 1522, 1524, 1526,
1530, 1534, 1536, 1539, 1542, 1543, 1546, 1550, 1553, 1554, 1555, 1558, 1565, 1570, 1574, 1578
, 1584, 1590, 1591, 1594, 1596, 1598, 1599, 1601, 1602, 1604, 1606, 1608, 1609, 1622, 1624, 16
30, 1637, 1638, 1645, 1646, 1648, 1650, 1661, 1662, 1666, 1671, 1674, 1678, 1680, 1683, 1684,
1687, 1689, 1690, 1692, 1694, 1707, 1714, 1726, 1727, 1728, 1730, 1732, 1737, 1738, 1742, 1745
, 1750, 1757, 1761, 1773, 1776, 1778, 1780, 1782, 1790, 1792, 1800, 1801, 1802, 1805, 1810, 18
14, 1820, 1826, 1834, 1836, 1841, 1842, 1846, 1847, 1850, 1851, 1854, 1856, 1862, 1866, 1870,
1874, 1876, 1881, 1882, 1885, 1886, 1889, 1896, 1906, 1910, 1914, 1917, 1918, 1919, 1921, 1924
, 1928, 1931, 1935, 1938, 1947, 1948, 1950, 1954, 1955, 1962, 1969, 1974, 1976, 1978, 1989, 19
90, 1995</partitions>
    <zone-id>1</zone-id>
  </server>
</cluster>

[redboy1972]

Here is my stores.xml

 

cat config/stores.xml

<stores>

  <store>

    <name>test</name>

    <persistence>bdb</persistence>

    <routing>client</routing>

    <routing-strategy>zone-routing</routing-strategy>

    <replication-factor>2</replication-factor>

    <required-reads>1</required-reads>

    <required-writes>1</required-writes>

    <zone-replication-factor>

        <replication-factor zone-id="0">1</replication-factor>

        <replication-factor zone-id="1">1</replication-factor>

    </zone-replication-factor>

    <zone-count-reads>0</zone-count-reads>

    <zone-count-writes>0</zone-count-writes>

[Brendan Harris (a.k.a. stotch on irc.oftc.net)]

That's because there are only two copies of each record, one copy in each zone. So, if you take out one server from each zone, you lose one full set of records. You want at least two copies in each zone. That would mean that you need a replication-factor of 4 and 2 in each zone. I recommend doing 6 and 3/3, really.

[redboy1972]

LOL - I'm pulling my hair out!

Here is my latest attempt:

2 zones (data centers)

3 servers per zone (6 total)

partitions provided by running "bin/rebalance-new-zoned-cluster.sh -c config/cluster.xml  -s config/stores.xml -o new"

Even servers are in zone 0.  Odd in zone 1.

I can survive a complete outage of a data center (servers 0,2,4 down - OR - 1.3.5 down).  I can survive any single server down.

I CAN'T survive a complete outage of a data center + 1 on remote.  This is worst case scenario we plan to handle.  See "1 DC + 1 Node out" below in image.

I thought that with the following that data would be written 4 times, twice in zone0 and twice in zone1.  With zone0 and one server in zone1 down there should still be one server in zone1 with the data.  However the client errors out.  

ERROR:> voldemort.store.InsufficientOperationalNodesException: Only 0 nodes up in preference list, but 1 required. Replication set: [4, 5, 0, 2]Nodes down: [0, 5, 2, 4, 5, 0, 2]

<replication-factor>4</replication-factor>

    <zone-replication-factor>
        <replication-factor zone-id="0">2</replication-factor>
        <replication-factor zone-id="1">2</replication-factor>
    </zone-replication-factor>

I want to handle:

[root voldemort]# cat config/stores.xml
<stores>
  <!-- http://www.project-voldemort.com/voldemort/configuration.html -->
  <store>
 <name>test</name>
 <!-- persistence— The persistence backend used by the store. Currently this could be one of    -->
 <!-- bdb, mysql, memory, readonly, and cache. The difference between cache and memory is that  -->
 <!-- memory will throw and OutOfMemory exception if it grows larger than the JVM heap whereas  -->
 <!-- cache will discard data.                                                                  -->
 <persistence>bdb</persistence>


 <!-- routing— Determines the routing policy. We support both client ( Client side routing )    -->
 <!-- and server ( Server side routing ).                                                       -->
 <routing>client</routing>


 <!-- routing-strategy— Determines how we store the replicas. Currently we support three        -->
 <!-- routing strategies -->
 <!--     consistent-routing (default) A Routing strategy that routes each request to the first -->
 <!--                                  N nodes where N is a user defined replication factor.    -->
 <!--     zone-routing A Zone Routing strategy that sits on top of the Consistent Routing       -->
 <!--                  strategy such that each request goes to zone specific replicas           -->
 <!--     all-routing. A routing strategy which just routes each request to all the nodes given.-->
 <routing-strategy>zone-routing</routing-strategy>


 <!-- preferred-reads (optional)—The number of successful reads the client will attempt to do   -->
 <!-- before returning a value to the application. This defaults to be equal to required reads  -->
 <preferred-reads>1</preferred-reads>
 <!-- required-reads—The least number of reads that can succeed without throwing an exception.  -->
 <required-reads>1</required-reads>


 <!-- preferred-writes(optional)—The number of successful writes the client attempts to block   -->
 <!-- for before returning success. Defaults to required-writes                                 -->
 <preferred-writes>1</preferred-writes>
 <!-- required-writes— The least number of writes that can succeed without the client getting   -->
 <!-- back an exception.                                                                        -->
 <required-writes>1</required-writes>


 <!-- replication-factor— This is the total number of times the data is stored.  Each put or    -->
 <!-- delete operation must eventually hit this many nodes. A replication factor of n means     -->
 <!-- it can be possible to tolerate up to n - 1 node failures without data loss.               -->
 <replication-factor>4</replication-factor>


 <!-- If you intend to use the zone-routing strategy we need to extend the store definition to  -->
 <!-- tell it how to replicate w.r.t. zones.  The important change here is the introduction of  -->
 <!-- zone-replication-factor which should contain a replication factor that you would want in  -->
 <!-- every zone.                                                                               -->
    <zone-replication-factor>
        <replication-factor zone-id="0">2</replication-factor>
        <replication-factor zone-id="1">2</replication-factor>
    </zone-replication-factor>


 <!-- zone-count-*— The number of zones we want to block for during reads / writes before we   -->
 <!-- return the request. The number 0 means we'll block for at least one request from the     -->
 <!-- local zone only. The number 1 means we'll block for at least one request from one other  -->
 <!-- zone.  A missing value of <zone-count-reads> / <zone-count-writes> defaults to 0, which  -->
 <!-- means we don’t care about the zones and fall back to zone unaware strategy.              -->
    <zone-count-reads>0</zone-count-reads>
    <zone-count-writes>0</zone-count-writes>


 <!-- hinted-handoff-strategy (optional) —  This       -->
 <!-- parameter defines the strategy we would use to decide which live nodes to write our      -->
 <!-- "hint" to. The various options are any-handoff, consistent-handoff and proximity-handoff.-->
 <hinted-handoff-strategy>any-handoff</hinted-handoff-strategy>




 <!-- retention-days (optional)— This optional parameter allows you to set a retention property -->
 <!-- to your data. Then every day, at a specified time on the servers, a scheduled job will be -->
 <!-- run to delete all data having timestamp > retention-days. This is useful to keep your     -->
 <!-- data trimmed.                                                                             -->
 <retention-days>90</retention-days>
 <!-- retention-scan-throttle-rate (optional)— If retention-days is specified this is the rate  -->
 <!-- at which we'll scan the tuples to delete data.                                            -->
 <retention-scan-throttle-rate>30</retention-scan-throttle-rate>


 <!-- key/value-serializer— The serialization type used for reading and writing keys/values.   -->
 <!-- Thetype can be json, java-serialization, string, protobuff, thrift, or identity (meaning -->
 <!-- raw bytes). Theschema-info gives information to the serializer about how to perform the  -->
    <!-- mapping (e.g. theJSON schema described in here).                                         -->

    <key-serializer>
      <type>string</type>
    </key-serializer>
    <value-serializer>
      <type>string</type>
    </value-serializer>
  </store>
</stores>

<cluster>
  <name>VZWCluster</name>

  <zone>
    <zone-id>0</zone-id>
    <proximity-list>1</proximity-list>
  </zone>
  <zone>
    <zone-id>1</zone-id>
    <proximity-list>0</proximity-list>
  </zone>
  <server>
    <id>0</id>
    <host>1.1.1.1</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>3, 7, 17, 20, 23, 35, 68, 71, 77, 80, 84, 87, 92, 95, 101, 109, 117, 119, 120, 132, 145, 147, 155, 159, 177, 189, 193, 197, 203, 208, 227, 229, 233, 244, 247, 252, 255, 265, 268, 271, 285, 288, 292, 299, 304, 311, 316, 321, 339, 342, 353, 359, 368, 369, 371, 373, 379, 386, 389, 400, 405, 407, 411, 420, 425, 437, 449, 453, 468, 482, 485, 492, 499, 515, 527, 531, 532, 533, 547, 549, 552, 563, 568, 571, 581, 583, 584, 589, 607, 608, 614, 623, 631, 635, 639, 640, 643, 653, 663, 665, 668, 673, 687, 692, 693, 704, 705, 708, 721, 723, 724, 731, 737, 745, 752, 753, 757, 767, 769, 777, 781, 783, 789, 793, 803, 806, 817, 825, 828, 829, 832, 835, 848, 853, 867, 871, 884, 885, 888, 891, 901, 919, 931, 936, 955, 956, 960, 968, 973, 975, 981, 983, 996, 1001, 1003, 1009, 1016, 1032, 1035, 1039, 1044, 1051, 1055, 1057, 1061, 1064, 1068, 1074, 1077, 1087, 1088, 1093, 1099, 1103, 1108, 1111, 1112, 1115, 1120, 1123, 1133, 1144, 1153, 1159, 1163, 1164, 1167, 1169, 1172, 1176, 1184, 1185, 1196, 1197, 1203, 1208, 1212, 1213, 1217, 1220, 1229, 1241, 1243, 1245, 1248, 1253, 1256, 1265, 1267, 1279, 1282, 1291, 1300, 1319, 1320, 1321, 1323, 1327, 1332, 1336, 1352, 1353, 1359, 1364, 1367, 1369, 1380, 1381, 1383, 1389, 1397, 1406, 1408, 1412, 1419, 1420, 1424, 1431, 1441, 1444, 1448, 1453, 1456, 1457, 1468, 1473, 1475, 1480, 1483, 1490, 1492, 1499, 1501, 1505, 1508, 1513, 1521, 1525, 1527, 1532, 1535, 1548, 1561, 1563, 1572, 1576, 1580, 1583, 1586, 1588, 1601, 1605, 1611, 1617, 1625, 1629, 1639, 1656, 1661, 1662, 1663, 1681, 1688, 1700, 1704, 1715, 1716, 1717, 1720, 1721, 1732, 1735, 1748, 1749, 1751, 1752, 1756, 1759, 1777, 1781, 1793, 1804, 1809, 1816, 1823, 1824, 1829, 1835, 1839, 1843, 1848, 1849, 1868, 1871, 1873, 1880, 1900, 1905, 1916, 1920, 1926, 1933, 1941, 1944, 1945, 1963, 1965, 1971, 1972, 1980, 1984, 1987, 1992, 1996</partitions>

    <zone-id>0</zone-id>
  </server>
  <server>
    <id>1</id>
    <host>2.2.2.2</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>1, 2, 12, 18, 21, 24, 29, 38, 45, 53, 55, 56, 66, 69, 70, 72, 74, 82, 89, 91, 98, 99, 110, 118, 123, 126, 130, 134, 138, 139, 146, 148, 153, 158, 161, 162, 178, 182, 190, 194, 218, 224, 230, 232, 235, 237, 240, 242, 246, 251, 257, 258, 263, 283, 286, 287, 289, 294, 305, 306, 313, 319, 333, 334, 337, 338, 343, 348, 351, 361, 362, 363, 375, 384, 385, 390, 392, 396, 398, 401, 416, 417, 426, 428, 435, 436, 442, 444, 447, 448, 454, 462, 469, 490, 512, 526, 536, 539, 550, 555, 576, 579, 586, 591, 592, 594, 595, 597, 601, 602, 606, 610, 618, 634, 647, 649, 650, 652, 671, 699, 700, 709, 711, 714, 718, 726, 732, 735, 758, 759, 761, 764, 773, 778, 780, 788, 796, 798, 804, 811, 815, 831, 838, 849, 860, 868, 869, 872, 877, 879, 882, 883, 890, 902, 905, 906, 908, 910, 914, 922, 934, 947, 951, 958, 962, 965, 966, 980, 990, 995, 1000, 1002, 1010, 1017, 1020, 1024, 1026, 1037, 1047, 1054, 1062, 1069, 1078, 1082, 1086, 1090, 1091, 1094, 1119, 1124, 1126, 1128, 1130, 1135, 1155, 1160, 1170, 1174, 1175, 1183, 1187, 1190, 1193, 1206, 1214, 1219, 1224, 1226, 1240, 1259, 1266, 1269, 1271, 1273, 1278, 1281, 1283, 1288, 1293, 1295, 1297, 1299, 1303, 1304, 1305, 1308, 1314, 1315, 1330, 1335, 1339, 1342, 1346, 1362, 1373, 1375, 1382, 1384, 1386, 1390, 1393, 1398, 1402, 1404, 1417, 1427, 1429, 1434, 1436, 1439, 1440, 1442, 1451, 1454, 1458, 1460, 1469, 1472, 1474, 1476, 1481, 1488, 1494, 1497, 1502, 1506, 1507, 1543, 1556, 1560, 1568, 1573, 1579, 1582, 1589, 1603, 1607, 1608, 1613, 1615, 1618, 1619, 1626, 1632, 1634, 1644, 1657, 1667, 1670, 1673, 1701, 1706, 1708, 1737, 1745, 1746, 1747, 1755, 1768, 1783, 1785, 1786, 1789, 1798, 1806, 1818, 1822, 1832, 1840, 1845, 1863, 1882, 1892, 1894, 1896, 1903, 1910, 1911, 1922, 1925, 1931, 1932, 1942, 1946, 1951, 1954, 1962, 1968, 1970, 1979, 1982, 1986, 1994, 1999</partitions>
    <zone-id>1</zone-id>
  </server>

  <server>
    <id>2</id>
    <host>3.3.3.3</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>10, 14, 27, 32, 34, 41, 46, 51, 58, 59, 62, 63, 83, 85, 93, 96, 102, 106, 113, 131, 135, 136, 143, 150, 151, 156, 157, 166, 168, 170, 175, 183, 201, 206, 209, 214, 217, 221, 223, 226, 243, 260, 262, 266, 270, 278, 290, 308, 312, 320, 349, 354, 355, 366, 367, 370, 374, 376, 378, 380, 388, 394, 395, 402, 406, 408, 410, 413, 421, 424, 430, 438, 459, 461, 467, 474, 478, 480, 489, 494, 497, 500, 502, 506, 509, 510, 520, 522, 530, 534, 559, 562, 569, 570, 588, 590, 603, 605, 616, 626, 627, 630, 655, 657, 658, 664, 667, 674, 680, 682, 686, 688, 690, 694, 695, 706, 710, 728, 734, 736, 741, 742, 744, 756, 762, 766, 776, 784, 795, 810, 818, 820, 821, 827, 834, 840, 842, 846, 857, 864, 870, 873, 886, 893, 896, 911, 913, 916, 918, 927, 930, 937, 941, 945, 952, 954, 957, 963, 970, 974, 977, 982, 989, 994, 1008, 1011, 1014, 1028, 1029, 1038, 1041, 1049, 1058, 1063, 1070, 1072, 1075, 1081, 1084, 1096, 1101, 1104, 1110, 1114, 1134, 1136, 1138, 1142, 1143, 1146, 1147, 1151, 1154, 1162, 1171, 1186, 1200, 1210, 1211, 1215, 1218, 1221, 1230, 1231, 1236, 1239, 1242, 1244, 1249, 1250, 1255, 1262, 1263, 1286, 1287, 1292, 1301, 1311, 1316, 1318, 1324, 1328, 1347, 1350, 1351, 1365, 1366, 1368, 1370, 1379, 1385, 1410, 1411, 1418, 1421, 1422, 1426, 1433, 1445, 1450, 1461, 1462, 1466, 1467, 1482, 1491, 1498, 1504, 1512, 1517, 1519, 1537, 1544, 1549, 1551, 1559, 1562, 1564, 1567, 1571, 1575, 1581, 1585, 1587, 1590, 1592, 1594, 1600, 1604, 1610, 1612, 1623, 1631, 1642, 1645, 1646, 1652, 1653, 1654, 1658, 1660, 1664, 1682, 1683, 1686, 1694, 1698, 1714, 1718, 1719, 1722, 1725, 1733, 1734, 1739, 1754, 1758, 1760, 1762, 1766, 1770, 1773, 1774, 1779, 1794, 1803, 1814, 1817, 1821, 1827, 1828, 1830, 1833, 1838, 1842, 1844, 1847, 1855, 1858, 1878, 1885, 1890, 1898, 1902, 1913, 1923, 1930, 1934, 1937, 1956, 1959, 1966, 1976</partitions>
    <zone-id>1</zone-id>
  </server>

  <server>
    <id>3</id>
    <host>4.4.4.4</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>6, 22, 30, 39, 48, 49, 61, 65, 67, 78, 90, 94, 100, 105, 107, 114, 116, 121, 122, 127, 140, 142, 144, 154, 160, 164, 174, 187, 198, 200, 202, 207, 210, 215, 220, 231, 238, 245, 249, 250, 254, 264, 269, 274, 275, 280, 284, 297, 298, 302, 303, 310, 315, 318, 324, 328, 330, 340, 341, 345, 357, 358, 381, 404, 418, 422, 434, 439, 441, 450, 451, 455, 457, 466, 470, 483, 484, 486, 496, 505, 508, 516, 517, 518, 538, 540, 542, 543, 545, 548, 551, 553, 556, 566, 572, 574, 582, 585, 598, 609, 612, 613, 622, 637, 638, 641, 644, 646, 662, 666, 670, 676, 677, 678, 702, 720, 722, 725, 729, 739, 746, 750, 754, 755, 768, 770, 774, 779, 782, 785, 790, 791, 794, 799, 802, 807, 812, 823, 830, 844, 850, 854, 858, 862, 865, 866, 876, 881, 895, 898, 917, 926, 928, 939, 940, 943, 946, 950, 959, 969, 971, 976, 986, 987, 992, 997, 1006, 1012, 1018, 1022, 1025, 1030, 1031, 1033, 1040, 1043, 1045, 1046, 1052, 1059, 1060, 1066, 1079, 1098, 1106, 1107, 1117, 1121, 1122, 1129, 1139, 1150, 1157, 1158, 1166, 1168, 1178, 1179, 1182, 1189, 1194, 1198, 1201, 1202, 1205, 1222, 1234, 1237, 1246, 1254, 1261, 1270, 1275, 1276, 1290, 1310, 1312, 1322, 1326, 1334, 1337, 1355, 1358, 1372, 1378, 1388, 1391, 1394, 1396, 1399, 1405, 1407, 1413, 1416, 1430, 1446, 1465, 1470, 1478, 1479, 1485, 1486, 1489, 1493, 1495, 1503, 1509, 1510, 1514, 1516, 1522, 1523, 1524, 1526, 1530, 1534, 1539, 1546, 1552, 1554, 1557, 1565, 1570, 1578, 1584, 1595, 1598, 1606, 1624, 1628, 1630, 1637, 1640, 1641, 1648, 1650, 1671, 1674, 1678, 1680, 1684, 1689, 1690, 1691, 1702, 1705, 1710, 1723, 1727, 1728, 1730, 1731, 1741, 1750, 1757, 1771, 1776, 1780, 1782, 1790, 1796, 1800, 1810, 1820, 1825, 1836, 1851, 1854, 1857, 1860, 1862, 1866, 1870, 1874, 1876, 1886, 1888, 1907, 1914, 1917, 1921, 1927, 1928, 1938, 1949, 1955, 1958, 1973, 1978, 1990, 1995, 1998, 2000</partitions>
    <zone-id>1</zone-id>
  </server>
  <server>
    <id>4</id>
    <host>5.5.5.5</host>

    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>8, 15, 16, 19, 25, 31, 33, 36, 37, 43, 60, 64, 73, 97, 104, 108, 112, 115, 124, 125, 128, 129, 133, 137, 141, 149, 152, 165, 169, 171, 176, 179, 181, 185, 195, 196, 199, 204, 216, 219, 225, 236, 241, 248, 267, 272, 276, 277, 279, 281, 291, 295, 300, 301, 317, 325, 327, 329, 331, 332, 335, 336, 347, 350, 360, 365, 387, 391, 393, 409, 412, 419, 427, 431, 432, 433, 440, 456, 460, 464, 475, 476, 479, 487, 493, 495, 501, 503, 507, 513, 521, 523, 528, 529, 537, 541, 544, 557, 561, 565, 587, 593, 596, 599, 604, 611, 617, 619, 625, 628, 633, 645, 648, 651, 660, 669, 681, 684, 689, 696, 701, 715, 716, 719, 740, 749, 765, 772, 787, 792, 801, 816, 819, 822, 824, 833, 836, 839, 841, 843, 845, 847, 851, 855, 859, 880, 887, 889, 892, 894, 897, 899, 903, 904, 909, 915, 920, 921, 924, 925, 929, 932, 933, 944, 948, 949, 953, 984, 985, 988, 991, 999, 1004, 1005, 1015, 1027, 1036, 1048, 1067, 1071, 1083, 1085, 1089, 1097, 1105, 1109, 1131, 1132, 1141, 1145, 1152, 1156, 1161, 1165, 1177, 1180, 1181, 1188, 1195, 1209, 1216, 1227, 1233, 1247, 1252, 1260, 1272, 1284, 1285, 1289, 1296, 1309, 1313, 1317, 1331, 1343, 1344, 1349, 1356, 1357, 1360, 1363, 1371, 1376, 1377, 1392, 1395, 1409, 1415, 1428, 1435, 1443, 1455, 1459, 1463, 1471, 1477, 1487, 1496, 1500, 1520, 1528, 1529, 1531, 1533, 1538, 1540, 1541, 1545, 1547, 1555, 1566, 1577, 1591, 1597, 1614, 1616, 1621, 1627, 1633, 1635, 1636, 1643, 1647, 1649, 1655, 1659, 1665, 1668, 1677, 1679, 1685, 1687, 1692, 1693, 1695, 1696, 1703, 1707, 1711, 1713, 1724, 1736, 1740, 1743, 1744, 1763, 1764, 1769, 1772, 1784, 1788, 1791, 1795, 1799, 1812, 1813, 1815, 1831, 1837, 1846, 1852, 1853, 1867, 1869, 1872, 1877, 1883, 1884, 1887, 1891, 1893, 1895, 1897, 1906, 1908, 1929, 1936, 1940, 1943, 1952, 1960, 1961, 1964, 1967, 1975, 1977, 1981, 1983, 1985, 1988, 1989, 1991, 1993</partitions>

    <zone-id>0</zone-id>
  </server>
  <server>

    <id>5</id>
    <host>6.6.6.6</host>

    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>0, 4, 5, 9, 11, 13, 26, 28, 40, 42, 44, 47, 50, 52, 54, 57, 75, 76, 79, 81, 86, 88, 103, 111, 163, 167, 172, 173, 180, 184, 186, 188, 191, 192, 205, 211, 212, 213, 222, 228, 234, 239, 253, 256, 259, 261, 273, 282, 293, 296, 307, 309, 314, 322, 323, 326, 344, 346, 352, 356, 364, 372, 377, 382, 383, 397, 399, 403, 414, 415, 423, 429, 443, 445, 446, 452, 458, 463, 465, 471, 472, 473, 477, 481, 488, 491, 498, 504, 511, 514, 519, 524, 525, 535, 546, 554, 558, 560, 564, 567, 573, 575, 577, 578, 580, 600, 615, 620, 621, 624, 629, 632, 636, 642, 654, 656, 659, 661, 672, 675, 679, 683, 685, 691, 697, 698, 703, 707, 712, 713, 717, 727, 730, 733, 738, 743, 747, 748, 751, 760, 763, 771, 775, 786, 797, 800, 805, 808, 809, 813, 814, 826, 837, 852, 856, 861, 863, 874, 875, 878, 900, 907, 912, 923, 935, 938, 942, 961, 964, 967, 972, 978, 979, 993, 998, 1007, 1013, 1019, 1021, 1023, 1034, 1042, 1050, 1053, 1056, 1065, 1073, 1076, 1080, 1092, 1095, 1100, 1102, 1113, 1116, 1118, 1125, 1127, 1137, 1140, 1148, 1149, 1173, 1191, 1192, 1199, 1204, 1207, 1223, 1225, 1228, 1232, 1235, 1238, 1251, 1257, 1258, 1264, 1268, 1274, 1277, 1280, 1294, 1298, 1302, 1306, 1307, 1325, 1329, 1333, 1338, 1340, 1341, 1345, 1348, 1354, 1361, 1374, 1387, 1400, 1401, 1403, 1414, 1423, 1425, 1432, 1437, 1438, 1447, 1449, 1452, 1464, 1484, 1511, 1515, 1518, 1536, 1542, 1550, 1553, 1558, 1569, 1574, 1593, 1596, 1599, 1602, 1609, 1620, 1622, 1638, 1651, 1666, 1669, 1672, 1675, 1676, 1697, 1699, 1709, 1712, 1726, 1729, 1738, 1742, 1753, 1761, 1765, 1767, 1775, 1778, 1787, 1792, 1797, 1801, 1802, 1805, 1807, 1808, 1811, 1819, 1826, 1834, 1841, 1850, 1856, 1859, 1861, 1864, 1865, 1875, 1879, 1881, 1889, 1899, 1901, 1904, 1909, 1912, 1915, 1918, 1919, 1924, 1935, 1939, 1947, 1948, 1950, 1953, 1957, 1969, 1974, 1997</partitions>
    <zone-id>0</zone-id>
  </server>
</cluster>

[Arunachalam]

Upgrade the client to use the following branch 

https://github.com/voldemort/voldemort/tree/nioFixes

The client should work with the fixes in there.

Thanks,

Arun.

[redboy1972]

Ok, I will try that and post back the results.

[redboy1972]

Is there a doc somewhere how to put together a build environment.

Out of the box there are build errors that are likely environmental.

[root voldemort]# ant
Buildfile: /home/ec2-user/temp/voldemort/build.xml


BUILD FAILED
/home/ec2-user/temp/voldemort/build.xml:555: taskdef class org.apache.catalina.ant.InstallTask cannot be found
 using the classloader AntClassLoader[]

Commenting them out let Ant continue but then it ran into other environment issues.

 

 [root voldemort]# cat tomcat-tasks.properties
deploy=org.apache.catalina.ant.DeployTask
install=org.apache.catalina.ant.DeployTask
#install=org.apache.catalina.ant.InstallTask
list=org.apache.catalina.ant.ListTask
reload=org.apache.catalina.ant.ReloadTask
#remove=org.apache.catalina.ant.RemoveTask
resources=org.apache.catalina.ant.ResourcesTask
#roles=org.apache.catalina.ant.RolesTask
start=org.apache.catalina.ant.StartTask
stop=org.apache.catalina.ant.StopTask
undeploy=org.apache.catalina.ant.UndeployTask

export CATALINA_HOME=/opt/apache-tomcat-8.0.21/
export TOMCAT_HOME=/opt/apache-tomcat-8.0.21/
export ANT_HOME=/opt/apache-ant-1.9.4/
cp /opt/apache-tomcat-8.0.21/lib/catalina-ant.jar private-lib/
cp /opt/apache-tomcat-8.0.21/lib/catalina-ant.jar public-lib/
cp private-lib/tehuti-0.7-SNAPSHOT.jar public-lib/

Led me to this error:

    [javac] /home/ec2-user/temp/voldemort/src/java/voldemort/serialization/DefaultSerializerFactory.java:23: error: package org.apache.thrift does
 not exist
    [javac] import org.apache.thrift.TBase;
    [javac]                         ^
    [javac] /home/ec2-user/temp/voldemort/src/java/voldemort/serialization/thrift/MemoryBuffer.java:5: error: package org.apache.thrift.transport
does not exist
    [javac] import org.apache.thrift.transport.TTransport;

cp /home/ec2-user/temp/thrift-0.9.2/lib/java/build/libthrift-0.9.2.jar ~ec2-user/temp/voldemort/public-lib/

    [javac] /home/ec2-user/temp/voldemort/src/java/voldemort/store/rocksdb/PartitionPrefixedRocksDbStorageEngine.java:7: error: package org.rocksdb does not exist
    [javac] import org.rocksdb.RocksDB;

Got rocksdbjni-3.6.1.jar

    [javac] /home/ec2-user/temp/voldemort/contrib/hadoop-store-builder/src/java/voldemort/store/readonly/mr/azkaban/VoldemortSwapJob.java:38: error: package azkaban.jobExecutor does not exist
    [javac] import azkaban.jobExecutor.AbstractJob;
    [javac]                           ^
    [javac] /home/ec2-user/temp/voldemort/contrib/hadoop-store-builder/src/java/voldemort/store/readonly/mr/azkaban/VoldemortSwapJob.java:44: error: cannot find symbol
    [javac] public class VoldemortSwapJob extends AbstractJob {

got azkaban-2.5.0.jar

BUILD SUCESSFUL

Rebuild platform around voldemort-1.9.11.jar and updated dist,bin,lib directories and propagated it to 6 servers.  Restarted client/servers

GET/PUT/GETALL work

stop a single server and all 6 clients report this:

Exception thrown during operation.

voldemort.VoldemortException: 3 metadata refresh attempts failed.

        at voldemort.client.DefaultStoreClient.getAll(DefaultStoreClient.java:201)

        at voldemort.VoldemortClientShell.processGetAll(VoldemortClientShell.java:267)

        at voldemort.VoldemortClientShell.processCommands(VoldemortClientShell.java:301)

        at voldemort.VoldemortClientShell.process(VoldemortClientShell.java:128)

        at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:181)

Will try more tomorrow.  I replaced folders from 1.9 into 1.6 and things seem to not work.  I will start over the whole environment minus cluster.xml, stores.xml tomorrow.

[Félix GV]

Sorry about that,

We really ought to remove the ant build entirely. It is not supported anymore and just causing confusion ): ...

We now build with gradle:

./gradlew build

Or if you just want to compile without running the tests (they take a while...):

./gradlew jar

I hope that helps.

-F

--

[Arunachalam]

Also it would be great if you can tell us how did you end up using ant instead of gradle, our documentation is updated to use gradle. 

Thanks,

Arun.

[redboy1972]

more release_notes.txt

Release 1.9.11 on 04/23/2015

I come from Ant so when I extracted the source it was natural for me to use it.  While I succeeded in the build process the issue was not Ant vs Gradle that caused me issues but the unknown dependencies.  The 3 or so jar files I had to track down.  Now one of those jars may be ant related (catalina-ant.jar), the source seemed dependent on tehuti-0.7-SNAPSHOT.jar, rocksdbjni-3.6.1.jar, libthrift-0.9.2.jar and maybe others.

From README.md

### Download Code ###


```bash
cd ~/workspace
git clone https://github.com/voldemort/voldemort.git
cd voldemort
ant release
```


### Start Server ###


```
# in one terminal
bin/voldemort-server.sh config/single_node_cluster
```


### Use Client Shell ###


Client shell gives you fast access to the store. We already have a test store defined in the "single_node_cluster", whose key and value a
re both String.


```bash
# in another terminal
cd ~/workspace/voldemort
bin/voldemort-shell.sh test tcp://localhost:6666/
```

[Arunachalam]

Thanks will fix the README.md

Gradle should automatically download the dependencies from maven central if you are building on an internet connected machine.

Thanks,

Arun.

[Arunachalam]

Fixed the Readme.

QuickStart

You can refer to http://www.project-voldemort.com for more info

Download Code

cd ~/workspace
git clone https://github.com/voldemort/voldemort.git
cd

 voldemort
./gradlew clean jar

[redboy1972]

I would recommend removing Ant build and updating the README.md file to use gradle.

I would also like if there was a gradle task that would make a production directory that could be tar/zip/copied.  Right now even the tasks for build, jar, zip, tar don't do this.  Instead they seem to want to port around the code, build stuff, the '.' directory.  This requires a piecemeal of copies to stitch together something that I would distribute to my 6 lab machines.

[Arunachalam]

The tar task should produce a tar ball in your dist directory. (Something like)

dist/voldemort-1.9.11.tar.gz 

This is complete working copy of the source and binaries and you should be able to run the Voldemort from this tar ball without any additional tasks on the Production machine. But this contains source code as well, but their size is negligible when compared to all the dependencies it packs with it.

Thanks,

Arun.

On Thu, Apr 30, 2015 at 10:54 AM, redboy1972 <dee.l.e...@gmail.com> wrote:

I would recommend removing Ant build and updating the README.md file to use gradle.

I would also like if there was a gradle task that would make a production directory that could be tar/zip/copied.  Right now even the tasks for build, jar, zip, tar don't do this.  Instead they seem to want to port around the code, build stuff, the '.' directory.  This requires a piecemeal of copies to stitch together something that I would distribute to my 6 lab machines.

--

[redboy1972]

BUILD
[root temp]# git clone https://github.com/voldemort/voldemort.git nioFixes
Cloning into 'nioFixes'...
remote: Counting objects: 65749, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 65749 (delta 0), reused 0 (delta 0), pack-reused 65746
Receiving objects: 100% (65749/65749), 205.66 MiB | 15.85 MiB/s, done.
Resolving deltas: 100% (43508/43508), done.


[root temp]# cd nioFixes/


[root nioFixes]# ./gradlew jar
java source target compatibility version 1.5
:compileJava
warning: [options] bootstrap class path not set in conjunction with -source
1.5
/home/ec2-user/temp/nioFixes/src/java/voldemort/store/readonly/io/
ByteBufferCloser.java:21: warning: Cleaner is internal proprietary API and
may be removed in a future release
sun.misc.Cleaner cl = ((sun.nio.ch.DirectBuffer) buff).cleaner();
^
/home/ec2-user/temp/nioFixes/src/java/voldemort/store/readonly/io/
ByteBufferCloser.java:21: warning: DirectBuffer is internal proprietary API
and may be removed in a future release
sun.misc.Cleaner cl = ((sun.nio.ch.DirectBuffer) buff).cleaner();
^
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
3 warnings
:compileScala
[ant:scalac] warning: -target:jvm-1.5 is deprecated: use target for Java 1.6
or above.
[ant:scalac] one warning found
:processResources
:classes
:compileTestJava
warning: [options] bootstrap class path not set in conjunction with -source
1.5
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 warning
:compileTestScala UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses
:voldJar
:compileContribJava
warning: [options] bootstrap class path not set in conjunction with -source
1.5
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 warning
:compileContribScala UP-TO-DATE
:processContribResources UP-TO-DATE
:contribClasses
:testJar
:contribJar
:copyDeps
:srcJar
:jar


BUILD SUCCESSFUL


Total time: 34.124 secs




[root nioFixes]# mv config config.ORIG
[root nioFixes]# cp -rp ../../voldemort/config/ .
[root nioFixes]# ./gradlew tar
java source target compatibility version 1.5
:compileJava UP-TO-DATE
:compileScala UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:compileTestJava UP-TO-DATE
:compileTestScala UP-TO-DATE
:processTestResources UP-TO-DATE
:testClasses UP-TO-DATE
:voldJar UP-TO-DATE
:compileContribJava UP-TO-DATE
:compileContribScala UP-TO-DATE
:processContribResources UP-TO-DATE
:contribClasses UP-TO-DATE
:testJar UP-TO-DATE
:contribJar UP-TO-DATE
:copyDeps UP-TO-DATE
:srcJar UP-TO-DATE
:jar UP-TO-DATE
:copySources
:tar


BUILD SUCCESSFUL


Total time: 18.632 secs





INSTALL
[root nioFixes]# cd ../..
[root voldemort_19]# tar -xzf
/home/ec2-user/temp/nioFixes/dist/voldemort-1.9.11.tar.gz
[root ec2-user]# cd voldemort-1.9.11
[root voldemort-1.9.11]# rm -rf build.gradle build.xml clients config.ORIG
contrib CONTRIBUTORS docs example gradle gradle.properties gradlew
gradlew.bat LICENSE NOTES NOTICE public-lib private-lib README.md
release_notes.txt settings.gradle src test tomcat-tasks.properties
voldemort-contrib web.xml bin/*.bat dist/nioFixes-1.9.11-src.jar
dist/voldemort-contrib-1.9.11.jar dist/voldemort-test-1.9.11.jar
[root voldemort-1.9.11]# ls -R
.:
bin config dist lib


./bin:
generate_cluster_xml.py repeat-junit.sh voldemort-
coordinator-admin.sh voldemort-shell.sh
rebalance-cluster-expansion.sh repeat-junit-test.sh voldemort-
coordinator.sh voldemort-stop.sh
rebalance-new-zoned-cluster.sh run-class.sh voldemort-
performance-tool.sh voldemort-thin-client-shell.sh
rebalance-shuffle.sh vadmin.sh voldemort-prod-
server.sh
rebalance-zone-expansion.sh voldemort-admin-tool.sh voldemort-scala-
shell.sh
rebalance-zone-shrinkage.sh voldemort-convert-bdb.sh voldemort-server.
sh


./config:
cluster.xml server.properties stores.xml


./dist:
voldemort-1.9.11.jar voldemort-contrib-1.9.11.jar


./lib:
activation-1.1.jar commons-logging-1.1.1.jar jetty-util-
6.1.26.jar pegasus-common-1.8.3.jar
annotations-api-6.0.43.jar commons-math-2.1.jar jline-0.9.
94.jar protobuf-java-2.3.0.jar
ant-1.6.5.jar commons-net-1.4.1.jar jna-3.2.
7.jar qdox-1.10.1.jar
ant-1.7.1.jar commons-pool-1.3.jar joda-time-
1.6.jar r2-1.8.3.jar
ant-launcher-1.7.1.jar compress-lzf-0.9.1.jar jopt-simple
-4.6.jar rocksdbjni-3.6.2.jar
asm-3.2.jar core-3.1.1.jar json-
20070829.jar scala-compiler-2.10.4.jar
avro-1.4.0.jar coyote-6.0.43.jar jsp-2.1-6.1
.14.jar scala-library-2.10.4.jar
azkaban-2.5.0.jar data-1.8.3.jar jsp-api-2.1
-6.1.14.jar scala-reflect-2.10.4.jar
catalina-6.0.43.jar guava-14.0.1.jar juli-6.0.
43.jar servlet-api-2.5-20081211.jar
catalina-ant-6.0.43.jar hadoop-auth-2.0.5-alpha.jar junit-3.8.
1.jar servlet-api-2.5-6.1.14.jar
commons-beanutils-1.7.0.jar hadoop-core-1.0.4.jar kfs-0.3.jar
servlet-api-2.5.jar
commons-beanutils-core-1.8.0.jar hsqldb-1.8.0.10.jar krati-0.4.
9.jar servlet-api-6.0.43.jar
commons-cli-1.2.jar httpclient-4.1.2.jar libthrift-
0.5.0.0.jar slf4j-api-1.6.2.jar
commons-codec-1.4.jar httpcore-4.1.2.jar log4j-1.2.
17.jar slf4j-log4j12-1.6.2.jar
commons-collections-3.2.1.jar jackson-core-asl-1.9.13.jar mail-1.4.
1.jar snappy-0.2.jar
commons-configuration-1.6.jar jackson-mapper-asl-1.9.13.jar mina-core-
1.1.7.jar tehuti-0.6.jar
commons-dbcp-1.2.2.jar jasper-compiler-5.5.12.jar netty-3.2.
3.Final.jar typica-1.7.2.jar
commons-digester-1.8.jar jasper-runtime-5.5.12.jar netty-3.5.
8.Final.jar velocity-1.6.4.jar
commons-el-1.0.jar jdom-1.1.jar oro-2.0.
8.jar xmlenc-0.52.jar
commons-httpclient-3.1.jar je-5.0.88.jar paranamer-
2.2.jar
commons-io-1.4.jar jets3t-0.7.1.jar paranamer-
ant-2.2.jar
commons-lang-2.5.jar jetty-6.1.26.jar paranamer-
generator-2.2.jar




[root ec2-user]# tar -czf voldemort.tar.19.prod.gz voldemort-1.9.11
Copied the above to all 6 Amazon instances
[root ec2-user]# tar -xzf voldemort.tar.19.prod.gz
[root ec2-user]# cd voldemort-1.9.11



CUSTOMIZE

[root voldemort-1.9.11]# cat config/cluster.xml

<cluster>
<name>VZWCluster</name>
<zone>
<zone-id>0</zone-id>
<proximity-list>1</proximity-list>
</zone>
<zone>
<zone-id>1</zone-id>
<proximity-list>0</proximity-list>
</zone>
<server>
<id>0</id>

<host>0.0.0.0</host>

<http-port>8081</http-port>
<socket-port>6666</socket-port>
<admin-port>6667</admin-port>

<partitions>1, 7, 17, 23, 32, 35, 60, 71, 72, 77, 80, 84, 87, 91, 95,
101, 109, 114, 117, 120, 128, 132, 142, 146, 151, 155, 159, 177, 189, 193,
197, 200, 203, 208, 216, 227, 244, 247, 255, 256, 265, 276, 279, 285, 288,
292, 299, 304, 311, 316, 321, 339, 342, 353, 356, 359, 369, 371, 373, 383,
386, 389, 400, 405, 407, 420, 425, 436, 437, 449, 453, 468, 471, 482, 485,
499, 502, 508, 522, 527, 531, 532, 533, 549, 550, 554, 563, 568, 571, 581,
583, 584, 589, 602, 607, 618, 623, 635, 640, 643, 665, 668, 673, 687, 704,
705, 708, 718, 721, 723, 724, 731, 737, 745, 752, 753, 767, 769, 775, 777,
781, 783, 789, 793, 797, 806, 817, 825, 832, 835, 838, 845, 848, 853, 867,
872, 885, 888, 891, 901, 906, 919, 927, 931, 936, 955, 960, 968, 973, 975,
981, 996, 1001, 1005, 1010, 1016, 1032, 1035, 1044, 1051, 1057, 1061, 1064,
1068, 1074, 1077, 1087, 1088, 1090, 1099, 1103, 1108, 1111, 1112, 1120, 1123
, 1133, 1153, 1159, 1163, 1164, 1167, 1169, 1172, 1176, 1184, 1185, 1196,
1197, 1203, 1208, 1212, 1213, 1217, 1220, 1228, 1229, 1241, 1245, 1248, 1253
, 1256, 1265, 1279, 1282, 1291, 1300, 1311, 1319, 1320, 1321, 1323, 1336,
1341, 1353, 1357, 1359, 1364, 1367, 1380, 1381, 1383, 1389, 1397, 1401, 1406
, 1408, 1412, 1419, 1420, 1424, 1431, 1440, 1441, 1444, 1448, 1456, 1457,
1468, 1473, 1475, 1480, 1483, 1490, 1499, 1501, 1505, 1508, 1513, 1521, 1525
, 1527, 1532, 1535, 1536, 1542, 1548, 1550, 1561, 1563, 1576, 1583, 1588,
1601, 1603, 1611, 1616, 1617, 1623, 1625, 1639, 1642, 1647, 1656, 1662, 1663
, 1672, 1675, 1679, 1681, 1685, 1688, 1699, 1700, 1715, 1716, 1717, 1720,
1721, 1726, 1732, 1748, 1749, 1751, 1752, 1756, 1759, 1777, 1793, 1804, 1805
, 1809, 1816, 1823, 1824, 1829, 1835, 1839, 1843, 1848, 1849, 1850, 1853,
1868, 1871, 1873, 1880, 1900, 1905, 1920, 1926, 1933, 1940, 1944, 1945, 1960
, 1963, 1965, 1971, 1980, 1984, 1987, 1992, 1996</partitions>

<zone-id>0</zone-id>
</server>
<server>

<id>2</id>

<host>2.2.2.2</host>
<http-port>8081</http-port>
<socket-port>6666</socket-port>
<admin-port>6667</admin-port>

<partitions>0, 4, 5, 11, 13, 18, 20, 26, 28, 42, 44, 47, 50, 52, 54, 57,
63, 68, 75, 79, 86, 92, 103, 111, 148, 163, 172, 173, 180, 182, 184, 186,
188, 192, 205, 211, 212, 213, 222, 228, 230, 239, 253, 259, 261, 268, 271,
273, 282, 289, 293, 296, 307, 309, 314, 322, 323, 326, 346, 352, 360, 364,
372, 377, 379, 382, 397, 398, 403, 414, 415, 423, 429, 432, 443, 445, 446,
452, 458, 463, 465, 472, 473, 477, 481, 488, 491, 498, 504, 511, 514, 519,
524, 525, 535, 546, 552, 555, 564, 567, 573, 575, 577, 580, 600, 615, 620,
621, 624, 629, 631, 632, 636, 642, 654, 656, 659, 661, 663, 672, 675, 679,
683, 685, 691, 692, 693, 697, 698, 703, 707, 712, 727, 730, 733, 738, 743,
747, 748, 751, 757, 760, 763, 771, 786, 800, 808, 809, 813, 814, 821, 826,
829, 837, 852, 856, 861, 863, 875, 878, 900, 902, 912, 923, 935, 938, 942,
961, 964, 967, 972, 978, 979, 993, 998, 1007, 1019, 1021, 1023, 1030, 1034,
1042, 1050, 1056, 1065, 1073, 1076, 1080, 1092, 1095, 1100, 1102, 1106, 1113
, 1116, 1118, 1126, 1137, 1140, 1144, 1148, 1149, 1170, 1173, 1191, 1199,
1204, 1207, 1223, 1232, 1235, 1238, 1251, 1257, 1262, 1264, 1268, 1273, 1274
, 1277, 1286, 1294, 1298, 1302, 1306, 1307, 1325, 1327, 1329, 1332, 1333,
1338, 1345, 1348, 1352, 1354, 1361, 1369, 1370, 1374, 1387, 1400, 1403, 1414
, 1423, 1425, 1432, 1437, 1438, 1447, 1449, 1452, 1458, 1464, 1484, 1492,
1506, 1515, 1518, 1546, 1553, 1558, 1569, 1572, 1574, 1586, 1593, 1596, 1599
, 1602, 1609, 1615, 1620, 1629, 1638, 1651, 1661, 1666, 1669, 1676, 1694,
1697, 1709, 1712, 1724, 1729, 1738, 1742, 1753, 1761, 1765, 1767, 1772, 1775
, 1778, 1781, 1787, 1792, 1797, 1798, 1801, 1802, 1808, 1811, 1819, 1826,
1834, 1841, 1856, 1859, 1861, 1864, 1865, 1875, 1879, 1899, 1901, 1904, 1909
, 1915, 1918, 1919, 1924, 1935, 1939, 1947, 1948, 1950, 1953, 1969, 1972,

1974, 1997</partitions>
<zone-id>0</zone-id>
</server>

<server>
<id>4</id>

<host>4.4.4.4</host>
<http-port>8081</http-port>
<socket-port>6666</socket-port>
<admin-port>6667</admin-port>

<partitions>8, 19, 25, 31, 33, 36, 37, 43, 64, 73, 76, 81, 97, 104, 108,
112, 115, 124, 125, 129, 133, 137, 140, 141, 149, 152, 165, 171, 176, 179,
181, 185, 191, 195, 196, 199, 204, 219, 225, 232, 236, 241, 248, 267, 272,
274, 277, 281, 291, 300, 301, 306, 315, 317, 324, 325, 327, 331, 335, 347,
350, 387, 391, 393, 409, 412, 419, 427, 431, 433, 438, 447, 455, 456, 460,
464, 467, 475, 476, 479, 487, 495, 501, 503, 507, 513, 515, 521, 523, 526,
537, 541, 544, 553, 557, 560, 561, 593, 596, 599, 604, 611, 617, 622, 625,
628, 645, 648, 651, 660, 669, 674, 681, 684, 689, 701, 706, 713, 715, 719,
732, 740, 749, 765, 772, 787, 801, 816, 819, 822, 824, 830, 833, 836, 839,
841, 843, 851, 855, 859, 880, 884, 886, 887, 889, 892, 899, 903, 904, 909,
915, 920, 921, 924, 925, 929, 932, 933, 944, 948, 949, 953, 980, 984, 985,
988, 991, 999, 1004, 1015, 1027, 1036, 1048, 1067, 1071, 1083, 1089, 1093,
1096, 1097, 1105, 1115, 1127, 1131, 1132, 1141, 1145, 1152, 1156, 1161, 1165
, 1171, 1174, 1177, 1181, 1188, 1192, 1195, 1209, 1216, 1227, 1233, 1247,
1249, 1260, 1267, 1280, 1284, 1285, 1289, 1296, 1299, 1305, 1309, 1313, 1317
, 1331, 1343, 1344, 1349, 1356, 1360, 1363, 1371, 1375, 1376, 1377, 1392,
1395, 1409, 1415, 1428, 1435, 1455, 1459, 1461, 1463, 1471, 1477, 1487, 1491

, 1496, 1500, 1520, 1528, 1529, 1531, 1533, 1538, 1540, 1541, 1545, 1547,

1555, 1577, 1581, 1591, 1597, 1614, 1621, 1627, 1633, 1635, 1636, 1643, 1649
, 1653, 1655, 1659, 1664, 1665, 1668, 1677, 1687, 1692, 1693, 1695, 1696,
1702, 1707, 1711, 1736, 1740, 1743, 1744, 1763, 1764, 1769, 1784, 1788, 1791
, 1795, 1799, 1807, 1812, 1813, 1831, 1837, 1846, 1852, 1867, 1869, 1872,
1877, 1881, 1883, 1884, 1887, 1891, 1893, 1895, 1897, 1906, 1908, 1911, 1927
, 1930, 1936, 1941, 1943, 1952, 1961, 1964, 1967, 1973, 1975, 1981, 1983,

1985, 1988, 1989, 1991, 1993</partitions>

<zone-id>1</zone-id>
</server>



<server>

<id>1</id>

<host>1.1.1.1</host>
<http-port>8081</http-port>
<socket-port>6666</socket-port>
<admin-port>6667</admin-port>

<partitions>2, 9, 12, 21, 24, 29, 38, 40, 45, 53, 56, 66, 69, 74, 82, 89
, 98, 99, 106, 110, 118, 123, 126, 130, 134, 138, 139, 145, 153, 158, 161,
162, 167, 170, 178, 190, 194, 218, 224, 233, 235, 237, 240, 242, 246, 251,
258, 263, 283, 286, 302, 305, 313, 319, 330, 333, 334, 337, 338, 343, 348,
351, 362, 363, 368, 375, 385, 390, 392, 396, 401, 417, 426, 428, 435, 440,
442, 454, 462, 469, 490, 493, 496, 512, 517, 536, 539, 543, 547, 558, 576,
579, 586, 592, 594, 595, 597, 601, 606, 608, 610, 614, 627, 634, 639, 647,
649, 650, 652, 653, 671, 700, 709, 711, 714, 726, 735, 741, 759, 761, 764,
773, 778, 780, 788, 796, 798, 803, 804, 811, 815, 828, 831, 842, 849, 860,
868, 869, 871, 874, 877, 879, 882, 883, 890, 894, 897, 905, 908, 910, 914,
917, 922, 934, 945, 947, 951, 956, 958, 965, 983, 987, 990, 995, 1000, 1002,
1009, 1013, 1017, 1020, 1024, 1026, 1037, 1040, 1043, 1047, 1053, 1054, 1062
, 1069, 1078, 1082, 1086, 1091, 1094, 1119, 1125, 1128, 1130, 1135, 1155,
1160, 1175, 1180, 1183, 1187, 1190, 1193, 1206, 1214, 1219, 1224, 1226, 1240
, 1243, 1259, 1266, 1269, 1271, 1278, 1281, 1283, 1288, 1292, 1295, 1297,
1303, 1304, 1308, 1314, 1315, 1330, 1335, 1339, 1342, 1346, 1362, 1373, 1378
, 1384, 1386, 1390, 1393, 1398, 1404, 1410, 1417, 1427, 1429, 1434, 1436,
1442, 1451, 1454, 1460, 1469, 1472, 1476, 1481, 1485, 1488, 1494, 1497, 1502
, 1511, 1530, 1539, 1543, 1556, 1560, 1566, 1568, 1573, 1579, 1580, 1582,
1589, 1605, 1607, 1608, 1613, 1619, 1626, 1632, 1634, 1644, 1650, 1654, 1657
, 1667, 1670, 1673, 1691, 1701, 1704, 1706, 1708, 1735, 1737, 1745, 1746,
1755, 1768, 1783, 1785, 1786, 1789, 1800, 1806, 1814, 1818, 1822, 1832, 1840
, 1845, 1863, 1882, 1886, 1889, 1892, 1894, 1896, 1903, 1910, 1912, 1916,
1922, 1925, 1929, 1931, 1932, 1942, 1951, 1954, 1957, 1962, 1968, 1977, 1979
, 1982, 1986, 1990, 1994, 1999</partitions>

<zone-id>0</zone-id>
</server>
<server>

<id>3</id>

<host>3.3.3.3</host>
<http-port>8081</http-port>
<socket-port>6666</socket-port>
<admin-port>6667</admin-port>

<partitions>3, 10, 15, 16, 27, 34, 41, 46, 51, 58, 59, 62, 65, 70, 83,
85, 93, 102, 113, 131, 135, 136, 143, 144, 147, 150, 156, 157, 166, 168, 169
, 175, 183, 201, 206, 209, 214, 217, 221, 223, 226, 229, 234, 243, 252, 260,
266, 270, 278, 290, 294, 308, 312, 320, 332, 344, 349, 354, 355, 366, 367,
374, 376, 378, 380, 388, 394, 395, 399, 402, 406, 408, 410, 413, 416, 421,
422, 424, 430, 448, 459, 461, 474, 478, 480, 486, 489, 494, 497, 500, 506,
509, 510, 520, 528, 530, 534, 559, 562, 569, 570, 587, 588, 590, 591, 603,
605, 616, 626, 630, 655, 657, 658, 664, 667, 680, 682, 686, 688, 690, 694,
695, 699, 710, 722, 728, 734, 736, 742, 744, 756, 762, 766, 776, 784, 795,
810, 818, 820, 834, 840, 846, 847, 857, 864, 870, 873, 893, 896, 911, 913,
916, 918, 930, 937, 941, 952, 954, 957, 962, 963, 966, 970, 977, 982, 986,
989, 994, 1003, 1008, 1011, 1014, 1028, 1029, 1038, 1039, 1041, 1049, 1055,
1058, 1063, 1070, 1072, 1075, 1081, 1084, 1101, 1104, 1110, 1114, 1134, 1138
, 1142, 1146, 1147, 1151, 1154, 1157, 1162, 1186, 1200, 1202, 1210, 1215,
1218, 1221, 1225, 1230, 1231, 1236, 1239, 1242, 1244, 1250, 1254, 1255, 1263
, 1287, 1293, 1301, 1316, 1318, 1324, 1328, 1340, 1347, 1350, 1351, 1365,
1366, 1368, 1379, 1385, 1391, 1402, 1407, 1411, 1418, 1421, 1422, 1426, 1433
, 1439, 1443, 1445, 1450, 1462, 1466, 1467, 1482, 1498, 1504, 1507, 1512,
1517, 1519, 1537, 1544, 1549, 1551, 1559, 1562, 1564, 1567, 1571, 1575, 1578
, 1585, 1590, 1592, 1594, 1600, 1604, 1610, 1612, 1618, 1630, 1631, 1645,
1646, 1652, 1658, 1660, 1682, 1683, 1686, 1698, 1703, 1714, 1719, 1722, 1725
, 1727, 1733, 1734, 1739, 1754, 1758, 1760, 1762, 1766, 1770, 1773, 1774,
1779, 1794, 1803, 1815, 1817, 1821, 1827, 1828, 1830, 1833, 1842, 1844, 1847
, 1855, 1858, 1870, 1878, 1885, 1890, 1898, 1902, 1913, 1923, 1934, 1937,
1946, 1956, 1959, 1966, 1976</partitions>

<zone-id>1</zone-id>
</server>


<server>

<id>5</id>

<host>5.5.5.5</host>
<http-port>8081</http-port>
<socket-port>6666</socket-port>
<admin-port>6667</admin-port>

<partitions>6, 14, 22, 30, 39, 48, 49, 55, 61, 67, 78, 88, 90, 94, 96,
100, 105, 107, 116, 119, 121, 122, 127, 154, 160, 164, 174, 187, 198, 202,
207, 210, 215, 220, 231, 238, 245, 249, 250, 254, 257, 262, 264, 269, 275,
280, 284, 287, 295, 297, 298, 303, 310, 318, 328, 329, 336, 340, 341, 345,
357, 358, 361, 365, 370, 381, 384, 404, 411, 418, 434, 439, 441, 444, 450,
451, 457, 466, 470, 483, 484, 492, 505, 516, 518, 529, 538, 540, 542, 545,
548, 551, 556, 565, 566, 572, 574, 578, 582, 585, 598, 609, 612, 613, 619,
633, 637, 638, 641, 644, 646, 662, 666, 670, 676, 677, 678, 696, 702, 716,
717, 720, 725, 729, 739, 746, 750, 754, 755, 758, 768, 770, 774, 779, 782,
785, 790, 791, 792, 794, 799, 802, 805, 807, 812, 823, 827, 844, 850, 854,
858, 862, 865, 866, 876, 881, 895, 898, 907, 926, 928, 939, 940, 943, 946,
950, 959, 969, 971, 974, 976, 992, 997, 1006, 1012, 1018, 1022, 1025, 1031,
1033, 1045, 1046, 1052, 1059, 1060, 1066, 1079, 1085, 1098, 1107, 1109, 1117
, 1121, 1122, 1124, 1129, 1136, 1139, 1143, 1150, 1158, 1166, 1168, 1178,
1179, 1182, 1189, 1194, 1198, 1201, 1205, 1211, 1222, 1234, 1237, 1246, 1252
, 1258, 1261, 1270, 1272, 1275, 1276, 1290, 1310, 1312, 1322, 1326, 1334,
1337, 1355, 1358, 1372, 1382, 1388, 1394, 1396, 1399, 1405, 1413, 1416, 1430
, 1446, 1453, 1465, 1470, 1474, 1478, 1479, 1486, 1489, 1493, 1495, 1503,
1509, 1510, 1514, 1516, 1522, 1523, 1524, 1526, 1534, 1552, 1554, 1557, 1565
, 1570, 1584, 1587, 1595, 1598, 1606, 1622, 1624, 1628, 1637, 1640, 1641,
1648, 1671, 1674, 1678, 1680, 1684, 1689, 1690, 1705, 1710, 1713, 1718, 1723
, 1728, 1730, 1731, 1741, 1747, 1750, 1757, 1771, 1776, 1780, 1782, 1790,
1796, 1810, 1820, 1825, 1836, 1838, 1851, 1854, 1857, 1860, 1862, 1866, 1874
, 1876, 1888, 1907, 1914, 1917, 1921, 1928, 1938, 1949, 1955, 1958, 1970,
1978, 1995, 1998, 2000</partitions>
<zone-id>1</zone-id>
</server>
</cluster>


[root voldemort-1.9.11]# cat config/stores.xml

[root voldemort-1.9.11]# vi config/server.properties
updated "node.id=###" for each server


TEST

[root voldemort-1.9.11]# bin/voldemort-server.sh .


Started clients
[ec2-user voldemort-1.9.11]$ bin/voldemort-shell.sh test tcp:
//localhost:6666
Established connection to test via tcp://localhost:6666
>
> put "test1" "1"
> put "test2" "2"
> put "test3" "3"
> put "test4" "4"
> put "test5" "5"
> getall "test1" "test2" "test3" "test4" "test5"
test1 => version(0:1) ts:1430419806869: "1"
test4 => version(0:1) ts:1430419811015: "4"
test5 => version(0:1) ts:1430419811021: "5"
test2 => version(0:1) ts:1430419808253: "2"
test3 => version(0:1) ts:1430419808995: "3"


US-West (zone0) US-East (zone1)
0 1
2 3
4 5


I stopped server 0.......PASS
I stopped server 1.......PASS
I stopped server 2.......PASS
I stopped server 0,2,4...FAIL (NOTE that Voldemort 1.6 did pass this test.)
> getall "test1" "test2" "test3" "test4" "test5"
Exception thrown during operation.
voldemort.store.InsufficientOperationalNodesException: 1 get alls required,
but 0 succeeded. Failing nodes : [Node 2.2.2.2 Id:2 in zone 0 partitionList
:[0, 4, 5, 11, 13, 18, 20, 26, 28, 42, 44, 47, 50, 52, 54, 57, 63, 68, 75,
79, 86, 92, 103, 111, 148, 163, 172, 173, 180, 182, 184, 186, 188, 192, 205,
211, 212, 213, 222, 228, 230, 239, 253, 259, 261, 268, 271, 273, 282, 289,
293, 296, 307, 309, 314, 322, 323, 326, 346, 352, 360, 364, 372, 377, 379,
382, 397, 398, 403, 414, 415, 423, 429, 432, 443, 445, 446, 452, 458, 463,
465, 472, 473, 477, 481, 488, 491, 498, 504, 511, 514, 519, 524, 525, 535,
546, 552, 555, 564, 567, 573, 575, 577, 580, 600, 615, 620, 621, 624, 629,
631, 632, 636, 642, 654, 656, 659, 661, 663, 672, 675, 679, 683, 685, 691,
692, 693, 697, 698, 703, 707, 712, 727, 730, 733, 738, 743, 747, 748, 751,
757, 760, 763, 771, 786, 800, 808, 809, 813, 814, 821, 826, 829, 837, 852,
856, 861, 863, 875, 878, 900, 902, 912, 923, 935, 938, 942, 961, 964, 967,
972, 978, 979, 993, 998, 1007, 1019, 1021, 1023, 1030, 1034, 1042, 1050,
1056, 1065, 1073, 1076, 1080, 1092, 1095, 1100, 1102, 1106, 1113, 1116, 1118
, 1126, 1137, 1140, 1144, 1148, 1149, 1170, 1173, 1191, 1199, 1204, 1207,
1223, 1232, 1235, 1238, 1251, 1257, 1262, 1264, 1268, 1273, 1274, 1277, 1286
, 1294, 1298, 1302, 1306, 1307, 1325, 1327, 1329, 1332, 1333, 1338, 1345,
1348, 1352, 1354, 1361, 1369, 1370, 1374, 1387, 1400, 1403, 1414, 1423, 1425
, 1432, 1437, 1438, 1447, 1449, 1452, 1458, 1464, 1484, 1492, 1506, 1515,
1518, 1546, 1553, 1558, 1569, 1572, 1574, 1586, 1593, 1596, 1599, 1602, 1609
, 1615, 1620, 1629, 1638, 1651, 1661, 1666, 1669, 1676, 1694, 1697, 1709,
1712, 1724, 1729, 1738, 1742, 1753, 1761, 1765, 1767, 1772, 1775, 1778, 1781
, 1787, 1792, 1797, 1798, 1801, 1802, 1808, 1811, 1819, 1826, 1834, 1841,
1856, 1859, 1861, 1864, 1865, 1875, 1879, 1899, 1901, 1904, 1909, 1915, 1918
, 1919, 1924, 1935, 1939, 1947, 1948, 1950, 1953, 1969, 1972, 1974, 1997],
Node 3.3.3.3 Id:3 in zone 1 partitionList:[3, 10, 15, 16, 27, 34, 41, 46, 51
, 58, 59, 62, 65, 70, 83, 85, 93, 102, 113, 131, 135, 136, 143, 144, 147,
150, 156, 157, 166, 168, 169, 175, 183, 201, 206, 209, 214, 217, 221, 223,
226, 229, 234, 243, 252, 260, 266, 270, 278, 290, 294, 308, 312, 320, 332,
344, 349, 354, 355, 366, 367, 374, 376, 378, 380, 388, 394, 395, 399, 402,
406, 408, 410, 413, 416, 421, 422, 424, 430, 448, 459, 461, 474, 478, 480,
486, 489, 494, 497, 500, 506, 509, 510, 520, 528, 530, 534, 559, 562, 569,
570, 587, 588, 590, 591, 603, 605, 616, 626, 630, 655, 657, 658, 664, 667,
680, 682, 686, 688, 690, 694, 695, 699, 710, 722, 728, 734, 736, 742, 744,
756, 762, 766, 776, 784, 795, 810, 818, 820, 834, 840, 846, 847, 857, 864,
870, 873, 893, 896, 911, 913, 916, 918, 930, 937, 941, 952, 954, 957, 962,
963, 966, 970, 977, 982, 986, 989, 994, 1003, 1008, 1011, 1014, 1028, 1029,
1038, 1039, 1041, 1049, 1055, 1058, 1063, 1070, 1072, 1075, 1081, 1084, 1101
, 1104, 1110, 1114, 1134, 1138, 1142, 1146, 1147, 1151, 1154, 1157, 1162,
1186, 1200, 1202, 1210, 1215, 1218, 1221, 1225, 1230, 1231, 1236, 1239, 1242
, 1244, 1250, 1254, 1255, 1263, 1287, 1293, 1301, 1316, 1318, 1324, 1328,
1340, 1347, 1350, 1351, 1365, 1366, 1368, 1379, 1385, 1391, 1402, 1407, 1411
, 1418, 1421, 1422, 1426, 1433, 1439, 1443, 1445, 1450, 1462, 1466, 1467,
1482, 1498, 1504, 1507, 1512, 1517, 1519, 1537, 1544, 1549, 1551, 1559, 1562
, 1564, 1567, 1571, 1575, 1578, 1585, 1590, 1592, 1594, 1600, 1604, 1610,
1612, 1618, 1630, 1631, 1645, 1646, 1652, 1658, 1660, 1682, 1683, 1686, 1698
, 1703, 1714, 1719, 1722, 1725, 1727, 1733, 1734, 1739, 1754, 1758, 1760,
1762, 1766, 1770, 1773, 1774, 1779, 1794, 1803, 1815, 1817, 1821, 1827, 1828
, 1830, 1833, 1842, 1844, 1847, 1855, 1858, 1870, 1878, 1885, 1890, 1898,
1902, 1913, 1923, 1934, 1937, 1946, 1956, 1959, 1966, 1976], Node 3.3.3.3 Id
:3 in zone 1 partitionList:[3, 10, 15, 16, 27, 34, 41, 46, 51, 58, 59, 62,
65, 70, 83, 85, 93, 102, 113, 131, 135, 136, 143, 144, 147, 150, 156, 157,
166, 168, 169, 175, 183, 201, 206, 209, 214, 217, 221, 223, 226, 229, 234,
243, 252, 260, 266, 270, 278, 290, 294, 308, 312, 320, 332, 344, 349, 354,
355, 366, 367, 374, 376, 378, 380, 388, 394, 395, 399, 402, 406, 408, 410,
413, 416, 421, 422, 424, 430, 448, 459, 461, 474, 478, 480, 486, 489, 494,
497, 500, 506, 509, 510, 520, 528, 530, 534, 559, 562, 569, 570, 587, 588,
590, 591, 603, 605, 616, 626, 630, 655, 657, 658, 664, 667, 680, 682, 686,
688, 690, 694, 695, 699, 710, 722, 728, 734, 736, 742, 744, 756, 762, 766,
776, 784, 795, 810, 818, 820, 834, 840, 846, 847, 857, 864, 870, 873, 893,
896, 911, 913, 916, 918, 930, 937, 941, 952, 954, 957, 962, 963, 966, 970,
977, 982, 986, 989, 994, 1003, 1008, 1011, 1014, 1028, 1029, 1038, 1039,
1041, 1049, 1055, 1058, 1063, 1070, 1072, 1075, 1081, 1084, 1101, 1104, 1110
, 1114, 1134, 1138, 1142, 1146, 1147, 1151, 1154, 1157, 1162, 1186, 1200,
1202, 1210, 1215, 1218, 1221, 1225, 1230, 1231, 1236, 1239, 1242, 1244, 1250
, 1254, 1255, 1263, 1287, 1293, 1301, 1316, 1318, 1324, 1328, 1340, 1347,
1350, 1351, 1365, 1366, 1368, 1379, 1385, 1391, 1402, 1407, 1411, 1418, 1421
, 1422, 1426, 1433, 1439, 1443, 1445, 1450, 1462, 1466, 1467, 1482, 1498,
1504, 1507, 1512, 1517, 1519, 1537, 1544, 1549, 1551, 1559, 1562, 1564, 1567
, 1571, 1575, 1578, 1585, 1590, 1592, 1594, 1600, 1604, 1610, 1612, 1618,
1630, 1631, 1645, 1646, 1652, 1658, 1660, 1682, 1683, 1686, 1698, 1703, 1714
, 1719, 1722, 1725, 1727, 1733, 1734, 1739, 1754, 1758, 1760, 1762, 1766,
1770, 1773, 1774, 1779, 1794, 1803, 1815, 1817, 1821, 1827, 1828, 1830, 1833
, 1842, 1844, 1847, 1855, 1858, 1870, 1878, 1885, 1890, 1898, 1902, 1913,
1923, 1934, 1937, 1946, 1956, 1959, 1966, 1976], Node 2.2.2.2 Id:2 in zone 0
partitionList:[0, 4, 5, 11, 13, 18, 20, 26, 28, 42, 44, 47, 50, 52, 54, 57,
63, 68, 75, 79, 86, 92, 103, 111, 148, 163, 172, 173, 180, 182, 184, 186,
188, 192, 205, 211, 212, 213, 222, 228, 230, 239, 253, 259, 261, 268, 271,
273, 282, 289, 293, 296, 307, 309, 314, 322, 323, 326, 346, 352, 360, 364,
372, 377, 379, 382, 397, 398, 403, 414, 415, 423, 429, 432, 443, 445, 446,
452, 458, 463, 465, 472, 473, 477, 481, 488, 491, 498, 504, 511, 514, 519,
524, 525, 535, 546, 552, 555, 564, 567, 573, 575, 577, 580, 600, 615, 620,
621, 624, 629, 631, 632, 636, 642, 654, 656, 659, 661, 663, 672, 675, 679,
683, 685, 691, 692, 693, 697, 698, 703, 707, 712, 727, 730, 733, 738, 743,
747, 748, 751, 757, 760, 763, 771, 786, 800, 808, 809, 813, 814, 821, 826,
829, 837, 852, 856, 861, 863, 875, 878, 900, 902, 912, 923, 935, 938, 942,
961, 964, 967, 972, 978, 979, 993, 998, 1007, 1019, 1021, 1023, 1030, 1034,
1042, 1050, 1056, 1065, 1073, 1076, 1080, 1092, 1095, 1100, 1102, 1106, 1113
, 1116, 1118, 1126, 1137, 1140, 1144, 1148, 1149, 1170, 1173, 1191, 1199,
1204, 1207, 1223, 1232, 1235, 1238, 1251, 1257, 1262, 1264, 1268, 1273, 1274
, 1277, 1286, 1294, 1298, 1302, 1306, 1307, 1325, 1327, 1329, 1332, 1333,
1338, 1345, 1348, 1352, 1354, 1361, 1369, 1370, 1374, 1387, 1400, 1403, 1414
, 1423, 1425, 1432, 1437, 1438, 1447, 1449, 1452, 1458, 1464, 1484, 1492,
1506, 1515, 1518, 1546, 1553, 1558, 1569, 1572, 1574, 1586, 1593, 1596, 1599
, 1602, 1609, 1615, 1620, 1629, 1638, 1651, 1661, 1666, 1669, 1676, 1694,
1697, 1709, 1712, 1724, 1729, 1738, 1742, 1753, 1761, 1765, 1767, 1772, 1775
, 1778, 1781, 1787, 1792, 1797, 1798, 1801, 1802, 1808, 1811, 1819, 1826,
1834, 1841, 1856, 1859, 1861, 1864, 1865, 1875, 1879, 1899, 1901, 1904, 1909
, 1915, 1918, 1919, 1924, 1935, 1939, 1947, 1948, 1950, 1953, 1969, 1972,
1974, 1997], Node 3.3.3.3 Id:3 in zone 1 partitionList:[3, 10, 15, 16, 27,
34, 41, 46, 51, 58, 59, 62, 65, 70, 83, 85, 93, 102, 113, 131, 135, 136, 143
, 144, 147, 150, 156, 157, 166, 168, 169, 175, 183, 201, 206, 209, 214, 217,
221, 223, 226, 229, 234, 243, 252, 260, 266, 270, 278, 290, 294, 308, 312,
320, 332, 344, 349, 354, 355, 366, 367, 374, 376, 378, 380, 388, 394, 395,
399, 402, 406, 408, 410, 413, 416, 421, 422, 424, 430, 448, 459, 461, 474,
478, 480, 486, 489, 494, 497, 500, 506, 509, 510, 520, 528, 530, 534, 559,
562, 569, 570, 587, 588, 590, 591, 603, 605, 616, 626, 630, 655, 657, 658,
664, 667, 680, 682, 686, 688, 690, 694, 695, 699, 710, 722, 728, 734, 736,
742, 744, 756, 762, 766, 776, 784, 795, 810, 818, 820, 834, 840, 846, 847,
857, 864, 870, 873, 893, 896, 911, 913, 916, 918, 930, 937, 941, 952, 954,
957, 962, 963, 966, 970, 977, 982, 986, 989, 994, 1003, 1008, 1011, 1014,
1028, 1029, 1038, 1039, 1041, 1049, 1055, 1058, 1063, 1070, 1072, 1075, 1081
, 1084, 1101, 1104, 1110, 1114, 1134, 1138, 1142, 1146, 1147, 1151, 1154,
1157, 1162, 1186, 1200, 1202, 1210, 1215, 1218, 1221, 1225, 1230, 1231, 1236
, 1239, 1242, 1244, 1250, 1254, 1255, 1263, 1287, 1293, 1301, 1316, 1318,
1324, 1328, 1340, 1347, 1350, 1351, 1365, 1366, 1368, 1379, 1385, 1391, 1402
, 1407, 1411, 1418, 1421, 1422, 1426, 1433, 1439, 1443, 1445, 1450, 1462,
1466, 1467, 1482, 1498, 1504, 1507, 1512, 1517, 1519, 1537, 1544, 1549, 1551
, 1559, 1562, 1564, 1567, 1571, 1575, 1578, 1585, 1590, 1592, 1594, 1600,
1604, 1610, 1612, 1618, 1630, 1631, 1645, 1646, 1652, 1658, 1660, 1682, 1683
, 1686, 1698, 1703, 1714, 1719, 1722, 1725, 1727, 1733, 1734, 1739, 1754,
1758, 1760, 1762, 1766, 1770, 1773, 1774, 1779, 1794, 1803, 1815, 1817, 1821
, 1827, 1828, 1830, 1833, 1842, 1844, 1847, 1855, 1858, 1870, 1878, 1885,
1890, 1898, 1902, 1913, 1923, 1934, 1937, 1946, 1956, 1959, 1966, 1976],
Node 2.2.2.2 Id:2 in zone 0 partitionList:[0, 4, 5, 11, 13, 18, 20, 26, 28,
42, 44, 47, 50, 52, 54, 57, 63, 68, 75, 79, 86, 92, 103, 111, 148, 163, 172,
173, 180, 182, 184, 186, 188, 192, 205, 211, 212, 213, 222, 228, 230, 239,
253, 259, 261, 268, 271, 273, 282, 289, 293, 296, 307, 309, 314, 322, 323,
326, 346, 352, 360, 364, 372, 377, 379, 382, 397, 398, 403, 414, 415, 423,
429, 432, 443, 445, 446, 452, 458, 463, 465, 472, 473, 477, 481, 488, 491,
498, 504, 511, 514, 519, 524, 525, 535, 546, 552, 555, 564, 567, 573, 575,
577, 580, 600, 615, 620, 621, 624, 629, 631, 632, 636, 642, 654, 656, 659,
661, 663, 672, 675, 679, 683, 685, 691, 692, 693, 697, 698, 703, 707, 712,
727, 730, 733, 738, 743, 747, 748, 751, 757, 760, 763, 771, 786, 800, 808,
809, 813, 814, 821, 826, 829, 837, 852, 856, 861, 863, 875, 878, 900, 902,
912, 923, 935, 938, 942, 961, 964, 967, 972, 978, 979, 993, 998, 1007, 1019,
1021, 1023, 1030, 1034, 1042, 1050, 1056, 1065, 1073, 1076, 1080, 1092, 1095
, 1100, 1102, 1106, 1113, 1116, 1118, 1126, 1137, 1140, 1144, 1148, 1149,
1170, 1173, 1191, 1199, 1204, 1207, 1223, 1232, 1235, 1238, 1251, 1257, 1262
, 1264, 1268, 1273, 1274, 1277, 1286, 1294, 1298, 1302, 1306, 1307, 1325,
1327, 1329, 1332, 1333, 1338, 1345, 1348, 1352, 1354, 1361, 1369, 1370, 1374
, 1387, 1400, 1403, 1414, 1423, 1425, 1432, 1437, 1438, 1447, 1449, 1452,
1458, 1464, 1484, 1492, 1506, 1515, 1518, 1546, 1553, 1558, 1569, 1572, 1574
, 1586, 1593, 1596, 1599, 1602, 1609, 1615, 1620, 1629, 1638, 1651, 1661,
1666, 1669, 1676, 1694, 1697, 1709, 1712, 1724, 1729, 1738, 1742, 1753, 1761
, 1765, 1767, 1772, 1775, 1778, 1781, 1787, 1792, 1797, 1798, 1801, 1802,
1808, 1811, 1819, 1826, 1834, 1841, 1856, 1859, 1861, 1864, 1865, 1875, 1879
, 1899, 1901, 1904, 1909, 1915, 1918, 1919, 1924, 1935, 1939, 1947, 1948,
1950, 1953, 1969, 1972, 1974, 1997], Node 3.3.3.3 Id:3 in zone 1
partitionList:[3, 10, 15, 16, 27, 34, 41, 46, 51, 58, 59, 62, 65, 70, 83, 85
, 93, 102, 113, 131, 135, 136, 143, 144, 147, 150, 156, 157, 166, 168, 169,
175, 183, 201, 206, 209, 214, 217, 221, 223, 226, 229, 234, 243, 252, 260,
266, 270, 278, 290, 294, 308, 312, 320, 332, 344, 349, 354, 355, 366, 367,
374, 376, 378, 380, 388, 394, 395, 399, 402, 406, 408, 410, 413, 416, 421,
422, 424, 430, 448, 459, 461, 474, 478, 480, 486, 489, 494, 497, 500, 506,
509, 510, 520, 528, 530, 534, 559, 562, 569, 570, 587, 588, 590, 591, 603,
605, 616, 626, 630, 655, 657, 658, 664, 667, 680, 682, 686, 688, 690, 694,
695, 699, 710, 722, 728, 734, 736, 742, 744, 756, 762, 766, 776, 784, 795,
810, 818, 820, 834, 840, 846, 847, 857, 864, 870, 873, 893, 896, 911, 913,
916, 918, 930, 937, 941, 952, 954, 957, 962, 963, 966, 970, 977, 982, 986,
989, 994, 1003, 1008, 1011, 1014, 1028, 1029, 1038, 1039, 1041, 1049, 1055,
1058, 1063, 1070, 1072, 1075, 1081, 1084, 1101, 1104, 1110, 1114, 1134, 1138
, 1142, 1146, 1147, 1151, 1154, 1157, 1162, 1186, 1200, 1202, 1210, 1215,
1218, 1221, 1225, 1230, 1231, 1236, 1239, 1242, 1244, 1250, 1254, 1255, 1263
, 1287, 1293, 1301, 1316, 1318, 1324, 1328, 1340, 1347, 1350, 1351, 1365,
1366, 1368, 1379, 1385, 1391, 1402, 1407, 1411, 1418, 1421, 1422, 1426, 1433
, 1439, 1443, 1445, 1450, 1462, 1466, 1467, 1482, 1498, 1504, 1507, 1512,
1517, 1519, 1537, 1544, 1549, 1551, 1559, 1562, 1564, 1567, 1571, 1575, 1578
, 1585, 1590, 1592, 1594, 1600, 1604, 1610, 1612, 1618, 1630, 1631, 1645,
1646, 1652, 1658, 1660, 1682, 1683, 1686, 1698, 1703, 1714, 1719, 1722, 1725
, 1727, 1733, 1734, 1739, 1754, 1758, 1760, 1762, 1766, 1770, 1773, 1774,
1779, 1794, 1803, 1815, 1817, 1821, 1827, 1828, 1830, 1833, 1842, 1844, 1847
, 1855, 1858, 1870, 1878, 1885, 1890, 1898, 1902, 1913, 1923, 1934, 1937,
1946, 1956, 1959, 1966, 1976], Node 0.0.0.0 Id:0 in zone 0 partitionList:[1,
7, 17, 23, 32, 35, 60, 71, 72, 77, 80, 84, 87, 91, 95, 101, 109, 114, 117,
120, 128, 132, 142, 146, 151, 155, 159, 177, 189, 193, 197, 200, 203, 208,
216, 227, 244, 247, 255, 256, 265, 276, 279, 285, 288, 292, 299, 304, 311,
316, 321, 339, 342, 353, 356, 359, 369, 371, 373, 383, 386, 389, 400, 405,
407, 420, 425, 436, 437, 449, 453, 468, 471, 482, 485, 499, 502, 508, 522,
527, 531, 532, 533, 549, 550, 554, 563, 568, 571, 581, 583, 584, 589, 602,
607, 618, 623, 635, 640, 643, 665, 668, 673, 687, 704, 705, 708, 718, 721,
723, 724, 731, 737, 745, 752, 753, 767, 769, 775, 777, 781, 783, 789, 793,
797, 806, 817, 825, 832, 835, 838, 845, 848, 853, 867, 872, 885, 888, 891,
901, 906, 919, 927, 931, 936, 955, 960, 968, 973, 975, 981, 996, 1001, 1005,
1010, 1016, 1032, 1035, 1044, 1051, 1057, 1061, 1064, 1068, 1074, 1077, 1087
, 1088, 1090, 1099, 1103, 1108, 1111, 1112, 1120, 1123, 1133, 1153, 1159,

1163, 1164, 1167, 1169, 1172, 1176, 1184, 1185, 1196, 1197, 1203, 1208, 1212

, 1213, 1217, 1220, 1228, 1229, 1241, 1245, 1248, 1253, 1256, 1265, 1279,
1282, 1291, 1300, 1311, 1319, 1320, 1321, 1323, 1336, 1341, 1353, 1357, 1359
, 1364, 1367, 1380, 1381, 1383, 1389, 1397, 1401, 1406, 1408, 1412, 1419,
1420, 1424, 1431, 1440, 1441, 1444, 1448, 1456, 1457, 1468, 1473, 1475, 1480
, 1483, 1490, 1499, 1501, 1505, 1508, 1513, 1521, 1525, 1527, 1532, 1535,
1536, 1542, 1548, 1550, 1561, 1563, 1576, 1583, 1588, 1601, 1603, 1611, 1616
, 1617, 1623, 1625, 1639, 1642, 1647, 1656, 1662, 1663, 1672, 1675, 1679,
1681, 1685, 1688, 1699, 1700, 1715, 1716, 1717, 1720, 1721, 1726, 1732, 1748
, 1749, 1751, 1752, 1756, 1759, 1777, 1793, 1804, 1805, 1809, 1816, 1823,
1824, 1829, 1835, 1839, 1843, 1848, 1849, 1850, 1853, 1868, 1871, 1873, 1880
, 1900, 1905, 1920, 1926, 1933, 1940, 1944, 1945, 1960, 1963, 1965, 1971,
1980, 1984, 1987, 1992, 1996], Node 4.4.4.4 Id:4 in zone 1 partitionList:[8,
19, 25, 31, 33, 36, 37, 43, 64, 73, 76, 81, 97, 104, 108, 112, 115, 124, 125
, 129, 133, 137, 140, 141, 149, 152, 165, 171, 176, 179, 181, 185, 191, 195,
196, 199, 204, 219, 225, 232, 236, 241, 248, 267, 272, 274, 277, 281, 291,
300, 301, 306, 315, 317, 324, 325, 327, 331, 335, 347, 350, 387, 391, 393,
409, 412, 419, 427, 431, 433, 438, 447, 455, 456, 460, 464, 467, 475, 476,
479, 487, 495, 501, 503, 507, 513, 515, 521, 523, 526, 537, 541, 544, 553,
557, 560, 561, 593, 596, 599, 604, 611, 617, 622, 625, 628, 645, 648, 651,
660, 669, 674, 681, 684, 689, 701, 706, 713, 715, 719, 732, 740, 749, 765,
772, 787, 801, 816, 819, 822, 824, 830, 833, 836, 839, 841, 843, 851, 855,
859, 880, 884, 886, 887, 889, 892, 899, 903, 904, 909, 915, 920, 921, 924,
925, 929, 932, 933, 944, 948, 949, 953, 980, 984, 985, 988, 991, 999, 1004,
1015, 1027, 1036, 1048, 1067, 1071, 1083, 1089, 1093, 1096, 1097, 1105, 1115
, 1127, 1131, 1132, 1141, 1145, 1152, 1156, 1161, 1165, 1171, 1174, 1177,
1181, 1188, 1192, 1195, 1209, 1216, 1227, 1233, 1247, 1249, 1260, 1267, 1280
, 1284, 1285, 1289, 1296, 1299, 1305, 1309, 1313, 1317, 1331, 1343, 1344,
1349, 1356, 1360, 1363, 1371, 1375, 1376, 1377, 1392, 1395, 1409, 1415, 1428
, 1435, 1455, 1459, 1461, 1463, 1471, 1477, 1487, 1491, 1496, 1500, 1520,
1528, 1529, 1531, 1533, 1538, 1540, 1541, 1545, 1547, 1555, 1577, 1581, 1591
, 1597, 1614, 1621, 1627, 1633, 1635, 1636, 1643, 1649, 1653, 1655, 1659,
1664, 1665, 1668, 1677, 1687, 1692, 1693, 1695, 1696, 1702, 1707, 1711, 1736
, 1740, 1743, 1744, 1763, 1764, 1769, 1784, 1788, 1791, 1795, 1799, 1807,
1812, 1813, 1831, 1837, 1846, 1852, 1867, 1869, 1872, 1877, 1881, 1883, 1884
, 1887, 1891, 1893, 1895, 1897, 1906, 1908, 1911, 1927, 1930, 1936, 1941,
1943, 1952, 1961, 1964, 1967, 1973, 1975, 1981, 1983, 1985, 1988, 1989, 1991
, 1993], Node 1.1.1.1 Id:1 in zone 0 partitionList:[2, 9, 12, 21, 24, 29, 38
, 40, 45, 53, 56, 66, 69, 74, 82, 89, 98, 99, 106, 110, 118, 123, 126, 130,
134, 138, 139, 145, 153, 158, 161, 162, 167, 170, 178, 190, 194, 218, 224,
233, 235, 237, 240, 242, 246, 251, 258, 263, 283, 286, 302, 305, 313, 319,
330, 333, 334, 337, 338, 343, 348, 351, 362, 363, 368, 375, 385, 390, 392,
396, 401, 417, 426, 428, 435, 440, 442, 454, 462, 469, 490, 493, 496, 512,
517, 536, 539, 543, 547, 558, 576, 579, 586, 592, 594, 595, 597, 601, 606,
608, 610, 614, 627, 634, 639, 647, 649, 650, 652, 653, 671, 700, 709, 711,
714, 726, 735, 741, 759, 761, 764, 773, 778, 780, 788, 796, 798, 803, 804,
811, 815, 828, 831, 842, 849, 860, 868, 869, 871, 874, 877, 879, 882, 883,
890, 894, 897, 905, 908, 910, 914, 917, 922, 934, 945, 947, 951, 956, 958,
965, 983, 987, 990, 995, 1000, 1002, 1009, 1013, 1017, 1020, 1024, 1026,
1037, 1040, 1043, 1047, 1053, 1054, 1062, 1069, 1078, 1082, 1086, 1091, 1094
, 1119, 1125, 1128, 1130, 1135, 1155, 1160, 1175, 1180, 1183, 1187, 1190,
1193, 1206, 1214, 1219, 1224, 1226, 1240, 1243, 1259, 1266, 1269, 1271, 1278
, 1281, 1283, 1288, 1292, 1295, 1297, 1303, 1304, 1308, 1314, 1315, 1330,
1335, 1339, 1342, 1346, 1362, 1373, 1378, 1384, 1386, 1390, 1393, 1398, 1404
, 1410, 1417, 1427, 1429, 1434, 1436, 1442, 1451, 1454, 1460, 1469, 1472,
1476, 1481, 1485, 1488, 1494, 1497, 1502, 1511, 1530, 1539, 1543, 1556, 1560
, 1566, 1568, 1573, 1579, 1580, 1582, 1589, 1605, 1607, 1608, 1613, 1619,
1626, 1632, 1634, 1644, 1650, 1654, 1657, 1667, 1670, 1673, 1691, 1701, 1704
, 1706, 1708, 1735, 1737, 1745, 1746, 1755, 1768, 1783, 1785, 1786, 1789,
1800, 1806, 1814, 1818, 1822, 1832, 1840, 1845, 1863, 1882, 1886, 1889, 1892
, 1894, 1896, 1903, 1910, 1912, 1916, 1922, 1925, 1929, 1931, 1932, 1942,
1951, 1954, 1957, 1962, 1968, 1977, 1979, 1982, 1986, 1990, 1994, 1999],
Node 4.4.4.4 Id:4 in zone 1 partitionList:[8, 19, 25, 31, 33, 36, 37, 43, 64
, 73, 76, 81, 97, 104, 108, 112, 115, 124, 125, 129, 133, 137, 140, 141, 149
, 152, 165, 171, 176, 179, 181, 185, 191, 195, 196, 199, 204, 219, 225, 232,
236, 241, 248, 267, 272, 274, 277, 281, 291, 300, 301, 306, 315, 317, 324,
325, 327, 331, 335, 347, 350, 387, 391, 393, 409, 412, 419, 427, 431, 433,
438, 447, 455, 456, 460, 464, 467, 475, 476, 479, 487, 495, 501, 503, 507,
513, 515, 521, 523, 526, 537, 541, 544, 553, 557, 560, 561, 593, 596, 599,
604, 611, 617, 622, 625, 628, 645, 648, 651, 660, 669, 674, 681, 684, 689,
701, 706, 713, 715, 719, 732, 740, 749, 765, 772, 787, 801, 816, 819, 822,
824, 830, 833, 836, 839, 841, 843, 851, 855, 859, 880, 884, 886, 887, 889,
892, 899, 903, 904, 909, 915, 920, 921, 924, 925, 929, 932, 933, 944, 948,
949, 953, 980, 984, 985, 988, 991, 999, 1004, 1015, 1027, 1036, 1048, 1067,
1071, 1083, 1089, 1093, 1096, 1097, 1105, 1115, 1127, 1131, 1132, 1141, 1145
, 1152, 1156, 1161, 1165, 1171, 1174, 1177, 1181, 1188, 1192, 1195, 1209,
1216, 1227, 1233, 1247, 1249, 1260, 1267, 1280, 1284, 1285, 1289, 1296, 1299
, 1305, 1309, 1313, 1317, 1331, 1343, 1344, 1349, 1356, 1360, 1363, 1371,
1375, 1376, 1377, 1392, 1395, 1409, 1415, 1428, 1435, 1455, 1459, 1461, 1463
, 1471, 1477, 1487, 1491, 1496, 1500, 1520, 1528, 1529, 1531, 1533, 1538,
1540, 1541, 1545, 1547, 1555, 1577, 1581, 1591, 1597, 1614, 1621, 1627, 1633
, 1635, 1636, 1643, 1649, 1653, 1655, 1659, 1664, 1665, 1668, 1677, 1687,
1692, 1693, 1695, 1696, 1702, 1707, 1711, 1736, 1740, 1743, 1744, 1763, 1764
, 1769, 1784, 1788, 1791, 1795, 1799, 1807, 1812, 1813, 1831, 1837, 1846,
1852, 1867, 1869, 1872, 1877, 1881, 1883, 1884, 1887, 1891, 1893, 1895, 1897
, 1906, 1908, 1911, 1927, 1930, 1936, 1941, 1943, 1952, 1961, 1964, 1967,
1973, 1975, 1981, 1983, 1985, 1988, 1989, 1991, 1993], Node 1.1.1.1 Id:1 in
zone 0 partitionList:[2, 9, 12, 21, 24, 29, 38, 40, 45, 53, 56, 66, 69, 74,
82, 89, 98, 99, 106, 110, 118, 123, 126, 130, 134, 138, 139, 145, 153, 158,
161, 162, 167, 170, 178, 190, 194, 218, 224, 233, 235, 237, 240, 242, 246,
251, 258, 263, 283, 286, 302, 305, 313, 319, 330, 333, 334, 337, 338, 343,
348, 351, 362, 363, 368, 375, 385, 390, 392, 396, 401, 417, 426, 428, 435,
440, 442, 454, 462, 469, 490, 493, 496, 512, 517, 536, 539, 543, 547, 558,
576, 579, 586, 592, 594, 595, 597, 601, 606, 608, 610, 614, 627, 634, 639,
647, 649, 650, 652, 653, 671, 700, 709, 711, 714, 726, 735, 741, 759, 761,
764, 773, 778, 780, 788, 796, 798, 803, 804, 811, 815, 828, 831, 842, 849,
860, 868, 869, 871, 874, 877, 879, 882, 883, 890, 894, 897, 905, 908, 910,
914, 917, 922, 934, 945, 947, 951, 956, 958, 965, 983, 987, 990, 995, 1000,
1002, 1009, 1013, 1017, 1020, 1024, 1026, 1037, 1040, 1043, 1047, 1053, 1054
, 1062, 1069, 1078, 1082, 1086, 1091, 1094, 1119, 1125, 1128, 1130, 1135,
1155, 1160, 1175, 1180, 1183, 1187, 1190, 1193, 1206, 1214, 1219, 1224, 1226
, 1240, 1243, 1259, 1266, 1269, 1271, 1278, 1281, 1283, 1288, 1292, 1295,
1297, 1303, 1304, 1308, 1314, 1315, 1330, 1335, 1339, 1342, 1346, 1362, 1373
, 1378, 1384, 1386, 1390, 1393, 1398, 1404, 1410, 1417, 1427, 1429, 1434,
1436, 1442, 1451, 1454, 1460, 1469, 1472, 1476, 1481, 1485, 1488, 1494, 1497
, 1502, 1511, 1530, 1539, 1543, 1556, 1560, 1566, 1568, 1573, 1579, 1580,
1582, 1589, 1605, 1607, 1608, 1613, 1619, 1626, 1632, 1634, 1644, 1650, 1654
, 1657, 1667, 1670, 1673, 1691, 1701, 1704, 1706, 1708, 1735, 1737, 1745,
1746, 1755, 1768, 1783, 1785, 1786, 1789, 1800, 1806, 1814, 1818, 1822, 1832
, 1840, 1845, 1863, 1882, 1886, 1889, 1892, 1894, 1896, 1903, 1910, 1912,
1916, 1922, 1925, 1929, 1931, 1932, 1942, 1951, 1954, 1957, 1962, 1968, 1977
, 1979, 1982, 1986, 1990, 1994, 1999]]
I stopped server 0,2,4,1...FAIL
> getall "test1" "test2" "test3" "test4" "test5"
Exception thrown during operation.

voldemort.store.InsufficientOperationalNodesException: Only 0 nodes up in

preference list, but 1 required. Replication set: [0, 3, 2, 4]Nodes down: [2
, 4, 0, 0, 3, 0, 3, 2, 4]
NOTE that it claims 3 is down but I actually stopped 1.

[Arunachalam]

It is hard for me to tell, what is going wrong without looking at the logs. But one thing I found that is unexpected is, when you are running clients from a zone, you need to specify the 

--client-zone-id parameter for the clients. otherwise the client chooses replicas at random,  and the behavior gets very unpredictable.

I also see the bootstrap URL is tcp://localhost:6666, in prod environments this really needs to be a VIP and not sure where the clients are running from. If you want to do failure tests, I suggest giving a bootstrap URL some like <machine1socketUrl>,<machine2SocketUrl>,<machine3SocketUrl> so that it can fall back when one node goes unavailable.

If the client-zone-id parameter does not work, I need to get the logs to see what is going on.

Thanks,

Arun.

On Thu, Apr 30, 2015 at 12:15 PM, redboy1972 <dee.l.e...@gmail.com> wrote:

BUILD

[root temp]# git clone https://github.com/voldemort/voldemort.git nioFixes
Cloning into 'nioFixes'...
remote: Counting objects: 65749, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 65749 (delta 0), reused 0 (delta 0), pack-reused 65746
Receiving objects: 100% (65749/65749), 205.66 MiB | 15.85 MiB/s, done.
Resolving deltas: 100% (43508/43508), done.


[root temp]# cd nioFixes/


[root nioFixes]# ./gradlew jar
java source target compatibility version 1.5
:compileJava
warning: [options] bootstrap class path not set in conjunction with -source 1.5

/home/ec2-user/temp/nioFixes/src/java/voldemort/store/readonly/io/ByteBufferCloser.java:21: warning: Cleaner is internal proprietary API and may be removed in a future release
        sun.misc.Cleaner cl = ((sun.nio.ch.DirectBuffer) buff).cleaner();
                ^
/home/ec2-user/temp/nioFixes/src/java/voldemort/store/readonly/io/ByteBufferCloser.java:21: warning: DirectBuffer is internal proprietary API and may be removed in a future release
        sun.misc.Cleaner cl = ((sun.nio.ch.DirectBuffer) buff).cleaner();

INSTALL

catalina-ant-6.0.43.jar           hadoop-auth-2.0.5-alpha.jar    junit-3.8.1.jar              servlet-api-2.5-6.1.14.jar

CUSTOMIZE

    <partitions>1, 7, 17, 23, 32, 35, 60, 71, 72, 77, 80, 84, 87, 91, 95, 101, 109, 114, 117, 120, 128, 132, 142, 146, 151, 155, 159, 177, 189, 193, 197, 200, 203, 208, 216, 227, 244, 247, 255, 256, 265, 276, 279, 285, 288, 292, 299, 304, 311, 316, 321, 339, 342, 353, 356, 359, 369, 371, 373, 383, 386, 389, 400, 405, 407, 420, 425, 436, 437, 449, 453, 468, 471, 482, 485, 499, 502, 508, 522, 527, 531, 532, 533, 549, 550, 554, 563, 568, 571, 581, 583, 584, 589, 602, 607, 618, 623, 635, 640, 643, 665, 668, 673, 687, 704, 705, 708, 718, 721, 723, 724, 731, 737, 745, 752, 753, 767, 769, 775, 777, 781, 783, 789, 793, 797, 806, 817, 825, 832, 835, 838, 845, 848, 853, 867, 872, 885, 888, 891, 901, 906, 919, 927, 931, 936, 955, 960, 968, 973, 975, 981, 996, 1001, 1005, 1010, 1016, 1032, 1035, 1044, 1051, 1057, 1061, 1064, 1068, 1074, 1077, 1087, 1088, 1090, 1099, 1103, 1108, 1111, 1112, 1120, 1123, 1133, 1153, 1159, 1163, 1164, 1167, 1169, 1172, 1176, 1184, 1185, 1196, 1197, 1203, 1208, 1212, 1213, 1217, 1220, 1228, 1229, 1241, 1245, 1248, 1253, 1256, 1265, 1279, 1282, 1291, 1300, 1311, 1319, 1320, 1321, 1323, 1336, 1341, 1353, 1357, 1359, 1364, 1367, 1380, 1381, 1383, 1389, 1397, 1401, 1406, 1408, 1412, 1419, 1420, 1424, 1431, 1440, 1441, 1444, 1448, 1456, 1457, 1468, 1473, 1475, 1480, 1483, 1490, 1499, 1501, 1505, 1508, 1513, 1521, 1525, 1527, 1532, 1535, 1536, 1542, 1548, 1550, 1561, 1563, 1576, 1583, 1588, 1601, 1603, 1611, 1616, 1617, 1623, 1625, 1639, 1642, 1647, 1656, 1662, 1663, 1672, 1675, 1679, 1681, 1685, 1688, 1699, 1700, 1715, 1716, 1717, 1720, 1721, 1726, 1732, 1748, 1749, 1751, 1752, 1756, 1759, 1777, 1793, 1804, 1805, 1809, 1816, 1823, 1824, 1829, 1835, 1839, 1843, 1848, 1849, 1850, 1853, 1868, 1871, 1873, 1880, 1900, 1905, 1920, 1926, 1933, 1940, 1944, 1945, 1960, 1963, 1965, 1971, 1980, 1984, 1987, 1992, 1996</partitions>
    <zone-id>0</zone-id>

  </server>
  <server>
    <id>2</id>
    <host>2.2.2.2</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>0, 4, 5, 11, 13, 18, 20, 26, 28, 42, 44, 47, 50, 52, 54, 57, 63, 68, 75, 79, 86, 92, 103, 111, 148, 163, 172, 173, 180, 182, 184, 186, 188, 192, 205, 211, 212, 213, 222, 228, 230, 239, 253, 259, 261, 268, 271, 273, 282, 289, 293, 296, 307, 309, 314, 322, 323, 326, 346, 352, 360, 364, 372, 377, 379, 382, 397, 398, 403, 414, 415, 423, 429, 432, 443, 445, 446, 452, 458, 463, 465, 472, 473, 477, 481, 488, 491, 498, 504, 511, 514, 519, 524, 525, 535, 546, 552, 555, 564, 567, 573, 575, 577, 580, 600, 615, 620, 621, 624, 629, 631, 632, 636, 642, 654, 656, 659, 661, 663, 672, 675, 679, 683, 685, 691, 692, 693, 697, 698, 703, 707, 712, 727, 730, 733, 738, 743, 747, 748, 751, 757, 760, 763, 771, 786, 800, 808, 809, 813, 814, 821, 826, 829, 837, 852, 856, 861, 863, 875, 878, 900, 902, 912, 923, 935, 938, 942, 961, 964, 967, 972, 978, 979, 993, 998, 1007, 1019, 1021, 1023, 1030, 1034, 1042, 1050, 1056, 1065, 1073, 1076, 1080, 1092, 1095, 1100, 1102, 1106, 1113, 1116, 1118, 1126, 1137, 1140, 1144, 1148, 1149, 1170, 1173, 1191, 1199, 1204, 1207, 1223, 1232, 1235, 1238, 1251, 1257, 1262, 1264, 1268, 1273, 1274, 1277, 1286, 1294, 1298, 1302, 1306, 1307, 1325, 1327, 1329, 1332, 1333, 1338, 1345, 1348, 1352, 1354, 1361, 1369, 1370, 1374, 1387, 1400, 1403, 1414, 1423, 1425, 1432, 1437, 1438, 1447, 1449, 1452, 1458, 1464, 1484, 1492, 1506, 1515, 1518, 1546, 1553, 1558, 1569, 1572, 1574, 1586, 1593, 1596, 1599, 1602, 1609, 1615, 1620, 1629, 1638, 1651, 1661, 1666, 1669, 1676, 1694, 1697, 1709, 1712, 1724, 1729, 1738, 1742, 1753, 1761, 1765, 1767, 1772, 1775, 1778, 1781, 1787, 1792, 1797, 1798, 1801, 1802, 1808, 1811, 1819, 1826, 1834, 1841, 1856, 1859, 1861, 1864, 1865, 1875, 1879, 1899, 1901, 1904, 1909, 1915, 1918, 1919, 1924, 1935, 1939, 1947, 1948, 1950, 1953, 1969, 1972, 1974, 1997</partitions>
    <zone-id>0</zone-id>

  </server>  
  <server>
    <id>4</id>
    <host>4.4.4.4</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>8, 19, 25, 31, 33, 36, 37, 43, 64, 73, 76, 81, 97, 104, 108, 112, 115, 124, 125, 129, 133, 137, 140, 141, 149, 152, 165, 171, 176, 179, 181, 185, 191, 195, 196, 199, 204, 219, 225, 232, 236, 241, 248, 267, 272, 274, 277, 281, 291, 300, 301, 306, 315, 317, 324, 325, 327, 331, 335, 347, 350, 387, 391, 393, 409, 412, 419, 427, 431, 433, 438, 447, 455, 456, 460, 464, 467, 475, 476, 479, 487, 495, 501, 503, 507, 513, 515, 521, 523, 526, 537, 541, 544, 553, 557, 560, 561, 593, 596, 599, 604, 611, 617, 622, 625, 628, 645, 648, 651, 660, 669, 674, 681, 684, 689, 701, 706, 713, 715, 719, 732, 740, 749, 765, 772, 787, 801, 816, 819, 822, 824, 830, 833, 836, 839, 841, 843, 851, 855, 859, 880, 884, 886, 887, 889, 892, 899, 903, 904, 909, 915, 920, 921, 924, 925, 929, 932, 933, 944, 948, 949, 953, 980, 984, 985, 988, 991, 999, 1004, 1015, 1027, 1036, 1048, 1067, 1071, 1083, 1089, 1093, 1096, 1097, 1105, 1115, 1127, 1131, 1132, 1141, 1145, 1152, 1156, 1161, 1165, 1171, 1174, 1177, 1181, 1188, 1192, 1195, 1209, 1216, 1227, 1233, 1247, 1249, 1260, 1267, 1280, 1284, 1285, 1289, 1296, 1299, 1305, 1309, 1313, 1317, 1331, 1343, 1344, 1349, 1356, 1360, 1363, 1371, 1375, 1376, 1377, 1392, 1395, 1409, 1415, 1428, 1435, 1455, 1459, 1461, 1463, 1471, 1477, 1487, 1491, 1496, 1500, 1520, 1528, 1529, 1531, 1533, 1538, 1540, 1541, 1545, 1547, 1555, 1577, 1581, 1591, 1597, 1614, 1621, 1627, 1633, 1635, 1636, 1643, 1649, 1653, 1655, 1659, 1664, 1665, 1668, 1677, 1687, 1692, 1693, 1695, 1696, 1702, 1707, 1711, 1736, 1740, 1743, 1744, 1763, 1764, 1769, 1784, 1788, 1791, 1795, 1799, 1807, 1812, 1813, 1831, 1837, 1846, 1852, 1867, 1869, 1872, 1877, 1881, 1883, 1884, 1887, 1891, 1893, 1895, 1897, 1906, 1908, 1911, 1927, 1930, 1936, 1941, 1943, 1952, 1961, 1964, 1967, 1973, 1975, 1981, 1983, 1985, 1988, 1989, 1991, 1993</partitions>
    <zone-id>1</zone-id>

  </server>


 
  <server>
    <id>1</id>
    <host>1.1.1.1</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>2, 9, 12, 21, 24, 29, 38, 40, 45, 53, 56, 66, 69, 74, 82, 89, 98, 99, 106, 110, 118, 123, 126, 130, 134, 138, 139, 145, 153, 158, 161, 162, 167, 170, 178, 190, 194, 218, 224, 233, 235, 237, 240, 242, 246, 251, 258, 263, 283, 286, 302, 305, 313, 319, 330, 333, 334, 337, 338, 343, 348, 351, 362, 363, 368, 375, 385, 390, 392, 396, 401, 417, 426, 428, 435, 440, 442, 454, 462, 469, 490, 493, 496, 512, 517, 536, 539, 543, 547, 558, 576, 579, 586, 592, 594, 595, 597, 601, 606, 608, 610, 614, 627, 634, 639, 647, 649, 650, 652, 653, 671, 700, 709, 711, 714, 726, 735, 741, 759, 761, 764, 773, 778, 780, 788, 796, 798, 803, 804, 811, 815, 828, 831, 842, 849, 860, 868, 869, 871, 874, 877, 879, 882, 883, 890, 894, 897, 905, 908, 910, 914, 917, 922, 934, 945, 947, 951, 956, 958, 965, 983, 987, 990, 995, 1000, 1002, 1009, 1013, 1017, 1020, 1024, 1026, 1037, 1040, 1043, 1047, 1053, 1054, 1062, 1069, 1078, 1082, 1086, 1091, 1094, 1119, 1125, 1128, 1130, 1135, 1155, 1160, 1175, 1180, 1183, 1187, 1190, 1193, 1206, 1214, 1219, 1224, 1226, 1240, 1243, 1259, 1266, 1269, 1271, 1278, 1281, 1283, 1288, 1292, 1295, 1297, 1303, 1304, 1308, 1314, 1315, 1330, 1335, 1339, 1342, 1346, 1362, 1373, 1378, 1384, 1386, 1390, 1393, 1398, 1404, 1410, 1417, 1427, 1429, 1434, 1436, 1442, 1451, 1454, 1460, 1469, 1472, 1476, 1481, 1485, 1488, 1494, 1497, 1502, 1511, 1530, 1539, 1543, 1556, 1560, 1566, 1568, 1573, 1579, 1580, 1582, 1589, 1605, 1607, 1608, 1613, 1619, 1626, 1632, 1634, 1644, 1650, 1654, 1657, 1667, 1670, 1673, 1691, 1701, 1704, 1706, 1708, 1735, 1737, 1745, 1746, 1755, 1768, 1783, 1785, 1786, 1789, 1800, 1806, 1814, 1818, 1822, 1832, 1840, 1845, 1863, 1882, 1886, 1889, 1892, 1894, 1896, 1903, 1910, 1912, 1916, 1922, 1925, 1929, 1931, 1932, 1942, 1951, 1954, 1957, 1962, 1968, 1977, 1979, 1982, 1986, 1990, 1994, 1999</partitions>
    <zone-id>0</zone-id>

  </server>
  <server>
    <id>3</id>
    <host>3.3.3.3</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>3, 10, 15, 16, 27, 34, 41, 46, 51, 58, 59, 62, 65, 70, 83, 85, 93, 102, 113, 131, 135, 136, 143, 144, 147, 150, 156, 157, 166, 168, 169, 175, 183, 201, 206, 209, 214, 217, 221, 223, 226, 229, 234, 243, 252, 260, 266, 270, 278, 290, 294, 308, 312, 320, 332, 344, 349, 354, 355, 366, 367, 374, 376, 378, 380, 388, 394, 395, 399, 402, 406, 408, 410, 413, 416, 421, 422, 424, 430, 448, 459, 461, 474, 478, 480, 486, 489, 494, 497, 500, 506, 509, 510, 520, 528, 530, 534, 559, 562, 569, 570, 587, 588, 590, 591, 603, 605, 616, 626, 630, 655, 657, 658, 664, 667, 680, 682, 686, 688, 690, 694, 695, 699, 710, 722, 728, 734, 736, 742, 744, 756, 762, 766, 776, 784, 795, 810, 818, 820, 834, 840, 846, 847, 857, 864, 870, 873, 893, 896, 911, 913, 916, 918, 930, 937, 941, 952, 954, 957, 962, 963, 966, 970, 977, 982, 986, 989, 994, 1003, 1008, 1011, 1014, 1028, 1029, 1038, 1039, 1041, 1049, 1055, 1058, 1063, 1070, 1072, 1075, 1081, 1084, 1101, 1104, 1110, 1114, 1134, 1138, 1142, 1146, 1147, 1151, 1154, 1157, 1162, 1186, 1200, 1202, 1210, 1215, 1218, 1221, 1225, 1230, 1231, 1236, 1239, 1242, 1244, 1250, 1254, 1255, 1263, 1287, 1293, 1301, 1316, 1318, 1324, 1328, 1340, 1347, 1350, 1351, 1365, 1366, 1368, 1379, 1385, 1391, 1402, 1407, 1411, 1418, 1421, 1422, 1426, 1433, 1439, 1443, 1445, 1450, 1462, 1466, 1467, 1482, 1498, 1504, 1507, 1512, 1517, 1519, 1537, 1544, 1549, 1551, 1559, 1562, 1564, 1567, 1571, 1575, 1578, 1585, 1590, 1592, 1594, 1600, 1604, 1610, 1612, 1618, 1630, 1631, 1645, 1646, 1652, 1658, 1660, 1682, 1683, 1686, 1698, 1703, 1714, 1719, 1722, 1725, 1727, 1733, 1734, 1739, 1754, 1758, 1760, 1762, 1766, 1770, 1773, 1774, 1779, 1794, 1803, 1815, 1817, 1821, 1827, 1828, 1830, 1833, 1842, 1844, 1847, 1855, 1858, 1870, 1878, 1885, 1890, 1898, 1902, 1913, 1923, 1934, 1937, 1946, 1956, 1959, 1966, 1976</partitions>
    <zone-id>1</zone-id>

  </server>


  <server>
    <id>5</id>
    <host>5.5.5.5</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>6, 14, 22, 30, 39, 48, 49, 55, 61, 67, 78, 88, 90, 94, 96, 100, 105, 107, 116, 119, 121, 122, 127, 154, 160, 164, 174, 187, 198, 202, 207, 210, 215, 220, 231, 238, 245, 249, 250, 254, 257, 262, 264, 269, 275, 280, 284, 287, 295, 297, 298, 303, 310, 318, 328, 329, 336, 340, 341, 345, 357, 358, 361, 365, 370, 381, 384, 404, 411, 418, 434, 439, 441, 444, 450, 451, 457, 466, 470, 483, 484, 492, 505, 516, 518, 529, 538, 540, 542, 545, 548, 551, 556, 565, 566, 572, 574, 578, 582, 585, 598, 609, 612, 613, 619, 633, 637, 638, 641, 644, 646, 662, 666, 670, 676, 677, 678, 696, 702, 716, 717, 720, 725, 729, 739, 746, 750, 754, 755, 758, 768, 770, 774, 779, 782, 785, 790, 791, 792, 794, 799, 802, 805, 807, 812, 823, 827, 844, 850, 854, 858, 862, 865, 866, 876, 881, 895, 898, 907, 926, 928, 939, 940, 943, 946, 950, 959, 969, 971, 974, 976, 992, 997, 1006, 1012, 1018, 1022, 1025, 1031, 1033, 1045, 1046, 1052, 1059, 1060, 1066, 1079, 1085, 1098, 1107, 1109, 1117, 1121, 1122, 1124, 1129, 1136, 1139, 1143, 1150, 1158, 1166, 1168, 1178, 1179, 1182, 1189, 1194, 1198, 1201, 1205, 1211, 1222, 1234, 1237, 1246, 1252, 1258, 1261, 1270, 1272, 1275, 1276, 1290, 1310, 1312, 1322, 1326, 1334, 1337, 1355, 1358, 1372, 1382, 1388, 1394, 1396, 1399, 1405, 1413, 1416, 1430, 1446, 1453, 1465, 1470, 1474, 1478, 1479, 1486, 1489, 1493, 1495, 1503, 1509, 1510, 1514, 1516, 1522, 1523, 1524, 1526, 1534, 1552, 1554, 1557, 1565, 1570, 1584, 1587, 1595, 1598, 1606, 1622, 1624, 1628, 1637, 1640, 1641, 1648, 1671, 1674, 1678, 1680, 1684, 1689, 1690, 1705, 1710, 1713, 1718, 1723, 1728, 1730, 1731, 1741, 1747, 1750, 1757, 1771, 1776, 1780, 1782, 1790, 1796, 1810, 1820, 1825, 1836, 1838, 1851, 1854, 1857, 1860, 1862, 1866, 1874, 1876, 1888, 1907, 1914, 1917, 1921, 1928, 1938, 1949, 1955, 1958, 1970, 1978, 1995, 1998, 2000</partitions>
    <zone-id>1</zone-id>

TEST

voldemort.store.InsufficientOperationalNodesException: 1 get alls required, but 0 succeeded. Failing nodes : [Node 2.2.2.2 Id:2 in zone 0 partitionList:[0, 4, 5, 11, 13, 18, 20, 26, 28, 42, 44, 47, 50, 52, 54, 57, 63, 68, 75, 79, 86, 92, 103, 111, 148, 163, 172, 173, 180, 182, 184, 186, 188, 192, 205, 211, 212, 213, 222, 228, 230, 239, 253, 259, 261, 268, 271, 273, 282, 289, 293, 296, 307, 309, 314, 322, 323, 326, 346, 352, 360, 364, 372, 377, 379, 382, 397, 398, 403, 414, 415, 423, 429, 432, 443, 445, 446, 452, 458, 463, 465, 472, 473, 477, 481, 488, 491, 498, 504, 511, 514, 519, 524, 525, 535, 546, 552, 555, 564, 567, 573, 575, 577, 580, 600, 615, 620, 621, 624, 629, 631, 632, 636, 642, 654, 656, 659, 661, 663, 672, 675, 679, 683, 685, 691, 692, 693, 697, 698, 703, 707, 712, 727, 730, 733, 738, 743, 747, 748, 751, 757, 760, 763, 771, 786, 800, 808, 809, 813, 814, 821, 826, 829, 837, 852, 856, 861, 863, 875, 878, 900, 902, 912, 923, 935, 938, 942, 961, 964, 967, 972, 978, 979, 993, 998, 1007, 1019, 1021, 1023, 1030, 1034, 1042, 1050, 1056, 1065, 1073, 1076, 1080, 1092, 1095, 1100, 1102, 1106, 1113, 1116, 1118, 1126, 1137, 1140, 1144, 1148, 1149, 1170, 1173, 1191, 1199, 1204, 1207, 1223, 1232, 1235, 1238, 1251, 1257, 1262, 1264, 1268, 1273, 1274, 1277, 1286, 1294, 1298, 1302, 1306, 1307, 1325, 1327, 1329, 1332, 1333, 1338, 1345, 1348, 1352, 1354, 1361, 1369, 1370, 1374, 1387, 1400, 1403, 1414, 1423, 1425, 1432, 1437, 1438, 1447, 1449, 1452, 1458, 1464, 1484, 1492, 1506, 1515, 1518, 1546, 1553, 1558, 1569, 1572, 1574, 1586, 1593, 1596, 1599, 1602, 1609, 1615, 1620, 1629, 1638, 1651, 1661, 1666, 1669, 1676, 1694, 1697, 1709, 1712, 1724, 1729, 1738, 1742, 1753, 1761, 1765, 1767, 1772, 1775, 1778, 1781, 1787, 1792, 1797, 1798, 1801, 1802, 1808, 1811, 1819, 1826, 1834, 1841, 1856, 1859, 1861, 1864, 1865, 1875, 1879, 1899, 1901, 1904, 1909, 1915, 1918, 1919, 1924, 1935, 1939, 1947, 1948, 1950, 1953, 1969, 1972, 1974, 1997], Node 3.3.3.3 Id:3 in zone 1 partitionList:[3, 10, 15, 16, 27, 34, 41, 46, 51, 58, 59, 62, 65, 70, 83, 85, 93, 102, 113, 131, 135, 136, 143, 144, 147, 150, 156, 157, 166, 168, 169, 175, 183, 201, 206, 209, 214, 217, 221, 223, 226, 229, 234, 243, 252, 260, 266, 270, 278, 290, 294, 308, 312, 320, 332, 344, 349, 354, 355, 366, 367, 374, 376, 378, 380, 388, 394, 395, 399, 402, 406, 408, 410, 413, 416, 421, 422, 424, 430, 448, 459, 461, 474, 478, 480, 486, 489, 494, 497, 500, 506, 509, 510, 520, 528, 530, 534, 559, 562, 569, 570, 587, 588, 590, 591, 603, 605, 616, 626, 630, 655, 657, 658, 664, 667, 680, 682, 686, 688, 690, 694, 695, 699, 710, 722, 728, 734, 736, 742, 744, 756, 762, 766, 776, 784, 795, 810, 818, 820, 834, 840, 846, 847, 857, 864, 870, 873, 893, 896, 911, 913, 916, 918, 930, 937, 941, 952, 954, 957, 962, 963, 966, 970, 977, 982, 986, 989, 994, 1003, 1008, 1011, 1014, 1028, 1029, 1038, 1039, 1041, 1049, 1055, 1058, 1063, 1070, 1072, 1075, 1081, 1084, 1101, 1104, 1110, 1114, 1134, 1138, 1142, 1146, 1147, 1151, 1154, 1157, 1162, 1186, 1200, 1202, 1210, 1215, 1218, 1221, 1225, 1230, 1231, 1236, 1239, 1242, 1244, 1250, 1254, 1255, 1263, 1287, 1293, 1301, 1316, 1318, 1324, 1328, 1340, 1347, 1350, 1351, 1365, 1366, 1368, 1379, 1385, 1391, 1402, 1407, 1411, 1418, 1421, 1422, 1426, 1433, 1439, 1443, 1445, 1450, 1462, 1466, 1467, 1482, 1498, 1504, 1507, 1512, 1517, 1519, 1537, 1544, 1549, 1551, 1559, 1562, 1564, 1567, 1571, 1575, 1578, 1585, 1590, 1592, 1594, 1600, 1604, 1610, 1612, 1618, 1630, 1631, 1645, 1646, 1652, 1658, 1660, 1682, 1683, 1686, 1698, 1703, 1714, 1719, 1722, 1725, 1727, 1733, 1734, 1739, 1754, 1758, 1760, 1762, 1766, 1770, 1773, 1774, 1779, 1794, 1803, 1815, 1817, 1821, 1827, 1828, 1830, 1833, 1842, 1844, 1847, 1855, 1858, 1870, 1878, 1885, 1890, 1898, 1902, 1913, 1923, 1934, 1937, 1946, 1956, 1959, 1966, 1976], Node 3.3.3.3 Id:3 in zone 1 partitionList:[3, 10, 15, 16, 27, 34, 41, 46, 51, 58, 59, 62, 65, 70, 83, 85, 93, 102, 113, 131, 135, 136, 143, 144, 147, 150, 156, 157, 166, 168, 169, 175, 183, 201, 206, 209, 214, 217, 221, 223, 226, 229, 234, 243, 252, 260, 266, 270, 278, 290, 294, 308, 312, 320, 332, 344, 349, 354, 355, 366, 367, 374, 376, 378, 380, 388, 394, 395, 399, 402, 406, 408, 410, 413, 416, 421, 422, 424, 430, 448, 459, 461, 474, 478, 480, 486, 489, 494, 497, 500, 506, 509, 510, 520, 528, 530, 534, 559, 562, 569, 570, 587, 588, 590, 591, 603, 605, 616, 626, 630, 655, 657, 658, 664, 667, 680, 682, 686, 688, 690, 694, 695, 699, 710, 722, 728, 734, 736, 742, 744, 756, 762, 766, 776, 784, 795, 810, 818, 820, 834, 840, 846, 847, 857, 864, 870, 873, 893, 896, 911, 913, 916, 918, 930, 937, 941, 952, 954, 957, 962, 963, 966, 970, 977, 982, 986, 989, 994, 1003, 1008, 1011, 1014, 1028, 1029, 1038, 1039, 1041, 1049, 1055, 1058, 1063, 1070, 1072, 1075, 1081, 1084, 1101, 1104, 1110, 1114, 1134, 1138, 1142, 1146, 1147, 1151, 1154, 1157, 1162, 1186, 1200, 1202, 1210, 1215, 1218, 1221, 1225, 1230, 1231, 1236, 1239, 1242, 1244, 1250, 1254, 1255, 1263, 1287, 1293, 1301, 1316, 1318, 1324, 1328, 1340, 1347, 1350, 1351, 1365, 1366, 1368, 1379, 1385, 1391, 1402, 1407, 1411, 1418, 1421, 1422, 1426, 1433, 1439, 1443, 1445, 1450, 1462, 1466, 1467, 1482, 1498, 1504, 1507, 1512, 1517, 1519, 1537, 1544, 1549, 1551, 1559, 1562, 1564, 1567, 1571, 1575, 1578, 1585, 1590, 1592, 1594, 1600, 1604, 1610, 1612, 1618, 1630, 1631, 1645, 1646, 1652, 1658, 1660, 1682, 1683, 1686, 1698, 1703, 1714, 1719, 1722, 1725, 1727, 1733, 1734, 1739, 1754, 1758, 1760, 1762, 1766, 1770, 1773, 1774, 1779, 1794, 1803, 1815, 1817, 1821, 1827, 1828, 1830, 1833, 1842, 1844, 1847, 1855, 1858, 1870, 1878, 1885, 1890, 1898, 1902, 1913, 1923, 1934, 1937, 1946, 1956, 1959, 1966, 1976], Node 2.2.2.2 Id:2 in zone 0 partitionList:[0, 4, 5, 11, 13, 18, 20, 26, 28, 42, 44, 47, 50, 52, 54, 57, 63, 68, 75, 79, 86, 92, 103, 111, 148, 163, 172, 173, 180, 182, 184, 186, 188, 192, 205, 211, 212, 213, 222, 228, 230, 239, 253, 259, 261, 268, 271, 273, 282, 289, 293, 296, 307, 309, 314, 322, 323, 326, 346, 352, 360, 364, 372, 377, 379, 382, 397, 398, 403, 414, 415, 423, 429, 432, 443, 445, 446, 452, 458, 463, 465, 472, 473, 477, 481, 488, 491, 498, 504, 511, 514, 519, 524, 525, 535, 546, 552, 555, 564, 567, 573, 575, 577, 580, 600, 615, 620, 621, 624, 629, 631, 632, 636, 642, 654, 656, 659, 661, 663, 672, 675, 679, 683, 685, 691, 692, 693, 697, 698, 703, 707, 712, 727, 730, 733, 738, 743, 747, 748, 751, 757, 760, 763, 771, 786, 800, 808, 809, 813, 814, 821, 826, 829, 837, 852, 856, 861, 863, 875, 878, 900, 902, 912, 923, 935, 938, 942, 961, 964, 967, 972, 978, 979, 993, 998, 1007, 1019, 1021, 1023, 1030, 1034, 1042, 1050, 1056, 1065, 1073, 1076, 1080, 1092, 1095, 1100, 1102, 1106, 1113, 1116, 1118, 1126, 1137, 1140, 1144, 1148, 1149, 1170, 1173, 1191, 1199, 1204, 1207, 1223, 1232, 1235, 1238, 1251, 1257, 1262, 1264, 1268, 1273, 1274, 1277, 1286, 1294, 1298, 1302, 1306, 1307, 1325, 1327, 1329, 1332, 1333, 1338, 1345, 1348, 1352, 1354, 1361, 1369, 1370, 1374, 1387, 1400, 1403, 1414, 1423, 1425, 1432, 1437, 1438, 1447, 1449, 1452, 1458, 1464, 1484, 1492, 1506, 1515, 1518, 1546, 1553, 1558, 1569, 1572, 1574, 1586, 1593, 1596, 1599, 1602, 1609, 1615, 1620, 1629, 1638, 1651, 1661, 1666, 1669, 1676, 1694, 1697, 1709, 1712, 1724, 1729, 1738, 1742, 1753, 1761, 1765, 1767, 1772, 1775, 1778, 1781, 1787, 1792, 1797, 1798, 1801, 1802, 1808, 1811, 1819, 1826, 1834, 1841, 1856, 1859, 1861, 1864, 1865, 1875, 1879, 1899, 1901, 1904, 1909, 1915, 1918, 1919, 1924, 1935, 1939, 1947, 1948, 1950, 1953, 1969, 1972, 1974, 1997], Node 3.3.3.3 Id:3 in zone 1 partitionList:[3, 10, 15, 16, 27, 34, 41, 46, 51, 58, 59, 62, 65, 70, 83, 85, 93, 102, 113, 131, 135, 136, 143, 144, 147, 150, 156, 157, 166, 168, 169, 175, 183, 201, 206, 209, 214, 217, 221, 223, 226, 229, 234, 243, 252, 260, 266, 270, 278, 290, 294, 308, 312, 320, 332, 344, 349, 354, 355, 366, 367, 374, 376, 378, 380, 388, 394, 395, 399, 402, 406, 408, 410, 413, 416, 421, 422, 424, 430, 448, 459, 461, 474, 478, 480, 486, 489, 494, 497, 500, 506, 509, 510, 520, 528, 530, 534, 559, 562, 569, 570, 587, 588, 590, 591, 603, 605, 616, 626, 630, 655, 657, 658, 664, 667, 680, 682, 686, 688, 690, 694, 695, 699, 710, 722, 728, 734, 736, 742, 744, 756, 762, 766, 776, 784, 795, 810, 818, 820, 834, 840, 846, 847, 857, 864, 870, 873, 893, 896, 911, 913, 916, 918, 930, 937, 941, 952, 954, 957, 962, 963, 966, 970, 977, 982, 986, 989, 994, 1003, 1008, 1011, 1014, 1028, 1029, 1038, 1039, 1041, 1049, 1055, 1058, 1063, 1070, 1072, 1075, 1081, 1084, 1101, 1104, 1110, 1114, 1134, 1138, 1142, 1146, 1147, 1151, 1154, 1157, 1162, 1186, 1200, 1202, 1210, 1215, 1218, 1221, 1225, 1230, 1231, 1236, 1239, 1242, 1244, 1250, 1254, 1255, 1263, 1287, 1293, 1301, 1316, 1318, 1324, 1328, 1340, 1347, 1350, 1351, 1365, 1366, 1368, 1379, 1385, 1391, 1402, 1407, 1411, 1418, 1421, 1422, 1426, 1433, 1439, 1443, 1445, 1450, 1462, 1466, 1467, 1482, 1498, 1504, 1507, 1512, 1517, 1519, 1537, 1544, 1549, 1551, 1559, 1562, 1564, 1567, 1571, 1575, 1578, 1585, 1590, 1592, 1594, 1600, 1604, 1610, 1612, 1618, 1630, 1631, 1645, 1646, 1652, 1658, 1660, 1682, 1683, 1686, 1698, 1703, 1714, 1719, 1722, 1725, 1727, 1733, 1734, 1739, 1754, 1758, 1760, 1762, 1766, 1770, 1773, 1774, 1779, 1794, 1803, 1815, 1817, 1821, 1827, 1828, 1830, 1833, 1842, 1844, 1847, 1855, 1858, 1870, 1878, 1885, 1890, 1898, 1902, 1913, 1923, 1934, 1937, 1946, 1956, 1959, 1966, 1976], Node 2.2.2.2 Id:2 in zone 0 partitionList:[0, 4, 5, 11, 13, 18, 20, 26, 28, 42, 44, 47, 50, 52, 54, 57, 63, 68, 75, 79, 86, 92, 103, 111, 148, 163, 172, 173, 180, 182, 184, 186, 188, 192, 205, 211, 212, 213, 222, 228, 230, 239, 253, 259, 261, 268, 271, 273, 282, 289, 293, 296, 307, 309, 314, 322, 323, 326, 346, 352, 360, 364, 372, 377, 379, 382, 397, 398, 403, 414, 415, 423, 429, 432, 443, 445, 446, 452, 458, 463, 465, 472, 473, 477, 481, 488, 491, 498, 504, 511, 514, 519, 524, 525, 535, 546, 552, 555, 564, 567, 573, 575, 577, 580, 600, 615, 620, 621, 624, 629, 631, 632, 636, 642, 654, 656, 659, 661, 663, 672, 675, 679, 683, 685, 691, 692, 693, 697, 698, 703, 707, 712, 727, 730, 733, 738, 743, 747, 748, 751, 757, 760, 763, 771, 786, 800, 808, 809, 813, 814, 821, 826, 829, 837, 852, 856, 861, 863, 875, 878, 900, 902, 912, 923, 935, 938, 942, 961, 964, 967, 972, 978, 979, 993, 998, 1007, 1019, 1021, 1023, 1030, 1034, 1042, 1050, 1056, 1065, 1073, 1076, 1080, 1092, 1095, 1100, 1102, 1106, 1113, 1116, 1118, 1126, 1137, 1140, 1144, 1148, 1149, 1170, 1173, 1191, 1199, 1204, 1207, 1223, 1232, 1235, 1238, 1251, 1257, 1262, 1264, 1268, 1273, 1274, 1277, 1286, 1294, 1298, 1302, 1306, 1307, 1325, 1327, 1329, 1332, 1333, 1338, 1345, 1348, 1352, 1354, 1361, 1369, 1370, 1374, 1387, 1400, 1403, 1414, 1423, 1425, 1432, 1437, 1438, 1447, 1449, 1452, 1458, 1464, 1484, 1492, 1506, 1515, 1518, 1546, 1553, 1558, 1569, 1572, 1574, 1586, 1593, 1596, 1599, 1602, 1609, 1615, 1620, 1629, 1638, 1651, 1661, 1666, 1669, 1676, 1694, 1697, 1709, 1712, 1724, 1729, 1738, 1742, 1753, 1761, 1765, 1767, 1772, 1775, 1778, 1781, 1787, 1792, 1797, 1798, 1801, 1802, 1808, 1811, 1819, 1826, 1834, 1841, 1856, 1859, 1861, 1864, 1865, 1875, 1879, 1899, 1901, 1904, 1909, 1915, 1918, 1919, 1924, 1935, 1939, 1947, 1948, 1950, 1953, 1969, 1972, 1974, 1997], Node 3.3.3.3 Id:3 in zone 1 partitionList:[3, 10, 15, 16, 27, 34, 41, 46, 51, 58, 59, 62, 65, 70, 83, 85, 93, 102, 113, 131, 135, 136, 143, 144, 147, 150, 156, 157, 166, 168, 169, 175, 183, 201, 206, 209, 214, 217, 221, 223, 226, 229, 234, 243, 252, 260, 266, 270, 278, 290, 294, 308, 312, 320, 332, 344, 349, 354, 355, 366, 367, 374, 376, 378, 380, 388, 394, 395, 399, 402, 406, 408, 410, 413, 416, 421, 422, 424, 430, 448, 459, 461, 474, 478, 480, 486, 489, 494, 497, 500, 506, 509, 510, 520, 528, 530, 534, 559, 562, 569, 570, 587, 588, 590, 591, 603, 605, 616, 626, 630, 655, 657, 658, 664, 667, 680, 682, 686, 688, 690, 694, 695, 699, 710, 722, 728, 734, 736, 742, 744, 756, 762, 766, 776, 784, 795, 810, 818, 820, 834, 840, 846, 847, 857, 864, 870, 873, 893, 896, 911, 913, 916, 918, 930, 937, 941, 952, 954, 957, 962, 963, 966, 970, 977, 982, 986, 989, 994, 1003, 1008, 1011, 1014, 1028, 1029, 1038, 1039, 1041, 1049, 1055, 1058, 1063, 1070, 1072, 1075, 1081, 1084, 1101, 1104, 1110, 1114, 1134, 1138, 1142, 1146, 1147, 1151, 1154, 1157, 1162, 1186, 1200, 1202, 1210, 1215, 1218, 1221, 1225, 1230, 1231, 1236, 1239, 1242, 1244, 1250, 1254, 1255, 1263, 1287, 1293, 1301, 1316, 1318, 1324, 1328, 1340, 1347, 1350, 1351, 1365, 1366, 1368, 1379, 1385, 1391, 1402, 1407, 1411, 1418, 1421, 1422, 1426, 1433, 1439, 1443, 1445, 1450, 1462, 1466, 1467, 1482, 1498, 1504, 1507, 1512, 1517, 1519, 1537, 1544, 1549, 1551, 1559, 1562, 1564, 1567, 1571, 1575, 1578, 1585, 1590, 1592, 1594, 1600, 1604, 1610, 1612, 1618, 1630, 1631, 1645, 1646, 1652, 1658, 1660, 1682, 1683, 1686, 1698, 1703, 1714, 1719, 1722, 1725, 1727, 1733, 1734, 1739, 1754, 1758, 1760, 1762, 1766, 1770, 1773, 1774, 1779, 1794, 1803, 1815, 1817, 1821, 1827, 1828, 1830, 1833, 1842, 1844, 1847, 1855, 1858, 1870, 1878, 1885, 1890, 1898, 1902, 1913, 1923, 1934, 1937, 1946, 1956, 1959, 1966, 1976], Node 0.0.0.0 Id:0 in zone 0 partitionList:[1, 7, 17, 23, 32, 35, 60, 71, 72, 77, 80, 84, 87, 91, 95, 101, 109, 114, 117, 120, 128, 132, 142, 146, 151, 155, 159, 177, 189, 193, 197, 200, 203, 208, 216, 227, 244, 247, 255, 256, 265, 276, 279, 285, 288, 292, 299, 304, 311, 316, 321, 339, 342, 353, 356, 359, 369, 371, 373, 383, 386, 389, 400, 405, 407, 420, 425, 436, 437, 449, 453, 468, 471, 482, 485, 499, 502, 508, 522, 527, 531, 532, 533, 549, 550, 554, 563, 568, 571, 581, 583, 584, 589, 602, 607, 618, 623, 635, 640, 643, 665, 668, 673, 687, 704, 705, 708, 718, 721, 723, 724, 731, 737, 745, 752, 753, 767, 769, 775, 777, 781, 783, 789, 793, 797, 806, 817, 825, 832, 835, 838, 845, 848, 853, 867, 872, 885, 888, 891, 901, 906, 919, 927, 931, 936, 955, 960, 968, 973, 975, 981, 996, 1001, 1005, 1010, 1016, 1032, 1035, 1044, 1051, 1057, 1061, 1064, 1068, 1074, 1077, 1087, 1088, 1090, 1099, 1103, 1108, 1111, 1112, 1120, 1123, 1133, 1153, 1159, 1163, 1164, 1167, 1169, 1172, 1176, 1184, 1185, 1196, 1197, 1203, 1208, 1212, 1213, 1217, 1220, 1228, 1229, 1241, 1245, 1248, 1253, 1256, 1265, 1279, 1282, 1291, 1300, 1311, 1319, 1320, 1321, 1323, 1336, 1341, 1353, 1357, 1359, 1364, 1367, 1380, 1381, 1383, 1389, 1397, 1401, 1406, 1408, 1412, 1419, 1420, 1424, 1431, 1440, 1441, 1444, 1448, 1456, 1457, 1468, 1473, 1475, 1480, 1483, 1490, 1499, 1501, 1505, 1508, 1513, 1521, 1525, 1527, 1532, 1535, 1536, 1542, 1548, 1550, 1561, 1563, 1576, 1583, 1588, 1601, 1603, 1611, 1616, 1617, 1623, 1625, 1639, 1642, 1647, 1656, 1662, 1663, 1672, 1675, 1679, 1681, 1685, 1688, 1699, 1700, 1715, 1716, 1717, 1720, 1721, 1726, 1732, 1748, 1749, 1751, 1752, 1756, 1759, 1777, 1793, 1804, 1805, 1809, 1816, 1823, 1824, 1829, 1835, 1839, 1843, 1848, 1849, 1850, 1853, 1868, 1871, 1873, 1880, 1900, 1905, 1920, 1926, 1933, 1940, 1944, 1945, 1960, 1963, 1965, 1971, 1980, 1984, 1987, 1992, 1996], Node 4.4.4.4 Id:4 in zone 1 partitionList:[8, 19, 25, 31, 33, 36, 37, 43, 64, 73, 76, 81, 97, 104, 108, 112, 115, 124, 125, 129, 133, 137, 140, 141, 149, 152, 165, 171, 176, 179, 181, 185, 191, 195, 196, 199, 204, 219, 225, 232, 236, 241, 248, 267, 272, 274, 277, 281, 291, 300, 301, 306, 315, 317, 324, 325, 327, 331, 335, 347, 350, 387, 391, 393, 409, 412, 419, 427, 431, 433, 438, 447, 455, 456, 460, 464, 467, 475, 476, 479, 487, 495, 501, 503, 507, 513, 515, 521, 523, 526, 537, 541, 544, 553, 557, 560, 561, 593, 596, 599, 604, 611, 617, 622, 625, 628, 645, 648, 651, 660, 669, 674, 681, 684, 689, 701, 706, 713, 715, 719, 732, 740, 749, 765, 772, 787, 801, 816, 819, 822, 824, 830, 833, 836, 839, 841, 843, 851, 855, 859, 880, 884, 886, 887, 889, 892, 899, 903, 904, 909, 915, 920, 921, 924, 925, 929, 932, 933, 944, 948, 949, 953, 980, 984, 985, 988, 991, 999, 1004, 1015, 1027, 1036, 1048, 1067, 1071, 1083, 1089, 1093, 1096, 1097, 1105, 1115, 1127, 1131, 1132, 1141, 1145, 1152, 1156, 1161, 1165, 1171, 1174, 1177, 1181, 1188, 1192, 1195, 1209, 1216, 1227, 1233, 1247, 1249, 1260, 1267, 1280, 1284, 1285, 1289, 1296, 1299, 1305, 1309, 1313, 1317, 1331, 1343, 1344, 1349, 1356, 1360, 1363, 1371, 1375, 1376, 1377, 1392, 1395, 1409, 1415, 1428, 1435, 1455, 1459, 1461, 1463, 1471, 1477, 1487, 1491, 1496, 1500, 1520, 1528, 1529, 1531, 1533, 1538, 1540, 1541, 1545, 1547, 1555, 1577, 1581, 1591, 1597, 1614, 1621, 1627, 1633, 1635, 1636, 1643, 1649, 1653, 1655, 1659, 1664, 1665, 1668, 1677, 1687, 1692, 1693, 1695, 1696, 1702, 1707, 1711, 1736, 1740, 1743, 1744, 1763, 1764, 1769, 1784, 1788, 1791, 1795, 1799, 1807, 1812, 1813, 1831, 1837, 1846, 1852, 1867, 1869, 1872, 1877, 1881, 1883, 1884, 1887, 1891, 1893, 1895, 1897, 1906, 1908, 1911, 1927, 1930, 1936, 1941, 1943, 1952, 1961, 1964, 1967, 1973, 1975, 1981, 1983, 1985, 1988, 1989, 1991, 1993], Node 1.1.1.1 Id:1 in zone 0 partitionList:[2, 9, 12, 21, 24, 29, 38, 40, 45, 53, 56, 66, 69, 74, 82, 89, 98, 99, 106, 110, 118, 123, 126, 130, 134, 138, 139, 145, 153, 158, 161, 162, 167, 170, 178, 190, 194, 218, 224, 233, 235, 237, 240, 242, 246, 251, 258, 263, 283, 286, 302, 305, 313, 319, 330, 333, 334, 337, 338, 343, 348, 351, 362, 363, 368, 375, 385, 390, 392, 396, 401, 417, 426, 428, 435, 440, 442, 454, 462, 469, 490, 493, 496, 512, 517, 536, 539, 543, 547, 558, 576, 579, 586, 592, 594, 595, 597, 601, 606, 608, 610, 614, 627, 634, 639, 647, 649, 650, 652, 653, 671, 700, 709, 711, 714, 726, 735, 741, 759, 761, 764, 773, 778, 780, 788, 796, 798, 803, 804, 811, 815, 828, 831, 842, 849, 860, 868, 869, 871, 874, 877, 879, 882, 883, 890, 894, 897, 905, 908, 910, 914, 917, 922, 934, 945, 947, 951, 956, 958, 965, 983, 987, 990, 995, 1000, 1002, 1009, 1013, 1017, 1020, 1024, 1026, 1037, 1040, 1043, 1047, 1053, 1054, 1062, 1069, 1078, 1082, 1086, 1091, 1094, 1119, 1125, 1128, 1130, 1135, 1155, 1160, 1175, 1180, 1183, 1187, 1190, 1193, 1206, 1214, 1219, 1224, 1226, 1240, 1243, 1259, 1266, 1269, 1271, 1278, 1281, 1283, 1288, 1292, 1295, 1297, 1303, 1304, 1308, 1314, 1315, 1330, 1335, 1339, 1342, 1346, 1362, 1373, 1378, 1384, 1386, 1390, 1393, 1398, 1404, 1410, 1417, 1427, 1429, 1434, 1436, 1442, 1451, 1454, 1460, 1469, 1472, 1476, 1481, 1485, 1488, 1494, 1497, 1502, 1511, 1530, 1539, 1543, 1556, 1560, 1566, 1568, 1573, 1579, 1580, 1582, 1589, 1605, 1607, 1608, 1613, 1619, 1626, 1632, 1634, 1644, 1650, 1654, 1657, 1667, 1670, 1673, 1691, 1701, 1704, 1706, 1708, 1735, 1737, 1745, 1746, 1755, 1768, 1783, 1785, 1786, 1789, 1800, 1806, 1814, 1818, 1822, 1832, 1840, 1845, 1863, 1882, 1886, 1889, 1892, 1894, 1896, 1903, 1910, 1912, 1916, 1922, 1925, 1929, 1931, 1932, 1942, 1951, 1954, 1957, 1962, 1968, 1977, 1979, 1982, 1986, 1990, 1994, 1999], Node 4.4.4.4 Id:4 in zone 1 partitionList:[8, 19, 25, 31, 33, 36, 37, 43, 64, 73, 76, 81, 97, 104, 108, 112, 115, 124, 125, 129, 133, 137, 140, 141, 149, 152, 165, 171, 176, 179, 181, 185, 191, 195, 196, 199, 204, 219, 225, 232, 236, 241, 248, 267, 272, 274, 277, 281, 291, 300, 301, 306, 315, 317, 324, 325, 327, 331, 335, 347, 350, 387, 391, 393, 409, 412, 419, 427, 431, 433, 438, 447, 455, 456, 460, 464, 467, 475, 476, 479, 487, 495, 501, 503, 507, 513, 515, 521, 523, 526, 537, 541, 544, 553, 557, 560, 561, 593, 596, 599, 604, 611, 617, 622, 625, 628, 645, 648, 651, 660, 669, 674, 681, 684, 689, 701, 706, 713, 715, 719, 732, 740, 749, 765, 772, 787, 801, 816, 819, 822, 824, 830, 833, 836, 839, 841, 843, 851, 855, 859, 880, 884, 886, 887, 889, 892, 899, 903, 904, 909, 915, 920, 921, 924, 925, 929, 932, 933, 944, 948, 949, 953, 980, 984, 985, 988, 991, 999, 1004, 1015, 1027, 1036, 1048, 1067, 1071, 1083, 1089, 1093, 1096, 1097, 1105, 1115, 1127, 1131, 1132, 1141, 1145, 1152, 1156, 1161, 1165, 1171, 1174, 1177, 1181, 1188, 1192, 1195, 1209, 1216, 1227, 1233, 1247, 1249, 1260, 1267, 1280, 1284, 1285, 1289, 1296, 1299, 1305, 1309, 1313, 1317, 1331, 1343, 1344, 1349, 1356, 1360, 1363, 1371, 1375, 1376, 1377, 1392, 1395, 1409, 1415, 1428, 1435, 1455, 1459, 1461, 1463, 1471, 1477, 1487, 1491, 1496, 1500, 1520, 1528, 1529, 1531, 1533, 1538, 1540, 1541, 1545, 1547, 1555, 1577, 1581, 1591, 1597, 1614, 1621, 1627, 1633, 1635, 1636, 1643, 1649, 1653, 1655, 1659, 1664, 1665, 1668, 1677, 1687, 1692, 1693, 1695, 1696, 1702, 1707, 1711, 1736, 1740, 1743, 1744, 1763, 1764, 1769, 1784, 1788, 1791, 1795, 1799, 1807, 1812, 1813, 1831, 1837, 1846, 1852, 1867, 1869, 1872, 1877, 1881, 1883, 1884, 1887, 1891, 1893, 1895, 1897, 1906, 1908, 1911, 1927, 1930, 1936, 1941, 1943, 1952, 1961, 1964, 1967, 1973, 1975, 1981, 1983, 1985, 1988, 1989, 1991, 1993], Node 1.1.1.1 Id:1 in zone 0 partitionList:[2, 9, 12, 21, 24, 29, 38, 40, 45, 53, 56, 66, 69, 74, 82, 89, 98, 99, 106, 110, 118, 123, 126, 130, 134, 138, 139, 145, 153, 158, 161, 162, 167, 170, 178, 190, 194, 218, 224, 233, 235, 237, 240, 242, 246, 251, 258, 263, 283, 286, 302, 305, 313, 319, 330, 333, 334, 337, 338, 343, 348, 351, 362, 363, 368, 375, 385, 390, 392, 396, 401, 417, 426, 428, 435, 440, 442, 454, 462, 469, 490, 493, 496, 512, 517, 536, 539, 543, 547, 558, 576, 579, 586, 592, 594, 595, 597, 601, 606, 608, 610, 614, 627, 634, 639, 647, 649, 650, 652, 653, 671, 700, 709, 711, 714, 726, 735, 741, 759, 761, 764, 773, 778, 780, 788, 796, 798, 803, 804, 811, 815, 828, 831, 842, 849, 860, 868, 869, 871, 874, 877, 879, 882, 883, 890, 894, 897, 905, 908, 910, 914, 917, 922, 934, 945, 947, 951, 956, 958, 965, 983, 987, 990, 995, 1000, 1002, 1009, 1013, 1017, 1020, 1024, 1026, 1037, 1040, 1043, 1047, 1053, 1054, 1062, 1069, 1078, 1082, 1086, 1091, 1094, 1119, 1125, 1128, 1130, 1135, 1155, 1160, 1175, 1180, 1183, 1187, 1190, 1193, 1206, 1214, 1219, 1224, 1226, 1240, 1243, 1259, 1266, 1269, 1271, 1278, 1281, 1283, 1288, 1292, 1295, 1297, 1303, 1304, 1308, 1314, 1315, 1330, 1335, 1339, 1342, 1346, 1362, 1373, 1378, 1384, 1386, 1390, 1393, 1398, 1404, 1410, 1417, 1427, 1429, 1434, 1436, 1442, 1451, 1454, 1460, 1469, 1472, 1476, 1481, 1485, 1488, 1494, 1497, 1502, 1511, 1530, 1539, 1543, 1556, 1560, 1566, 1568, 1573, 1579, 1580, 1582, 1589, 1605, 1607, 1608, 1613, 1619, 1626, 1632, 1634, 1644, 1650, 1654, 1657, 1667, 1670, 1673, 1691, 1701, 1704, 1706, 1708, 1735, 1737, 1745, 1746, 1755, 1768, 1783, 1785, 1786, 1789, 1800, 1806, 1814, 1818, 1822, 1832, 1840, 1845, 1863, 1882, 1886, 1889, 1892, 1894, 1896, 1903, 1910, 1912, 1916, 1922, 1925, 1929, 1931, 1932, 1942, 1951, 1954, 1957, 1962, 1968, 1977, 1979, 1982, 1986, 1990, 1994, 1999]]

I stopped server 0,2,4,1...FAIL
> getall "test1" "test2" "test3" "test4" "test5"
Exception thrown during operation.

voldemort.store.InsufficientOperationalNodesException: Only 0 nodes up in preference list, but 1 required. Replication set: [0, 3, 2, 4]Nodes down: [2, 4, 0, 0, 3, 0, 3, 2, 4]

NOTE that it claims 3 is down but I actually stopped 1.

[redboy1972]

How do I do this?

[ec2-user voldemort-1.9.11]$ bin/voldemort-shell.sh test tcp://localhost:6666,1.1.1.1:6666  --client-zone-id=0

log4j:ERROR Could not read configuration file from URL [file:/home/ec2-user/voldemort-1.9.11/src/java/log4j.properties].

java.io.FileNotFoundException: /home/ec2-user/voldemort-1.9.11/src/java/log4j.properties (No such file or directory)

        at java.io.FileInputStream.open(Native Method)

        at java.io.FileInputStream.<init>(FileInputStream.java:146)

        at java.io.FileInputStream.<init>(FileInputStream.java:101)

        at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90)

        at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188)

        at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)

        at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)

        at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)

        at org.apache.log4j.Logger.getLogger(Logger.java:104)

        at voldemort.xml.StoreDefinitionsMapper.<clinit>(StoreDefinitionsMapper.java:103)

        at voldemort.client.AbstractStoreClientFactory.<clinit>(AbstractStoreClientFactory.java:93)

        at voldemort.VoldemortClientShell.<init>(VoldemortClientShell.java:113)

        at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:190)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at jline.ConsoleRunner.main(ConsoleRunner.java:69)

log4j:ERROR Ignoring configuration file [file:/home/ec2-user/voldemort-1.9.11/src/java/log4j.properties].

Could not connect to server: Illegal scheme in bootstrap URL, must specify a host, URL: tcp://localhost:6666,1.1.1.1:6666

[Arunachalam]

It is the list of individual socket urls separated by comma.

so it should be tcp://0.0.0.0:6666,tcp://1.1.1.1:6666,<more urls>

and so on.

Thanks,

Arun.

[redboy1972]

Tried many variations including:

 1008  bin/voldemort-shell.sh test   tcp://1.1.1.1:6666,tcp://2.2.2.2:6666

 1011  bin/voldemort-shell.sh test  (tcp://1.1.1.1:6666,tcp://2.2.2.2:6666)  --client-zone-id=0

 1012  bin/voldemort-shell.sh test \(tcp://1.1.1.1:6666,tcp://2.2.2.2:6666\)  --client-zone-id=0

 1014  bin/voldemort-shell.sh test --client-zone-id=0  tcp://1.1.1.1:6666,tcp://2.2.2.2:6666

 1016  bin/voldemort-shell.sh test --client-zone-id=0 'tcp://1.1.1.1:6666,tcp://2.2.2.2:6666'

 1018  bin/voldemort-shell.sh test --client-zone-id=0 "tcp://1.1.1.1:6666,tcp://2.2.2.2:6666"

 1020  bin/voldemort-shell.sh test --client-zone-id=0  tcp://1.1.1.1:6666,tcp://2.2.2.2:6666

They all return the same error.  See my last post.

[redboy1972]

Also, what debug log level would you like to see?

NOTE: This seems to be a bug.  In prod, why would I ever have a src dir?

more bin/run-class.sh

# add '-Dlog4j.debug ' to debug log4j issues if not specify log4j.properties

if [ -z ${LOG4JPROPERTIES} ]; then

  LOG4JPROPERTIES="-Dlog4j.configuration=file://${base_dir}/src/java/log4j.properties"

fi

[Arunachalam]

I looked at the Client code and it seems like client shell does not support multiple URLs. I fixed a push to my branch,

https://github.com/arunthirupathi/voldemort/commit/355758d1424b1e9a24547d3e2a1795746a3deb22

Can you apply this patch and retry ?

About the Prod startup shell, it is just indicative of what you might want to do. It is in no-way complete and you might need to configure it little more. I need info level logs.

[redboy1972]

Ok, give me a few to test with patch.

[redboy1972]

Ok, with the patch it seems to accept multiple bootstrap URLs on the command line but with 6 servers and any single one active it fails.

[root voldemort-1.9.11]# bin/voldemort-shell.sh test tcp://1.1.1.1:6666,tcp://2.2.2.2:6666,tcp://3.3.3.3:6666,tcp://4.4.4.4:6666,tcp://5.5.5.5:6666,tcp://52.6.167.43:6666  --client-zone-id=0 | grep "Failed to bootstrap from"
2015-04-30 17:29:18 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://1.1.1.1:6666
2015-04-30 17:29:18 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://2.2.2.2:6666
2015-04-30 17:29:18 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://3.3.3.3:6666
2015-04-30 17:29:19 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://4.4.4.4:6666
2015-04-30 17:29:19 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://5.5.5.5:6666
2015-04-30 17:29:19 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://1.1.1.1:6666
2015-04-30 17:29:20 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://2.2.2.2:6666
2015-04-30 17:29:20 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://3.3.3.3:6666
2015-04-30 17:29:20 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://4.4.4.4:6666
2015-04-30 17:29:20 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://5.5.5.5:6666
2015-04-30 17:29:28 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://1.1.1.1:6666
2015-04-30 17:29:33 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://1.1.1.1:6666
Could not connect to server: No available bootstrap servers found!

I am also unsure if the zone id is being honored.

> [root voldemort-1.9.11]# bin/voldemort-shell.sh
USAGE: bin/voldemort-shell.sh store_name bootstrap_url [command_file] [--client-zone-id <zone-id>]


[root voldemort-1.9.11]# bin/voldemort-shell.sh test tcp://localhost:6666  --client-zone-id 0
2015-04-30 17:20:45 INFO  VoldemortService:61 - Starting scheduler-service
2015-04-30 17:20:45 INFO  ZenStoreClient:190 - Bootstrapping metadata for store test
2015-04-30 17:20:45 INFO  AbstractStoreClientFactory:276 - Client zone-id [0] Attempting to get raw store [test]
2015-04-30 17:20:45 INFO  ZenStoreClient:202 - Creating system stores for store test
2015-04-30 17:20:45 INFO  AbstractStoreClientFactory:276 - Client zone-id [0] Attempting to get raw store [voldsys$_client_registry]
2015-04-30 17:20:45 INFO  AbstractStoreClientFactory:276 - Client zone-id [0] Attempting to get raw store [voldsys$_metadata_version_persistence]
2015-04-30 17:20:45 INFO  AbstractStoreClientFactory:276 - Client zone-id [0] Attempting to get raw store [voldsys$_store_quotas]
2015-04-30 17:20:45 INFO  ZenStoreClient:178 - Metadata version check thread started. Frequency = Every 5000 ms
2015-04-30 17:20:49 INFO  ClientRegistryRefresher:51 - Initial version obtained from client registry: version(0:2) ts:1430428849025
2015-04-30 17:20:49 INFO  ZenStoreClient:136 - Client registry refresher thread started, refresh interval: 43200 seconds
2015-04-30 17:20:49 INFO  ZenStoreClient:114 - Voldemort client created: .test@ip-1.1.1.1:/home/ec2-user/voldemort-1.9.11


bootstrapTime=1430428845331
context=
deploymentPath=/home/ec2-user/voldemort-1.9.11
localHostName=ip-1.1.1.1.us-west-1.compute.internal
sequence=0
storeName=test
updateTime=1430428845006
releaseVersion=1.9.11
clusterMetadataVersion=0
max_connections=50
max_total_connections=500
connection_timeout_ms=500
socket_timeout_ms=5000
routing_timeout_ms=5000
client_zone_id=0
failuredetector_implementation=voldemort.cluster.failuredetector.ThresholdFailureDetector
failuredetector_threshold=95
failuredetector_threshold_count_minimum=30
failuredetector_threshold_interval=300000
failuredetector_threshold_async_recovery_interval=10000


2015-04-30 17:20:49 INFO  AbstractStoreClientFactory:276 - Client zone-id [-1] Attempting to get raw store [voldsys$_metadata_version_persistence]
2015-04-30 17:20:49 WARN  PipelineRoutedStore:148 - Client Zone is not specified. Default to Zone 0. The servers could be in a remote zone
2015-04-30 17:20:49 INFO  AbstractStoreClientFactory:276 - Client zone-id [-1] Attempting to get raw store [voldsys$_store_quotas]
2015-04-30 17:20:49 WARN  PipelineRoutedStore:148 - Client Zone is not specified. Default to Zone 0. The servers could be in a remote zone

[Arunachalam]

Instead of running tcp://localhost:6666 as your URL, did you try running with URL ( in your comma separated list one at time), What does it say ?

[redboy1972]

All TCP params are real IPs.

With both servers up...

Individually both work

bin/voldemort-shell.sh test tcp://1.1.1.1  --client-zone-id=0

bin/voldemort-shell.sh test tcp://2.2.2.2:6666  --client-zone-id=0

Together they both work

bin/voldemort-shell.sh test tcp://1.1.1.1,tcp://2.2.2.2:6666  --client-zone-id=0

bin/voldemort-shell.sh test tcp://2.2.2.2,tcp://1.1.1.1:6666  --client-zone-id=0

With ONLY 1.1.1.1 down

FAILS    - bin/voldemort-shell.sh test tcp://1.1.1.1,tcp://2.2.2.2:6666  --client-zone-id=0

WORKS - bin/voldemort-shell.sh test tcp://2.2.2.2,tcp://1.1.1.1:6666  --client-zone-id=0

With ONLY 2.2.2.2 down

WORKS- bin/voldemort-shell.sh test tcp://1.1.1.1,tcp://2.2.2.2:6666  --client-zone-id=0

FAILS   - bin/voldemort-shell.sh test tcp://2.2.2.2,tcp://1.1.1.1:6666  --client-zone-id=0

[redboy1972]

Hold off for a little while.

It seems that using public IPs in the configuration to circumvent the need for setting up VPN in Amazon EC2 may be part of the problem.  I will setup using all private IPs within the same Amazon zone and re-test.

[ro...@ip-1.1.1.1 voldemort-1.9.11]# bin/voldemort-server.sh .

2015-04-30 18:33:52 INFO  MetadataStore:1152 - metadata init().

2015-04-30 18:33:52 INFO  MetadataStore:846 - Updating routing strategy for all stores

2015-04-30 18:33:52 INFO  VoldemortServer:185 - List of all IPs & Hostnames for the current node:[1.1.1.1, ip-1.1.1.1.us-west-1.compute.internal, localhost]

2015-04-30 18:33:52 INFO  VoldemortServer:186 - Configured hostname [111.222.333.444] does not seem to match current node.

Where 1.1.1.1 is private but 111.222.333.444 is public.

[Arunachalam]

OK.

If possible always run the clients from different machine than the Server. You don't need many clients just one or two per zone. 

Never use localhost in your URL (both in bootstrap and cluster hostname) as it messes up things in unimaginable ways.

Thanks,

Arun.

[redboy1972]

Good to know.

This is already being done.  Localhost is NEVER used by clients or servers, neither is 0.0.0.0 (although it is abstracted as such below).  I am cross connecting 6 clients to each server to "double check" results.

I have re-setup my environment to use the patched branch recommended above and am still getting failures.  I think I have tracked it down to the following.

    <replication-factor>4</replication-factor>

    <zone-replication-factor>

        <replication-factor zone-id="0">2</replication-factor>

        <replication-factor zone-id="1">2</replication-factor>

    </zone-replication-factor>

Remember that even servers are in zone 0 and odd are in zone 1.  The client is NOT writing twice in each zone.  It in this case is writing three times in zone 0 and once in zone 1.

> preflist "test21"

Node 4

host:  172.31.0.220

port: 6666

available: no

last checked: 1430442577641 ms ago

Node 0

host:  172.31.0.122

port: 6666

available: no

last checked: 1430442577192 ms ago

Node 5

host:  172.31.8.93

port: 6666

available: yes

last checked: 1430442366547 ms ago

Node 2

host:  172.31.9.103

port: 6666

available: no

last checked: 1430442578342 ms ago

I have re-run the tool "bin/rebalance-new-zoned-cluster.sh -c config/cluster.xml  -s config/stores.xml -o new" and get the same result although sometimes the "key" which acts up swaps around.  This time it was "test21" previous time it was "test11".

<cluster>

  <zone>
    <zone-id>0</zone-id>
    <proximity-list>1</proximity-list>
  </zone>
  <zone>
    <zone-id>1</zone-id>
    <proximity-list>0</proximity-list>
  </zone>
  <server>
    <id>0</id>
    <host>0.0.0.0</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>1, 7, 17, 23, 32, 35, 60, 71, 72, 77, 80, 84, 87, 91, 95, 101, 109, 114, 117, 120, 128, 132, 142, 146, 151, 155, 159, 177, 189, 193, 197, 200, 203, 208, 216, 227, 244, 247, 253, 255, 256, 265, 276, 279, 285, 288, 292, 299, 304, 311, 316, 321, 339, 342, 353, 356, 359, 369, 371, 373, 383, 386, 389, 400, 405, 407, 411, 420, 425, 436, 437, 449, 453, 468, 471, 482, 485, 499, 502, 508, 522, 527, 531, 532, 533, 549, 550, 552, 554, 561, 563, 568, 571, 583, 584, 589, 602, 607, 618, 623, 635, 640, 643, 665, 668, 673, 687, 704, 705, 708, 718, 721, 723, 724, 731, 737, 745, 752, 753, 767, 769, 775, 777, 781, 783, 789, 793, 797, 806, 817, 825, 832, 835, 838, 845, 848, 853, 867, 872, 885, 888, 891, 901, 906, 919, 927, 931, 936, 955, 960, 968, 973, 975, 981, 996, 1001, 1005, 1010, 1016, 1032, 1035, 1044, 1051, 1057, 1061, 1064, 1068, 1074, 1077, 1080, 1087, 1088, 1090, 1099, 1103, 1108, 1111, 1112, 1119, 1120, 1123, 1133, 1153, 1159, 1163, 1164, 1167, 1169, 1172, 1176, 1184, 1185, 1196, 1197, 1203, 1208, 1212, 1213, 1217, 1220, 1228, 1229, 1241, 1245, 1253, 1256, 1265, 1279, 1282, 1291, 1300, 1311, 1319, 1320, 1321, 1323, 1336, 1341, 1353, 1357, 1364, 1367, 1380, 1381, 1383, 1389, 1397, 1401, 1406, 1408, 1412, 1420, 1424, 1431, 1440, 1441, 1444, 1448, 1456, 1457, 1468, 1473, 1475, 1480, 1483, 1490, 1499, 1501, 1505, 1508, 1513, 1521, 1525, 1527, 1532, 1535, 1536, 1542, 1548, 1550, 1563, 1576, 1583, 1588, 1601, 1603, 1611, 1616, 1617, 1623, 1625, 1639, 1642, 1647, 1656, 1662, 1663, 1672, 1675, 1679, 1681, 1685, 1688, 1699, 1700, 1715, 1716, 1717, 1720, 1721, 1726, 1732, 1748, 1749, 1751, 1752, 1756, 1759, 1777, 1793, 1804, 1805, 1809, 1816, 1823, 1824, 1829, 1835, 1839, 1843, 1848, 1849, 1850, 1853, 1868, 1871, 1873, 1900, 1905, 1920, 1926, 1933, 1940, 1944, 1945, 1960, 1963, 1965, 1969, 1971, 1980, 1984, 1987, 1996</partitions>

    <zone-id>0</zone-id>
  </server>
  <server>

    <id>1</id>
    <host>1.1.1.1</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>2, 9, 12, 21, 24, 29, 38, 40, 45, 53, 56, 66, 69, 74, 82, 89, 98, 99, 106, 110, 118, 123, 126, 130, 134, 138, 139, 145, 153, 158, 161, 162, 167, 170, 178, 190, 194, 218, 224, 233, 235, 237, 240, 242, 246, 251, 258, 263, 283, 286, 302, 305, 313, 319, 330, 333, 334, 337, 338, 343, 348, 351, 362, 363, 368, 375, 385, 390, 392, 396, 401, 417, 426, 428, 435, 440, 442, 454, 462, 469, 490, 493, 496, 512, 517, 536, 539, 543, 547, 558, 576, 579, 581, 586, 592, 594, 595, 597, 601, 606, 608, 610, 614, 627, 634, 639, 647, 649, 650, 652, 653, 671, 700, 709, 711, 714, 726, 735, 741, 759, 761, 764, 773, 778, 780, 788, 796, 798, 803, 804, 811, 815, 828, 831, 842, 849, 860, 868, 869, 871, 874, 877, 879, 882, 883, 890, 894, 897, 905, 908, 910, 914, 917, 922, 934, 945, 947, 951, 956, 958, 965, 983, 987, 990, 995, 1000, 1002, 1009, 1013, 1017, 1020, 1024, 1026, 1037, 1040, 1043, 1047, 1053, 1054, 1062, 1069, 1078, 1082, 1086, 1091, 1094, 1125, 1128, 1130, 1135, 1155, 1160, 1175, 1180, 1183, 1187, 1190, 1193, 1206, 1214, 1219, 1224, 1226, 1240, 1243, 1259, 1264, 1269, 1271, 1278, 1281, 1283, 1288, 1292, 1295, 1297, 1303, 1304, 1308, 1314, 1315, 1330, 1333, 1335, 1339, 1342, 1346, 1362, 1373, 1378, 1384, 1386, 1390, 1393, 1404, 1410, 1417, 1427, 1429, 1434, 1436, 1442, 1451, 1454, 1460, 1469, 1472, 1476, 1481, 1485, 1488, 1494, 1497, 1502, 1511, 1530, 1539, 1543, 1556, 1560, 1566, 1568, 1573, 1579, 1580, 1582, 1589, 1605, 1607, 1608, 1613, 1619, 1626, 1632, 1634, 1644, 1650, 1654, 1657, 1667, 1670, 1673, 1691, 1701, 1704, 1706, 1708, 1735, 1737, 1745, 1746, 1755, 1768, 1783, 1785, 1786, 1789, 1800, 1806, 1814, 1818, 1822, 1832, 1840, 1845, 1863, 1882, 1886, 1889, 1892, 1894, 1896, 1903, 1910, 1912, 1916, 1922, 1925, 1929, 1931, 1932, 1942, 1951, 1954, 1957, 1962, 1968, 1977, 1979, 1982, 1986, 1990, 1994, 1999</partitions>

    <zone-id>0</zone-id>
  </server>
  <server>

    <id>2</id>
    <host>2.2.2.2</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>0, 4, 5, 11, 13, 18, 20, 26, 28, 42, 44, 47, 50, 52, 54, 57, 63, 68, 75, 79, 86, 92, 103, 111, 148, 163, 172, 173, 180, 182, 184, 186, 188, 192, 205, 211, 212, 213, 222, 228, 230, 239, 259, 261, 268, 271, 273, 282, 289, 293, 296, 307, 309, 314, 322, 323, 326, 346, 352, 360, 364, 372, 377, 379, 382, 397, 398, 403, 414, 415, 423, 429, 432, 443, 445, 446, 452, 458, 463, 465, 472, 473, 477, 481, 488, 491, 498, 504, 511, 514, 519, 524, 525, 535, 546, 555, 564, 567, 573, 575, 577, 580, 600, 615, 620, 621, 624, 629, 631, 632, 636, 642, 654, 656, 659, 661, 663, 672, 675, 679, 683, 685, 691, 692, 693, 697, 698, 703, 707, 712, 727, 730, 733, 738, 743, 747, 748, 751, 757, 760, 763, 771, 786, 800, 808, 809, 813, 814, 821, 826, 829, 837, 852, 856, 861, 863, 875, 878, 900, 902, 912, 923, 935, 938, 942, 961, 964, 967, 972, 978, 993, 998, 1007, 1019, 1021, 1023, 1030, 1034, 1042, 1050, 1056, 1065, 1073, 1076, 1092, 1095, 1100, 1102, 1106, 1113, 1116, 1118, 1126, 1137, 1140, 1144, 1148, 1149, 1170, 1173, 1191, 1199, 1204, 1207, 1223, 1232, 1235, 1238, 1248, 1251, 1257, 1262, 1266, 1268, 1273, 1274, 1277, 1286, 1294, 1298, 1302, 1306, 1307, 1325, 1327, 1329, 1332, 1338, 1345, 1348, 1352, 1354, 1359, 1361, 1369, 1370, 1374, 1387, 1398, 1400, 1403, 1414, 1419, 1423, 1425, 1432, 1437, 1438, 1447, 1449, 1452, 1458, 1464, 1484, 1492, 1506, 1515, 1518, 1546, 1553, 1558, 1561, 1569, 1572, 1574, 1586, 1593, 1596, 1599, 1602, 1609, 1615, 1620, 1629, 1638, 1651, 1661, 1666, 1669, 1676, 1694, 1697, 1709, 1712, 1724, 1729, 1738, 1742, 1753, 1761, 1765, 1767, 1772, 1775, 1778, 1781, 1787, 1792, 1797, 1798, 1801, 1802, 1808, 1811, 1819, 1826, 1834, 1841, 1856, 1859, 1861, 1864, 1865, 1875, 1879, 1899, 1901, 1904, 1909, 1915, 1918, 1919, 1924, 1935, 1939, 1947, 1948, 1950, 1953, 1972, 1974, 1992, 1997</partitions>

    <zone-id>0</zone-id>
  </server>
  <server>

    <id>3</id>
    <host>3.3.3.3</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>3, 10, 15, 16, 27, 34, 41, 46, 51, 58, 59, 62, 65, 70, 83, 85, 93, 102, 113, 131, 135, 136, 143, 144, 147, 150, 156, 157, 166, 168, 169, 175, 183, 201, 206, 209, 214, 217, 221, 223, 226, 229, 234, 243, 252, 260, 266, 270, 278, 290, 294, 308, 320, 332, 344, 349, 354, 355, 366, 367, 374, 376, 378, 380, 388, 394, 395, 399, 402, 408, 410, 413, 416, 421, 422, 424, 430, 448, 459, 461, 474, 478, 480, 486, 489, 494, 497, 500, 506, 509, 510, 520, 528, 530, 534, 559, 562, 569, 570, 587, 588, 590, 591, 603, 605, 616, 626, 630, 655, 657, 658, 664, 667, 680, 682, 686, 688, 690, 694, 695, 699, 710, 722, 728, 734, 736, 742, 744, 756, 762, 766, 776, 784, 795, 810, 818, 820, 834, 840, 847, 857, 859, 864, 870, 873, 893, 896, 911, 913, 916, 918, 930, 937, 941, 952, 954, 957, 962, 963, 966, 970, 977, 982, 986, 989, 994, 1003, 1008, 1011, 1014, 1028, 1029, 1038, 1039, 1041, 1049, 1055, 1058, 1063, 1070, 1072, 1075, 1081, 1084, 1101, 1104, 1110, 1114, 1134, 1138, 1142, 1146, 1147, 1151, 1154, 1157, 1162, 1186, 1200, 1202, 1210, 1215, 1218, 1221, 1225, 1230, 1231, 1236, 1239, 1242, 1244, 1250, 1254, 1255, 1263, 1287, 1293, 1301, 1316, 1318, 1324, 1328, 1340, 1347, 1350, 1351, 1365, 1366, 1368, 1379, 1385, 1391, 1402, 1407, 1411, 1418, 1421, 1422, 1426, 1433, 1439, 1443, 1445, 1450, 1462, 1466, 1467, 1482, 1498, 1504, 1507, 1512, 1517, 1519, 1537, 1544, 1549, 1551, 1559, 1562, 1564, 1565, 1567, 1571, 1575, 1578, 1585, 1590, 1592, 1594, 1600, 1604, 1610, 1612, 1618, 1630, 1631, 1645, 1646, 1652, 1658, 1660, 1682, 1683, 1686, 1698, 1703, 1714, 1719, 1722, 1725, 1727, 1733, 1734, 1739, 1754, 1758, 1760, 1762, 1766, 1770, 1773, 1774, 1779, 1794, 1803, 1815, 1817, 1821, 1827, 1828, 1830, 1833, 1842, 1844, 1847, 1855, 1858, 1870, 1878, 1880, 1885, 1890, 1898, 1902, 1913, 1923, 1934, 1937, 1946, 1956, 1959, 1966, 1976</partitions>

    <zone-id>1</zone-id>
  </server>
  <server>

    <id>4</id>
    <host>4.4.4.4</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>8, 19, 25, 31, 33, 36, 37, 43, 64, 73, 76, 81, 97, 104, 108, 112, 115, 124, 125, 129, 133, 137, 140, 141, 149, 152, 165, 171, 176, 179, 181, 185, 191, 195, 196, 199, 204, 219, 225, 232, 236, 241, 248, 267, 272, 274, 277, 281, 291, 300, 301, 306, 312, 315, 317, 324, 325, 327, 331, 335, 347, 350, 387, 391, 393, 409, 412, 419, 427, 431, 433, 438, 447, 455, 456, 460, 464, 467, 475, 476, 479, 487, 495, 501, 503, 507, 513, 515, 521, 523, 526, 537, 541, 544, 553, 557, 560, 593, 596, 599, 604, 611, 617, 622, 625, 628, 645, 648, 651, 660, 669, 674, 681, 684, 689, 701, 706, 713, 715, 719, 732, 740, 749, 765, 772, 787, 801, 816, 819, 822, 824, 830, 833, 836, 839, 841, 843, 846, 851, 855, 880, 884, 886, 887, 889, 892, 899, 903, 904, 909, 915, 920, 921, 924, 925, 929, 932, 933, 944, 948, 949, 953, 980, 984, 985, 988, 991, 997, 999, 1004, 1015, 1027, 1036, 1048, 1067, 1071, 1083, 1089, 1093, 1096, 1097, 1105, 1115, 1127, 1131, 1132, 1141, 1145, 1152, 1156, 1161, 1165, 1171, 1174, 1177, 1181, 1188, 1192, 1195, 1209, 1216, 1227, 1233, 1249, 1260, 1267, 1280, 1284, 1285, 1289, 1296, 1299, 1305, 1309, 1313, 1317, 1331, 1343, 1344, 1349, 1356, 1360, 1363, 1371, 1375, 1392, 1395, 1409, 1415, 1428, 1435, 1455, 1459, 1461, 1463, 1471, 1477, 1487, 1491, 1496, 1500, 1520, 1528, 1529, 1531, 1533, 1538, 1540, 1541, 1545, 1547, 1555, 1577, 1581, 1591, 1597, 1614, 1621, 1627, 1633, 1635, 1636, 1643, 1649, 1653, 1655, 1659, 1664, 1665, 1668, 1677, 1687, 1692, 1693, 1695, 1696, 1702, 1707, 1711, 1736, 1740, 1743, 1744, 1763, 1764, 1769, 1784, 1788, 1791, 1795, 1799, 1807, 1812, 1813, 1831, 1837, 1838, 1846, 1852, 1867, 1869, 1872, 1877, 1881, 1883, 1884, 1887, 1891, 1893, 1895, 1897, 1906, 1908, 1911, 1927, 1930, 1936, 1938, 1941, 1943, 1952, 1961, 1964, 1967, 1973, 1975, 1981, 1983, 1985, 1988, 1989, 1991, 1993</partitions>

    <zone-id>1</zone-id>
  </server>
  <server>

    <id>5</id>
    <host>5.5.5.5</host>
    <http-port>8081</http-port>
    <socket-port>6666</socket-port>
    <admin-port>6667</admin-port>

    <partitions>6, 14, 22, 30, 39, 48, 49, 55, 61, 67, 78, 88, 90, 94, 96, 100, 105, 107, 116, 119, 121, 122, 127, 154, 160, 164, 174, 187, 198, 202, 207, 210, 215, 220, 231, 238, 245, 249, 250, 254, 257, 262, 264, 269, 275, 280, 284, 287, 295, 297, 298, 303, 310, 318, 328, 329, 336, 340, 341, 345, 357, 358, 361, 365, 370, 381, 384, 404, 406, 418, 434, 439, 441, 444, 450, 451, 457, 466, 470, 483, 484, 492, 505, 516, 518, 529, 538, 540, 542, 545, 548, 551, 556, 565, 566, 572, 574, 578, 582, 585, 598, 609, 612, 613, 619, 633, 637, 638, 641, 644, 646, 662, 666, 670, 676, 677, 678, 696, 702, 716, 717, 720, 725, 729, 739, 746, 750, 754, 755, 758, 768, 770, 774, 779, 782, 785, 790, 791, 792, 794, 799, 802, 805, 807, 812, 823, 827, 844, 850, 854, 858, 862, 865, 866, 876, 881, 895, 898, 907, 926, 928, 939, 940, 943, 946, 950, 959, 969, 971, 974, 976, 979, 992, 1006, 1012, 1018, 1022, 1025, 1031, 1033, 1045, 1046, 1052, 1059, 1060, 1066, 1079, 1085, 1098, 1107, 1109, 1117, 1121, 1122, 1124, 1129, 1136, 1139, 1143, 1150, 1158, 1166, 1168, 1178, 1179, 1182, 1189, 1194, 1198, 1201, 1205, 1211, 1222, 1234, 1237, 1246, 1247, 1252, 1258, 1261, 1270, 1272, 1275, 1276, 1290, 1310, 1312, 1322, 1326, 1334, 1337, 1355, 1358, 1372, 1376, 1377, 1382, 1388, 1394, 1396, 1399, 1405, 1413, 1416, 1430, 1446, 1453, 1465, 1470, 1474, 1478, 1479, 1486, 1489, 1493, 1495, 1503, 1509, 1510, 1514, 1516, 1522, 1523, 1524, 1526, 1534, 1552, 1554, 1557, 1570, 1584, 1587, 1595, 1598, 1606, 1622, 1624, 1628, 1637, 1640, 1641, 1648, 1671, 1674, 1678, 1680, 1684, 1689, 1690, 1705, 1710, 1713, 1718, 1723, 1728, 1730, 1731, 1741, 1747, 1750, 1757, 1771, 1776, 1780, 1782, 1790, 1796, 1810, 1820, 1825, 1836, 1851, 1854, 1857, 1860, 1862, 1866, 1874, 1876, 1888, 1907, 1914, 1917, 1921, 1928, 1949, 1955, 1958, 1970, 1978, 1995, 1998, 2000</partitions>

    <zone-id>1</zone-id>
  </server>
</cluster>

[Arunachalam]

Can you paste your store definition again ? It is somehow not treating it as zoned store.

Thanks,

Arun.

[redboy1972]

Sure.  Did you mean another file as it is in last post but perhaps trimmed.

Should I manually balance it to see if this is a server or client issue.  Thinking of exactly 6 partitions matching ID of server.

[arunac...@gmail.com]

I just rechecked your cluster.xml it has only 2nodes in zone 0 (nodes 0 and 2 ) and 4 nodes in zone 1 ( nodes 4 ,1,3and 5). Can you please fix that?

Sent from Windows Mail

[redboy1972]

/facepalm!!!  :)

When I get in I will redo.  Hopefully it will work after that.

[redboy1972]

Ok, server HA is now working.  Yeah!

This is due to this patch:

https://github.com/voldemort/voldemort/tree/nioFixes

Now back to client:

With the following patch the client now accepts multiple bootstrap URLs on the command line:

https://github.com/arunthirupathi/voldemort/commit/355758d1424b1e9a24547d3e2a1795746a3deb22

However, it will not connect to the second server in the list if the first one is down.

Where 1.1.1.1 is down and 2.2.2.2 is up.  Client is started with

bin/voldemort-shell.sh test tcp://1.1.1.1:6666,tcp://2.2.2.2:6666  --client-zone-id=0

[ro...@ip-2.2.2.2 voldemort-1.9.11]# clear;bin/voldemort-server.sh .

2015-05-01 12:52:22 INFO  MetadataStore:1152 - metadata init().

2015-05-01 12:52:22 INFO  MetadataStore:846 - Updating routing strategy for all stores

2015-05-01 12:52:22 INFO  VoldemortServer:302 - Using NIO Connector for Admin Service.

2015-05-01 12:52:22 INFO  VoldemortServer:243 - Using NIO Connector.

2015-05-01 12:52:22 INFO  VoldemortService:61 - Starting voldemort-server

2015-05-01 12:52:22 INFO  JNAUtils:52 - mlockall() on JVM Heap successful

2015-05-01 12:52:22 INFO  VoldemortServer:363 - Starting 6 services.

2015-05-01 12:52:22 INFO  VoldemortService:61 - Starting storage-service

2015-05-01 12:52:22 INFO  StorageService:227 - Initializing bdb storage engine.

2015-05-01 12:52:22 INFO  StorageService:227 - Initializing read-only storage engine.

2015-05-01 12:52:22 INFO  StorageService:491 - Opening system store 'voldsys$_client_registry' (memory).

2015-05-01 12:52:22 INFO  StorageService:1005 - Scheduling data retention cleanup job for store 'voldsys$_client_registry' at Sat May 02 00:00:00 EDT 2015 with retention scan throttle rate:2147483647 Entries/second.

2015-05-01 12:52:22 INFO  StorageService:491 - Opening system store 'voldsys$_metadata_version_persistence' (file-backed-cache).

2015-05-01 12:52:22 INFO  StorageService:491 - Opening system store 'voldsys$_store_quotas' (file-backed-cache).

2015-05-01 12:52:22 INFO  StorageService:297 - Initializing the slop store using bdb

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:337 - Creating environment for slop:

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:370 -     BDB cache size = 1073741824

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:371 -     BDB je.cleaner.threads = 1

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:373 -     BDB je.cleaner.minUtilization = 50

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:375 -     BDB je.cleaner.minFileUtilization = 0

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:377 -     BDB je.log.fileMax = 62914560

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:379 -     BDB allowCreate=true,cacheSize=1073741824,txnNoSync=false,txnWriteNoSync=false,{je.tree.maxDelta=100, je.cleaner.bytesInterval=31457280, je.maxMemory=1073741824, je.cleaner.adjustUtilization=false, je.log.faultReadSize=2048, je.env.isTransactional=true, je.cleaner.maxBatchFiles=0, je.cleaner.minUtilization=50, je.env.recoveryForceCheckpoint=false, je.tree.binDelta=75, je.cleaner.fetchObsoleteSize=true, je.cleaner.lookAheadCacheSize=8192, je.lock.nLockTables=7, je.cleaner.threads=1, je.log.iteratorReadSize=8192, je.lock.timeout=500 MILLISECONDS, je.checkpointer.wakeupInterval=30000000, je.checkpointer.highPriority=false, je.checkpointer.bytesInterval=209715200, je.cleaner.lazyMigration=false, je.cleaner.minFileUtilization=0, je.txn.durability=NO_SYNC,NO_SYNC,SIMPLE_MAJORITY, je.sharedCache=true, je.env.fairLatches=false, je.log.fileMax=62914560},

2015-05-01 12:52:23 INFO  StorageService:345 - Initializing slop pusher job type streaming at Fri May 01 12:57:23 EDT 2015

2015-05-01 12:52:23 INFO  StorageService:366 - Initializing Slop Purge job

2015-05-01 12:52:23 INFO  StorageService:378 - Initializing repair job.

2015-05-01 12:52:23 INFO  StorageService:389 - Intializing prune job

2015-05-01 12:52:23 INFO  StorageService:399 - Initializing stores:

2015-05-01 12:52:23 INFO  StorageService:401 - Validating schemas:

2015-05-01 12:52:23 INFO  StoreDefinitionUtils:223 - Validating schema for store:  test

2015-05-01 12:52:23 INFO  StorageService:635 - Opening store 'test' (bdb).

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:337 - Creating environment for test:

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:370 -     BDB cache size = 1073741824

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:371 -     BDB je.cleaner.threads = 1

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:373 -     BDB je.cleaner.minUtilization = 50

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:375 -     BDB je.cleaner.minFileUtilization = 0

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:377 -     BDB je.log.fileMax = 62914560

2015-05-01 12:52:23 INFO  BdbStorageConfiguration:379 -     BDB allowCreate=true,cacheSize=1073741824,txnNoSync=false,txnWriteNoSync=false,{je.tree.maxDelta=100, je.cleaner.bytesInterval=31457280, je.maxMemory=1073741824, je.cleaner.adjustUtilization=false, je.log.faultReadSize=2048, je.env.isTransactional=true, je.cleaner.maxBatchFiles=0, je.cleaner.minUtilization=50, je.env.recoveryForceCheckpoint=false, je.tree.binDelta=75, je.cleaner.fetchObsoleteSize=true, je.cleaner.lookAheadCacheSize=8192, je.lock.nLockTables=7, je.cleaner.threads=1, je.log.iteratorReadSize=8192, je.lock.timeout=500 MILLISECONDS, je.checkpointer.wakeupInterval=30000000, je.checkpointer.highPriority=false, je.checkpointer.bytesInterval=209715200, je.cleaner.lazyMigration=false, je.cleaner.minFileUtilization=0, je.txn.durability=NO_SYNC,NO_SYNC,SIMPLE_MAJORITY, je.sharedCache=true, je.env.fairLatches=false, je.log.fileMax=62914560},

2015-05-01 12:52:23 INFO  StorageService:1005 - Scheduling data retention cleanup job for store 'test' at Sat May 02 00:00:00 EDT 2015 with retention scan throttle rate:30 Entries/second.

2015-05-01 12:52:23 INFO  StorageService:429 - All stores initialized.

2015-05-01 12:52:23 INFO  VoldemortService:61 - Starting scheduler-service

2015-05-01 12:52:23 INFO  VoldemortService:61 - Starting async-scheduler

2015-05-01 12:52:23 INFO  AsyncOperationService:191 - Starting asyncOperationRunner

2015-05-01 12:52:23 INFO  VoldemortService:61 - Starting rebalance-service

2015-05-01 12:52:23 INFO  VoldemortService:61 - Starting socket-service

2015-05-01 12:52:23 INFO  NioSocketService:120 - Starting Voldemort NIO socket server (admin-server) on port 6667

2015-05-01 12:52:23 INFO  NioSocketService:225 - Server now listening for connections on port 6667

2015-05-01 12:52:23 INFO  VoldemortService:61 - Starting jmx-service

2015-05-01 12:52:23 WARN  JmxService:115 - Overwriting mbean voldemort.server.niosocket:type=NioSocketService

2015-05-01 12:52:23 INFO  VoldemortService:61 - Starting http-service

2015-05-01 12:52:23 INFO  log:67 - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog

2015-05-01 12:52:23 INFO  log:67 - jetty-6.1.26

2015-05-01 12:52:23 INFO  log:67 - Started SelectChann...@0.0.0.0:8081

2015-05-01 12:52:23 INFO  HttpService:112 - HTTP service started on port 8081

2015-05-01 12:52:23 INFO  VoldemortService:61 - Starting socket-service

2015-05-01 12:52:23 INFO  NioSocketService:120 - Starting Voldemort NIO socket server (nio-socket-server) on port 6666

2015-05-01 12:52:23 INFO  VoldemortServer:370 - Startup completed in 578 ms.

2015-05-01 12:52:23 INFO  NioSocketService:225 - Server now listening for connections on port 6666

2015-05-01 12:52:38 INFO  AsyncRequestHandler:406 - Protocol negotiated for Socket[addr=/1.1.1.1,port=43493,localport=6666]: voldemort-native-v3

2015-05-01 12:52:38 INFO  AsyncRequestHandler:406 - Protocol negotiated for Socket[addr=/1.1.1.1,port=43497,localport=6666]: voldemort-native-v1

[ro...@ip-3.3.3.3 voldemort-1.9.11]# bin/voldemort-shell.sh test tcp://1.1.1.1:6666,tcp://2.2.2.2:6666  --client-zone-id=0

2015-05-01 12:52:38 INFO  VoldemortService:61 - Starting scheduler-service

2015-05-01 12:52:38 INFO  ZenStoreClient:190 - Bootstrapping metadata for store test

2015-05-01 12:52:38 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://1.1.1.1:6666

voldemort.VoldemortException: voldemort.store.UnreachableStoreException: Failure while checking out socket for 1.1.1.1:6666(vp3):

        at voldemort.client.SocketStoreClientFactory.getRemoteMetadata(SocketStoreClientFactory.java:105)

        at voldemort.client.AbstractStoreClientFactory.bootstrapMetadata(AbstractStoreClientFactory.java:560)

        at voldemort.client.AbstractStoreClientFactory.bootstrapMetadataWithRetries(AbstractStoreClientFactory.java:529)

        at voldemort.client.AbstractStoreClientFactory.bootstrapMetadataWithRetries(AbstractStoreClientFactory.java:554)

        at voldemort.client.ZenStoreClient.bootStrap(ZenStoreClient.java:196)

        at voldemort.client.ZenStoreClient.<init>(ZenStoreClient.java:106)

        at voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:234)

        at voldemort.client.SocketStoreClientFactory.getParentStoreClient(SocketStoreClientFactory.java:91)

        at voldemort.client.SocketStoreClientFactory.getStoreClient(SocketStoreClientFactory.java:86)

        at voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:199)

        at voldemort.VoldemortClientShell.<init>(VoldemortClientShell.java:114)

        at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:191)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at jline.ConsoleRunner.main(ConsoleRunner.java:69)

Caused by: voldemort.store.UnreachableStoreException: Failure while checking out socket for 1.1.1.1:6666(vp3):

        at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:217)

        at voldemort.store.socket.SocketStore.request(SocketStore.java:279)

        at voldemort.store.socket.SocketStore.get(SocketStore.java:201)

        at voldemort.store.socket.SocketStore.get(SocketStore.java:63)

        at voldemort.store.serialized.SerializingStore.get(SerializingStore.java:107)

        at voldemort.client.AbstractStoreClientFactory.getRemoteMetadata(AbstractStoreClientFactory.java:579)

        at voldemort.client.SocketStoreClientFactory.getRemoteMetadata(SocketStoreClientFactory.java:97)

        ... 16 more

Caused by: java.net.ConnectException: Connection refused

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:155)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:51)

        at voldemort.utils.pool.KeyedResourcePool$Pool.attemptGrow(KeyedResourcePool.java:455)

        at voldemort.utils.pool.KeyedResourcePool.attemptNonBlockingCheckout(KeyedResourcePool.java:177)

        at voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:132)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:208)

        ... 22 more

2015-05-01 12:52:38 INFO  AbstractStoreClientFactory:276 - Client zone-id [0] Attempting to get raw store [test]

2015-05-01 12:52:38 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://1.1.1.1:6666

voldemort.VoldemortException: voldemort.store.UnreachableStoreException: Failure while checking out socket for 1.1.1.1:6666(vp3):

        at voldemort.client.SocketStoreClientFactory.getRemoteMetadata(SocketStoreClientFactory.java:105)

        at voldemort.client.AbstractStoreClientFactory.bootstrapMetadata(AbstractStoreClientFactory.java:560)

        at voldemort.client.AbstractStoreClientFactory.bootstrapMetadataWithRetries(AbstractStoreClientFactory.java:529)

        at voldemort.client.AbstractStoreClientFactory.getRawStore(AbstractStoreClientFactory.java:302)

        at voldemort.client.ZenStoreClient.bootStrap(ZenStoreClient.java:199)

        at voldemort.client.ZenStoreClient.<init>(ZenStoreClient.java:106)

        at voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:234)

        at voldemort.client.SocketStoreClientFactory.getParentStoreClient(SocketStoreClientFactory.java:91)

        at voldemort.client.SocketStoreClientFactory.getStoreClient(SocketStoreClientFactory.java:86)

        at voldemort.client.AbstractStoreClientFactory.getStoreClient(AbstractStoreClientFactory.java:199)

        at voldemort.VoldemortClientShell.<init>(VoldemortClientShell.java:114)

        at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:191)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at jline.ConsoleRunner.main(ConsoleRunner.java:69)

Caused by: voldemort.store.UnreachableStoreException: Failure while checking out socket for 1.1.1.1:6666(vp3):

        at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:217)

        at voldemort.store.socket.SocketStore.request(SocketStore.java:279)

        at voldemort.store.socket.SocketStore.get(SocketStore.java:201)

        at voldemort.store.socket.SocketStore.get(SocketStore.java:63)

        at voldemort.store.serialized.SerializingStore.get(SerializingStore.java:107)

        at voldemort.client.AbstractStoreClientFactory.getRemoteMetadata(AbstractStoreClientFactory.java:579)

        at voldemort.client.SocketStoreClientFactory.getRemoteMetadata(SocketStoreClientFactory.java:97)

        ... 16 more

Caused by: java.net.ConnectException: Connection refused

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:155)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:51)

        at voldemort.utils.pool.KeyedResourcePool$Pool.attemptGrow(KeyedResourcePool.java:455)

        at voldemort.utils.pool.KeyedResourcePool.attemptNonBlockingCheckout(KeyedResourcePool.java:177)

        at voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:132)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:208)

        ... 22 more

2015-05-01 12:52:38 INFO  ZenStoreClient:202 - Creating system stores for store test

2015-05-01 12:52:38 INFO  AbstractStoreClientFactory:276 - Client zone-id [0] Attempting to get raw store [voldsys$_client_registry]

2015-05-01 12:52:38 INFO  AbstractStoreClientFactory:276 - Client zone-id [0] Attempting to get raw store [voldsys$_metadata_version_persistence]

2015-05-01 12:52:38 INFO  AbstractStoreClientFactory:276 - Client zone-id [0] Attempting to get raw store [voldsys$_store_quotas]

2015-05-01 12:52:38 INFO  ZenStoreClient:178 - Metadata version check thread started. Frequency = Every 5000 ms

2015-05-01 12:52:38 INFO  ClientRegistryRefresher:51 - Initial version obtained from client registry: version(3:1, 4:3, 5:1) ts:1430499158704

2015-05-01 12:52:38 INFO  ZenStoreClient:136 - Client registry refresher thread started, refresh interval: 43200 seconds

2015-05-01 12:52:38 INFO  ZenStoreClient:114 - Voldemort client created: .test@ip-3.3.3.3.us-west-1.compute.internal:/home/ec2-user/voldemort-1.9.11

bootstrapTime=1430499158547

context=

deploymentPath=/home/ec2-user/voldemort-1.9.11

localHostName=ip-3.3.3.3.us-west-1.compute.internal

sequence=0

storeName=test

updateTime=1430499158111

releaseVersion=1.9.11

clusterMetadataVersion=0

max_connections=50

max_total_connections=500

connection_timeout_ms=500

socket_timeout_ms=5000

routing_timeout_ms=5000

client_zone_id=0

failuredetector_implementation=voldemort.cluster.failuredetector.ThresholdFailureDetector

failuredetector_threshold=95

failuredetector_threshold_count_minimum=30

failuredetector_threshold_interval=300000

failuredetector_threshold_async_recovery_interval=10000

2015-05-01 12:52:38 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://1.1.1.1:6666

voldemort.VoldemortException: voldemort.store.UnreachableStoreException: Failure while checking out socket for 1.1.1.1:6666(vp1):

        at voldemort.client.SocketStoreClientFactory.getRemoteMetadata(SocketStoreClientFactory.java:105)

        at voldemort.client.AbstractStoreClientFactory.bootstrapMetadata(AbstractStoreClientFactory.java:560)

        at voldemort.client.AbstractStoreClientFactory.bootstrapMetadataWithRetries(AbstractStoreClientFactory.java:529)

        at voldemort.client.protocol.admin.AdminClient$HelperOperations.getClusterFromBootstrapURL(AdminClient.java:384)

        at voldemort.client.protocol.admin.AdminClient$HelperOperations.access$100(AdminClient.java:339)

        at voldemort.client.protocol.admin.AdminClient.<init>(AdminClient.java:234)

        at voldemort.VoldemortClientShell.<init>(VoldemortClientShell.java:115)

        at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:191)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at jline.ConsoleRunner.main(ConsoleRunner.java:69)

Caused by: voldemort.store.UnreachableStoreException: Failure while checking out socket for 1.1.1.1:6666(vp1):

        at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:217)

        at voldemort.store.socket.SocketStore.request(SocketStore.java:279)

        at voldemort.store.socket.SocketStore.get(SocketStore.java:201)

        at voldemort.store.socket.SocketStore.get(SocketStore.java:63)

        at voldemort.store.serialized.SerializingStore.get(SerializingStore.java:107)

        at voldemort.client.AbstractStoreClientFactory.getRemoteMetadata(AbstractStoreClientFactory.java:579)

        at voldemort.client.SocketStoreClientFactory.getRemoteMetadata(SocketStoreClientFactory.java:97)

        ... 12 more

Caused by: java.net.ConnectException: Connection refused

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:155)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:51)

        at voldemort.utils.pool.KeyedResourcePool$Pool.attemptGrow(KeyedResourcePool.java:455)

        at voldemort.utils.pool.KeyedResourcePool.attemptNonBlockingCheckout(KeyedResourcePool.java:177)

        at voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:132)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:208)

        ... 18 more

2015-05-01 12:52:38 WARN  AbstractStoreClientFactory:533 - Failed to bootstrap store

2015-05-01 12:52:38 WARN  AbstractStoreClientFactory:536 - Will try to bootstrap will try again after 5 seconds.

2015-05-01 12:52:43 WARN  AbstractStoreClientFactory:564 - Failed to bootstrap from tcp://1.1.1.1:6666

voldemort.VoldemortException: voldemort.store.UnreachableStoreException: Failure while checking out socket for 1.1.1.1:6666(vp1):

        at voldemort.client.SocketStoreClientFactory.getRemoteMetadata(SocketStoreClientFactory.java:105)

        at voldemort.client.AbstractStoreClientFactory.bootstrapMetadata(AbstractStoreClientFactory.java:560)

        at voldemort.client.AbstractStoreClientFactory.bootstrapMetadataWithRetries(AbstractStoreClientFactory.java:529)

        at voldemort.client.protocol.admin.AdminClient$HelperOperations.getClusterFromBootstrapURL(AdminClient.java:384)

        at voldemort.client.protocol.admin.AdminClient$HelperOperations.access$100(AdminClient.java:339)

        at voldemort.client.protocol.admin.AdminClient.<init>(AdminClient.java:234)

        at voldemort.VoldemortClientShell.<init>(VoldemortClientShell.java:115)

        at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:191)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at jline.ConsoleRunner.main(ConsoleRunner.java:69)

Caused by: voldemort.store.UnreachableStoreException: Failure while checking out socket for 1.1.1.1:6666(vp1):

        at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:217)

        at voldemort.store.socket.SocketStore.request(SocketStore.java:279)

        at voldemort.store.socket.SocketStore.get(SocketStore.java:201)

        at voldemort.store.socket.SocketStore.get(SocketStore.java:63)

        at voldemort.store.serialized.SerializingStore.get(SerializingStore.java:107)

        at voldemort.client.AbstractStoreClientFactory.getRemoteMetadata(AbstractStoreClientFactory.java:579)

        at voldemort.client.SocketStoreClientFactory.getRemoteMetadata(SocketStoreClientFactory.java:97)

        ... 12 more

Caused by: java.net.ConnectException: Connection refused

        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:155)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorFactory.create(ClientRequestExecutorFactory.java:51)

        at voldemort.utils.pool.KeyedResourcePool$Pool.attemptGrow(KeyedResourcePool.java:455)

        at voldemort.utils.pool.KeyedResourcePool.attemptNonBlockingCheckout(KeyedResourcePool.java:177)

        at voldemort.utils.pool.KeyedResourcePool.checkout(KeyedResourcePool.java:132)

        at voldemort.store.socket.clientrequest.ClientRequestExecutorPool.checkout(ClientRequestExecutorPool.java:208)

        ... 18 more

2015-05-01 12:52:43 WARN  AbstractStoreClientFactory:533 - Failed to bootstrap store

2015-05-01 12:52:43 INFO  VoldemortService:67 - Stopping scheduler-service

[Arunachalam]

We have never tested the multiple bootstrap URLs thoroughly, in theory it should work. But I have never seen it in use.

What I would suggest is, having a Virtual IP (which can seamlessly route to existing nodes if some of them are down). This is what we use in Production now. The virtual IP Is used only for bootstrapping and after that the client talks to the nodes directly, so the performance impact is non-existent.

Thanks,

Arun.

[redboy1972]

Bug is here:

Code creates AdminClient with first server passed as a bootstrap url instead of the whole array.  When attempting to get the cluster.xml this fails due to assumption about availability of the first node in list.  I attempted to convert AdminClient to use the array form but the changes started to cascade and I became less certain about what I was doing.

VoldemortClientShell.java ~ln 100

    public VoldemortClientShell(ClientConfig clientConfig,

                                String storeName,

                                BufferedReader commandReader,

                                PrintStream commandOutput,

                                PrintStream errorStream) {

        this.commandReader = commandReader;

        this.commandOutput = commandOutput;

        this.errorStream = errorStream;

        String bootstrapUrl = clientConfig.getBootstrapUrls()[0];

        try {

            factory = new SocketStoreClientFactory(clientConfig);

            client = factory.getStoreClient(storeName);

            adminClient = new AdminClient(bootstrapUrl, new AdminClientConfig(), new ClientConfig());

2015-05-01 16:23:03 INFO  AbstractStoreClientFactory:560 - bootstrapMetadata called for key: cluster.xml with Bootstrap urls:
2015-05-01 16:23:03 INFO  AbstractStoreClientFactory:562 -      url [tcp://172.31.4.209:6666]
java.lang.Exception: Stack trace
        at java.lang.Thread.dumpStack(Thread.java:1365)
        at voldemort.client.AbstractStoreClientFactory.bootstrapMetadata(AbstractStoreClientFactory.java:565)

        at voldemort.client.AbstractStoreClientFactory.bootstrapMetadataWithRetries(AbstractStoreClientFactory.java:529)
        at voldemort.client.protocol.admin.AdminClient$HelperOperations.getClusterFromBootstrapURL(AdminClient.java:384)
        at voldemort.client.protocol.admin.AdminClient$HelperOperations.access$100(AdminClient.java:339)
        at voldemort.client.protocol.admin.AdminClient.<init>(AdminClient.java:234)
        at voldemort.VoldemortClientShell.<init>(VoldemortClientShell.java:115)
        at voldemort.VoldemortClientShell.main(VoldemortClientShell.java:191)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at jline.ConsoleRunner.main(ConsoleRunner.java:69)

2015-05-01 16:23:03 INFO  AbstractStoreClientFactory:570 - Calling getRemoteMetadata using url [tcp://172.31.4.209:6666]
2015-05-01 16:23:03 DEBUG SocketStore:199 - GET keyRef: 2115796134 requestRef: 3913465
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory:131 - Creating socket 2 for 172.31.4.209:6666 using protocol vp1
2015-05-01 16:23:03 WARN  AbstractStoreClientFactory:576 - Failed to bootstrap from tcp://172.31.4.209:6666  Exception: voldemort.store.UnreachableStoreException: Failure while checking out socket for 172.31.4.209:6666(vp1):
2015-05-01 16:23:03 WARN  AbstractStoreClientFactory:533 - Failed to bootstrap store
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory$ClientRequestSelectorManager:198 - SelectorManager is closed, exiting
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory$ClientRequestSelectorManager:198 - SelectorManager is closed, exiting
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory$ClientRequestSelectorManager:198 - SelectorManager is closed, exiting
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory$ClientRequestSelectorManager:198 - SelectorManager is closed, exiting
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory$ClientRequestSelectorManager:198 - SelectorManager is closed, exiting
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory$ClientRequestSelectorManager:198 - SelectorManager is closed, exiting
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory$ClientRequestSelectorManager:198 - SelectorManager is closed, exiting
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory$ClientRequestSelectorManager:198 - SelectorManager is closed, exiting
2015-05-01 16:23:03 DEBUG ClientRequestExecutor:140 - Closing remote connection from Socket[unconnected]
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory:118 - Destroyed socket 1 connection to 172.31.0.122:6666
2015-05-01 16:23:03 INFO  VoldemortService:67 - Stopping scheduler-service
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory$ClientRequestSelectorManager:198 - SelectorManager is closed, exiting
2015-05-01 16:23:03 DEBUG ClientRequestExecutor:140 - Closing remote connection from Socket[unconnected]
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory:118 - Destroyed socket 1 connection to 172.31.0.122:6666
2015-05-01 16:23:03 DEBUG ClientRequestExecutor:140 - Closing remote connection from Socket[unconnected]
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory:118 - Destroyed socket 2 connection to 172.31.9.103:6666
2015-05-01 16:23:03 DEBUG ClientRequestExecutor:140 - Closing remote connection from Socket[unconnected]
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory:118 - Destroyed socket 3 connection to 172.31.9.103:6666
2015-05-01 16:23:03 DEBUG ClientRequestExecutor:140 - Closing remote connection from Socket[unconnected]
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory:118 - Destroyed socket 4 connection to 172.31.0.220:6666
2015-05-01 16:23:03 DEBUG ClientRequestExecutor:140 - Closing remote connection from Socket[unconnected]
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory:118 - Destroyed socket 5 connection to 172.31.8.93:6666
2015-05-01 16:23:03 DEBUG ClientRequestExecutor:140 - Closing remote connection from Socket[unconnected]
2015-05-01 16:23:03 DEBUG ClientRequestExecutorFactory:118 - Destroyed socket 6 connection to 172.31.14.80:6666

[Arunachalam]

I am not surprised, we never exercised the code and it is easier to get vip.

[redboy1972]

I wanted to say thank you to all who helped out along the way.  I probably would have figured some of this myself but it sure helped having quick responses from knowledgeable people.  Many thanks.

I consider this topic closed and if I have further questions will start another thread.

Thanks...

[Brendan Harris (a.k.a. stotch on irc.oftc.net)]

Coincidentally we were talking about the multiple bootstrap URL option last week (a topic that has only come up twice in my 4.5 years working with the product) and I was theorizing that a comma-separated list should work.

Anyway, glad everything is working for you now.

Brendan

[redboy1972]

Yup, thanks for the help.

I only debugged long enough to figure out cause of failure.  Multiple URLs passed via array seems to be partially implemented. More not then present but partially.  I noticed part of the purpose of bootstrapping was to get store.xml and cluster.xml.   Since these rarely change, would it help to seed clients with the same info as the servers from a file instead or trying to download it?  It would be another way to get started.

I did try to see if I could do a patch based on: https://github.com/voldemort/voldemort/tree/nioFixes but the changes cascaded too much.

[Brendan Harris]

Because those files are mastered by the servers, I'd not recommend attempting to provide them outside of the bootstrap process. There's too much risk inherited from taking mastership away from the servers. Part of what I consider to be a severe shortcoming in the voldemort product is the fact that the client has an intimate knowledge of the servers in the cluster, knowing exactly which partitions they have, what servers exist in the cluster and routing all calls directly to the servers based on the partition mapping. This means that any attempt to take the mastering of the cluster topology away from the servers can put your clients in a very unstable state if anything _should_ change. And you may not notice the lack of synchronization between client and server until a host ends up shutdown for maintenance or a hardware failure before things go really bad.

If you don't have the capacity to setup a VIP, then just create a DNS round robin list of hosts under one record and use that. So long as you don't suffer a routing failure or packets getting dropped from a non-responsive server, the voldemort client's FailureDetector will safely manage which hosts it can talk to and will retry bootstrapping against live hosts. But I recommend, in the long run, using a more full-feature VIP of some sort to reduce the odds of poor performance during host failures.

Brendan

--

You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--

Brendan Harris
1.650.291.5761