Redis Configuration

[rame...@familysearch.org]

As I am working to setup an instance of CMB in our data center, I am struggling with configuring Redis.  I have a 4 node C* ring, 4 boxes dedicated to CMB configured such that there is 1 cqs api, 1 cns api, 2 cns workers, 1 consumer and 1 producer.  I also have 4 boxes dedicated to Redis, each running one instance of Redis.  If I list all 4 of the Redis boxes in the cmb.redis.serverList property, the api servers will start but they do not function properly.  I can publish messages but I can not receive messages.  If I list only one server in that property, everything seems to work.  I was able to reproduce this on my local host with running the tests.  When I start an instance of cmb with only a single redis server listed, then run all the tests they all pass, if the cmb instance has multiple redis instances listed, multiple tests fail.  The com.comcast.cqs.test.unit.CQSLongPollTest hung for three hours before I finally killed them, so the tests never finish.  I am running with the most up to date code from github, 2.2.41.  I have included the output from both the JVM running CMV and maven running the tests from both having a single redis instance and having multiple.  I have not looked into a master slave setup for redis yet, but I am wondering what is going wrong here. 

[boriwo]

I just tested this and can confirm this behavior for the latest version of CMB: For configurations with more than one redis shard, receiving messages occasionally fails. The issue should occur more frequently if more shards are present. I just checked a tentative fix into head but will have to do some more testing before creating a new release. Will provide an update early next week.

[rame...@familysearch.org]

I have confirmed that this change is working, I am able to configure CMB to use multiple redis shards, then have the tests pass.