Redis Failover Issue (using Booksleeve for .Net)

[Eric Maxwell]

I have Redis running on CentOS in the following configuration:

master mymaster @ 6379

slaveof mymaster @ 6380

slaveof mymaster @ 6381

sentinel @ 26379

I'm using c# and Booksleeve library and a modified ConnectionGateway similar to the one in this thread:

In testing, I'm trying to ensure failover happens gracefully. I've tested that Sentinel does what's expected, from the Redis end of things meaning I can take mymaster offline and watch Sentinel promote a slave to master and configure the additional slave appropriately. So I know Sentinel is working as expected. I wrote a quick test which adds a million key/values in a simple loop. 

Sample code that works:

string key = "eric";
string value = "boss"; 

for (int i = 1; i < 100000; i++)
{

int milliseconds = 250;

Thread.Sleep(milliseconds);

AddKeyValue(key + i.ToString(), value + i.ToString());

}
public void AddKeyValue(string key, string value)
{

Redis = ConnectionManager.GetConnection(); // Uses ConnectionGateway linked above

Redis.Strings.Set(0, key, value);

}

Running the above code, I can be mid-loop and pull mymaster 6379 offline and the code performs as expected... the ConnectionGateway detects a closed connection, pings the Sentinel for a set of endpoints, receives the new master which was promoted via failover, and the code continues to process as expected. When I remove the Thread.Sleep, then things start crashing.

Sample code that does NOT work:

string key = "eric";
string value = "boss";
for (int i = 1; i < 100000; i++)
{

AddKeyValue(key + i.ToString(), value + i.ToString());

}
public void AddKeyValue(string key, string value)
{

Redis = ConnectionManager.GetConnection(); // Uses ConnectionGateway linked above
Redis.Strings.Set(0, key, value);

}

When I run the above code without a Thread.Sleep, I get the following error:

System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host.
System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.IO.BufferedStream.Flush()
   at BookSleeve.RedisConnectionBase.Flush(Boolean all) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1499
   at BookSleeve.RedisConnectionBase.EnqueueMessage(RedisMessage message, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1965
   at BookSleeve.RedisConnectionBase.ExecuteVoid(RedisMessage message, Boolean queueJump, Object state) in d:\Dev\BookSleeve\BookSleeve\RedisConnectionBase.cs:line 1695
   at BookSleeve.RedisConnection.BookSleeve.IStringCommands.Set(Int32 db, String key, String value, Boolean queueJump) in d:\Dev\BookSleeve\BookSleeve\IStringCommands.cs:line 413
   at xxxxxx.CloudDataAccess.CloudDataAccessor.AddKeyValue(String key, String value) in C:\ xxxxxx\Server\Data Model\CloudDataAccess\DataAccessor.cs:line 51
   at xxxxxx.Clerks.CloudClerk.TestRedis() in C:\xxxxxx\Server\Data Model\Clerks\Cloud\CloudClerk.cs:line 73
   at xxxxxx.Managers.CloudManager.TestRedis() in C:\ xxxxxx\Server\Managers\Cloud\CloudManager.cs:line 302
   at xxxxxx.Admin.General.RedisTools.uxTestRedis_Click(Object sender, EventArgs e) in C:\ xxxxxx\Server\Presentation\Admin\RedisTools.aspx.cs:line 92

Note: I've tested this code while just connected to the Master mymaster and not using Sentinel, and have received the same results. 

We have many task processors that will be simultaneously hammering Redis, so this test condition seems valid and is not an edge case in our current configuration / architecture. Any insight or help is greatly appreciated.