Cannot get sentinels to work

[Steve V]

 What is the recommended way to troubleshoot sentinel configurations? I have a master and slave on different machines connected to each other successfully. I have three sentinels configured to monitor the master server. When I start the sentinels the one on the master thinks the slave server is down, the other two sentinels think the master is sdown. None of the redis or sentinel logs give any indication that any of the sentinels are finding each other.

The correct firewall ports are open, and all sentinels can communicate with the servers and each other on ports 6379 and 26379. I have verified by using redis-cli to connect to other machines and all can communicate.

I turned the loglevel to debug on the sentinels and I see it rapidly repeating the following on the master:

[11792] 04 Feb 15:37:27.520 . -cmd-link-reconnection slave 192.168.179.237:6379 192.168.179.237 6379 @ staging 192.168.152.67 6379 #Invalid argument

[11792] 04 Feb 15:37:27.520 . -pubsub-link-reconnection slave 192.168.179.237:6379 192.168.179.237 6379 @ staging 192.168.152.67 6379 #Invalid argument

and it repeats like this on the non-master sentinels:

[23140] 04 Feb 15:36:45.361 . -cmd-link-reconnection master staging 192.168.152.67 6379 #Invalid argument

[23140] 04 Feb 15:36:45.362 . -pubsub-link-reconnection master staging 192.168.152.67 6379 #Invalid argument

The sentinel.conf looks like this:

bind 127.0.0.1 192.168.152.67

port 26379

#sentinel announce-ip 192.168.152.67

logfile /var/log/redis/redis-sentinel.log

loglevel debug

dir /tmp

sentinel monitor staging 192.168.152.67 6379 2

sentinel down-after-milliseconds staging 10000

Any help figuring this out would be greatly appreciated.

Thanks,

Steve

[Steve V]

It turns out that even though in redis 2.8 you can bind multiple interfaces, sentinel does not like this. When I took away the 127.0.0.1 everything started working fine. I ended up having to remove the bind declaration altogether though, because having it bound to the private interface only meant that when using redis-cli, I had to always specify the host name of the current machine.

[ch...@netki.com]

Hi Steve, 

I ran into the same errors on sentinel with loglevel debug on; 

5994:X 09 Jun 00:54:03.827 . -cmd-link-reconnection slave 10.0.20.83:6379 10.0.20.83 6379 @ mymaster 10.0.11.127 6379 #Invalid argument
5994:X 09 Jun 00:54:03.827 . -pubsub-link-reconnection slave 10.0.20.83:6379 10.0.20.83 6379 @ mymaster 10.0.11.127 6379 #Invalid argument

But instead of removing the bind declaration from sentinel.conf, I simply re-arranged the order --

from 'bind 127.0.0.1 10.0.11.127' to 'bind 10.0.11.127 127.0.0.1' 

Also, I couldn't even get sentinel to start without a bind declaration in sentinel.conf (this is in Redis 3.2.0); there was no error, just nothing. I guess its not possible to do on my version of Redis. 

Hope that helps someone else, because I'm not getting those hours back. ;)

Cheers,

chet