Redis HA with Sentinel enabled loses master when adding read-only replica through HAProxy

141 views
Skip to first unread message

Aaron Layfield

unread,
Jul 16, 2021, 8:00:01 AM7/16/21
to Redis DB
I am the maintainer of this Kubernetes helm chart: https://github.com/DandyDeveloper/charts/tree/master/charts/redis-ha

This uses HAProxy to expose a Redis HA instance running with Sentinel. 

I have it deployed in an environment working with Sentinel and working as expected. 

My master is showing as OK: 

```
# Replication
role:master
connected_slaves:2
min_slaves_good_slaves:2
slave0:ip=10.98.218.154,port=6379,state=online,offset=27183,lag=0
slave1:ip=10.102.203.218,port=6379,state=online,offset=25304,lag=1
master_failover_state:no-failover
master_replid:2d82e881938a7de85b6278b3229aa7b784ce72fc
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:27430
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:27430
```

However, the moment I try to externally attach a Redis Readonly using the following: 

`docker run --name redis-slave redis:6.2.4-alpine --slaveof <haproxy address> 6379 --replica-read-only yes`

The master just becomes a slave suddenly and everything effectively become RO: 

```
1:M 15 Jul 2021 02:33:22.480 * Replica 10.244.3.125:6379 asks for synchronization
1:M 15 Jul 2021 02:33:22.480 * Full resync requested by replica 10.244.3.125:6379
1:M 15 Jul 2021 02:33:22.480 * Delay next BGSAVE for diskless SYNC
1:M 15 Jul 2021 02:33:27.859 * Starting BGSAVE for SYNC with target: replicas sockets
1:M 15 Jul 2021 02:33:27.860 * Background RDB transfer started by pid 168
168:C 15 Jul 2021 02:33:33.710 * RDB: 6 MB of memory used by copy-on-write
1:M 15 Jul 2021 02:33:33.710 # Diskless rdb transfer, done reading from pipe, 1 replicas still up.
1:M 15 Jul 2021 02:33:33.798 * Background RDB transfer terminated with success
1:M 15 Jul 2021 02:33:33.798 * Streamed RDB transfer with replica 10.244.3.125:6379 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming
1:S 15 Jul 2021 02:33:36.029 # Connection with replica 10.98.218.154:6379 lost.
1:S 15 Jul 2021 02:33:36.030 # Connection with replica 10.102.203.218:6379 lost.
1:S 15 Jul 2021 02:33:36.030 # Connection with replica 10.244.3.125:6379 lost.
1:S 15 Jul 2021 02:33:36.030 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 15 Jul 2021 02:33:36.030 * Connecting to MASTER 10.110.213.143:6379
1:S 15 Jul 2021 02:33:36.030 * MASTER <-> REPLICA sync started
1:S 15 Jul 2021 02:33:36.030 * REPLICAOF 10.110.213.143:6379 enabled (user request from 'id=1356 addr=10.244.3.125:42800 laddr=10.244.7.99:6379 fd=669 name=sentinel-525e6976-cmd age=11 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=202 qbuf-free=40752 argv-mem=4 obl=45 oll=0 omem=0 tot-mem=61468 events=r cmd=exec user=default redir=-1')
```

One interesting thing is this only happens running through HAProxy over TCP. This address `10.244.3.125` is HAProxy.

If I do a local `port-forward` from Kubernetes and directly go to Redis, it seems to succeed without error. 

Anyone have any ideas what might be causing this? `Waiting for REPLCONF ACK from slave to enable streaming` is it possible HAProxy isn't passing back this ACK?
Reply all
Reply to author
Forward
0 new messages