Redis (Sentinel) behind Proxy (HAProxy) becomes read only (no master) when adding a new replica.

231 views
Skip to first unread message

Aaron Layfield

unread,
Jul 16, 2021, 8:00:02 AM7/16/21
to Redis DB
I have a HAproxy instance proxying requests to Redis running HA via Sentinel. 

For reading and writing keys, this seems to work perfectly. 

However, when trying to add a new RO Replica outside of the cluster, Redis seems to break down and the elected master suddenly become a slave; 

This is what the Redis master reports when the new replica comes in; 

```
1:M 16 Jul 2021 00:55:30.576 * Starting BGSAVE for SYNC with target: replicas sockets
1:M 16 Jul 2021 00:55:30.577 * Background RDB transfer started by pid 60841
1:S 16 Jul 2021 00:55:43.621 # Connection with replica 10.98.218.154:6379 lost.
1:S 16 Jul 2021 00:55:43.621 # Connection with replica 10.102.203.218:6379 lost.
1:S 16 Jul 2021 00:55:43.621 # Connection with replica 10.244.2.248:6379 lost.
1:S 16 Jul 2021 00:55:43.621 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
1:S 16 Jul 2021 00:55:43.621 * Connecting to MASTER 10.110.213.143:6379
1:S 16 Jul 2021 00:55:43.621 * MASTER <-> REPLICA sync started
1:S 16 Jul 2021 00:55:43.621 * REPLICAOF 10.110.213.143:6379 enabled (user request from 'id=284265 addr=10.244.2.248:47506 laddr=10.244.7.105:6379 fd=263 name=sentinel-c75c7ad1-cmd age=10 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=202 qbuf-free=40752 argv-mem=4 obl=45 oll=0 omem=0 tot-mem=61468 events=r cmd=exec user=default redir=-1')
1:S 16 Jul 2021 00:55:43.657 # CONFIG REWRITE executed with success.
1:S 16 Jul 2021 00:55:43.660 # Diskless rdb transfer, last replica dropped, killing fork child.
60841:signal-handler (1626396943) Received SIGUSR1 in child, exiting now.
1:S 16 Jul 2021 00:55:43.668 # Background transfer terminated by signal 10
```

The replica says this;

```
1:S 16 Jul 2021 00:55:22.812 * Ready to accept connections
1:S 16 Jul 2021 00:55:22.813 * Connecting to MASTER redis-instance:6379
1:S 16 Jul 2021 00:55:24.101 * MASTER <-> REPLICA sync started
1:S 16 Jul 2021 00:55:24.317 * Non blocking connect for SYNC fired the event.
1:S 16 Jul 2021 00:55:24.534 * Master replied to PING, replication can continue...
1:S 16 Jul 2021 00:55:24.750 * Partial resynchronization not possible (no cached master)
1:S 16 Jul 2021 00:55:30.425 * Full resync from master: 527a33af7421b8c10693f63cb9291e310a03ab87:684836278
1:S 16 Jul 2021 00:55:30.427 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
1:S 16 Jul 2021 00:55:47.049 # I/O error trying to sync with MASTER: connection lost
1:S 16 Jul 2021 00:55:47.050 * Reconnecting to MASTER dns-redis.eencloud.com:6379 after failure
1:S 16 Jul 2021 00:55:47.602 * MASTER <-> REPLICA sync started
1:S 16 Jul 2021 00:55:47.818 * Non blocking connect for SYNC fired the event.
1:S 16 Jul 2021 00:55:48.033 # Error reply to PING from master: '-Reading from master: Invalid argument'
```

If I bypass HAProxy entirely, this works. It seems to be something specific with the proxy.

This is the command I am running to attach the RO Replica;

`docker run --name redis-slave redis:6.2.4-alpine --slaveof <redis-address> 6379 --replica-read-only yes`

I've tried to enable `PROXY` protocol, but it looks like Redis doesn't like this either; 


Reply all
Reply to author
Forward
0 new messages