Unable to connect to MASTER: Resource temporarily unavailable : Connecting to MASTER no:0

[Toshiki Inami]

Hi,

I'm currently working on Redis cluster for query caching with Keepalived. 

The point is

 - Redis + Keepalived simple active stand/by implementation

 

This makes you more clear to understand the overview.

The problem I'm facing is when I shutdown/start ( restart )the host1 server, it should keep the exact same virtual ip, and start replication with host 2.

But syslog outputs "Unable to connect to MASTER: Resource temporarily unavailable : Connecting to MASTER no:0", and fail the replication.

I guess that is happened because of OOM, and the kernel kill the process.

So I implement these below

- vm.overcommit _memory = 1

- set up swap file

and made sure that I have enough amount of memory for redis ( 2 times  the memory usually used ) 

Do you have any ideas?

Thanks,

[陈逸云]

I think when host1 server shutdown ,host 2 will hold the virtual ip(ip a cmd will show the result) which means host2 is the master;when you restart host1,keepalived will make it slaveof host2(shell script) 

在 2016年1月5日星期二 UTC+8下午5:10:28，Toshiki Inami写道：

[Toshiki Inami]

Hello,

Thanks for visiting this issue to help me out.

However I'm so sorry that I didn't put enough information to understand the design & implementation within the post above, so I'll update this issue.

During Master-slave replication, the replication fails and I irregularly see the errors :

Unable to connect to MASTER: Resource temporarily unavailable : Connecting to MASTER no:0in slave syslog.

####################################################################

First off, I need to explain the design concept & layout little here, cause I have no clue about what’s behind this error message above.

I'm currently working on a highly available redis cluster for query caching using Redis + Keepalived. ( I don’t use Redis cluster. )

Tools

• Redis:
• Keepalived: network level load balancer, we will use keepalived to publish an virtual ip and manage failover.

Layout

• Keepalived1 ( belongs to host1 server)
• Keepalived2 ( belongs to host2 server)
• reddis1 ( belongs to host1 server )

• reddis2 ( belongs to host2 server )

• ( I plant to use supervisor to revive processes but not implemented yet. )

How do we reach 100% availability using the above set up?

Keepalived is not able to either switch Redis state or failover as default, so I made some scripts triggered whenever notify_master/notify_backup/notify_fault called by Keepalived.
So each hosts has notify_master/notify_backup/notify_fall scripts.

Within the scripts,

• notify_master.sh

triggers the command : `redid-cli slaveof no one`, and 
rewrite redis config with the same state as above.( redid-cli slaveof no one )

• notify_backup.sh

triggers the command : `redid-cli slaveof <IP> <PORT>`
rewrite redis config with the same state as above.( `redid-cli slaveof <IP> <PORT>`)

• notify_fault.sh

triggers the command : service keepalived stop ( force to failover.)

Keepalived also has check_script for health checking whether Redis is alive.

• redis_health_checker.sh ( Send ping and get pong periodically )

#!/bin/bash

REDIS_CLI='/usr/local/bin/redis-cli'

for TRIES in $(seq 1 2)
do
  RESULT=$($REDIS_CLI ping)
  if [ "${RESULT}" = "PONG" ]; then
    exit 0
  fi
  echo "Ping failed ${TRIES}" | logger -t redis-health-checker
  sleep 1
done

exit 1 # Makes Keepalived FALL

####################################################################

The suspected issue

For testing I shuts down&starts ( restarts ) whichever server having redis (master) state, and is supported to start replication with redis (slave), but I found it sometimes fail. ( It’s not always but sometimes, I mean irregularly.)
However I found in redis ( slave ) outputs below in syslog.

Unable to connect to MASTER: Resource temporarily unavailable : Connecting to MASTER no:0

I doubt something like

• fails because of lack of memory when forking
• fails because of lack of swap space so that system calls fails
• fails because the user can’t make enough process

But it didn’t work in this case.

I just wonder why this happens irregularly.
What can I do to find out what went wrong? I couldn’t figure out Unable to connect to MASTER: Resource temporarily unavailable : Connecting to MASTER no:0 in this situation.

Does anyone ever encounter this kind of an issue with Redis?

[Toshiki Inami]

Hello, 

Thanks for posting.

Yes, that's true. I use notify_master/notify_backup to handle these situation so that Keepalived can failsover to the other one.

And I  wrote some shellscript to make redis2( host2 ) starts replication with redis1 ( host1 ) as master, but irregularly fails replication with error "Unable to connect to MASTER: Resource temporarily unavailable : Connecting to MASTER no:0". 

I also updated the my previous post, so that you can see the little more of the design.

Thanks,

[Greg Andrews]

This part of the error message looks suspicious:   "Connecting to MASTER no:0"

It looks like the Redis process is being configured to connect to a host named "no" on port number 0.  That doesn't make a lot of sense to me.  I would expect the ip address of the master (perhaps the virtual ip address that keepalived manages) and the port number 6379.  Perhaps you're using a different port number for Redis, but it's not 0, correct?

It looks like the re-configuration is wrong, so Redis is trying to connect to the wrong host and port.

  -Greg

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Toshiki Inami]

Thank you so much Greg. You just saved my life.

I only focused on Unable to connect to MASTER: Resource temporarily unavailable but actually all problems stem from Connecting to MASTER no:0.

It occurred because of the order of the boot up process. 

When the server got rebooted, it actually starts off with Redis => Keepalived, which ends up host2 server doesn't execute redis-cli slaveof <IP> <PORT>. 

That's why I got the error such as Connecting to MASTER no:0, which means connected to nobody. I modified boot up process with update-rc.d command, and fixed it up.

 Thanks you all ,you guys