redis 3.0.0 servers don't seem to like being slaved to redis 3.0.0 masters

[Bharath Krishnan]

Master running redis 3.0.0 from this ubuntu ppa: https://launchpad.net/~chris-lea/+archive/ubuntu/redis-server

127.0.0.1:6379> info

# Server

redis_version:3.0.0

redis_git_sha1:00000000

redis_git_dirty:0

redis_build_id:484ddb700505ad73

redis_mode:standalone

os:Linux 3.13.0-48-generic x86_64

arch_bits:64

multiplexing_api:epoll

gcc_version:4.8.2

process_id:14033

run_id:ee7ba46ba8a8c54001e58c9c9d3b6e72168dc04f

tcp_port:6379

uptime_in_seconds:175980

uptime_in_days:2

hz:10

Slave also running same version of redis

On the slave, this works fine:

redis-cli -h master

But this doesn't:

redis-cli slaveof master 6379

I see this error in the logs:

15149:S 16 Apr 19:40:47.040 # Unable to connect to MASTER: Invalid argument

It looks like I can replicate the same problem with the slave on os x running redis 3.0 from homebrew.

A slave running redis 2.8.* slaves itself fine to the same master.

Has the semantics of slaveof changed with 3.0 ? Any ideas/help appreciated!

Thanks,

-bharath

[Jan-Erik Rediger]

In principle nothing changed regarding the replication setup.
I can't reproduce this here locally, though I didn't test with the ppa yet.

Somehow one of the internal calls returns EINVAL (Invalid argument), but
only from the log line it's impossible to say which method is causing
this. I suspect the `connect` call.

To which IP does `master` resolv?

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.

[Bharath Krishnan]

master & slave are both on an internal network with 10.x ips. 

slave can connect to master using redis-cli -h using the same ip.

You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/ZrpyfQP8o4Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.

[Bharath Krishnan]

Also, I see this stackoverflow post from a couple of days ago from someone else, looks like the same problem:

http://stackoverflow.com/questions/29622059/redis-slave-unable-to-connect-to-master-invalid-argument

[Tomas Kramar]

I'm running into the same error (EINVAL) in the context of redis cluster and cluster meet command. I tracked the error down to anetTcpGenericConnect and to the connect system call. I managed to extract a minimal test case that fails with EINVAL by pulling out anetTcpGenericConnect and the functions it depends on. Curiously, it also fails with EINVAL on my machine. The code's here, in case it is helpful: https://gist.github.com/kremso/0bb8edffd58720b5d492. That's as far as I got, I don't know much about network programming.

[Jan-Erik Rediger]

What `bind ...` line do you have in the config? 127.0.0.1 first?
It might be that the socket is bound to the local interface and thus it
fails to connect to a remote ip.

Same for the provided gist, it binds to 127.0.0.1, then tries to connect
to 10.x.y.z. It works if I replace 127.0.0.1 with my public ip

[Tomas Kramar]

On 16 April 2015 at 23:15, Jan-Erik Rediger <jan...@fnordig.de> wrote:

What `bind ...` line do you have in the config? 127.0.0.1 first?
It might be that the socket is bound to the local interface and thus it
fails to connect to a remote ip.

No, the bind starts with the private network ip.

 

Same for the provided gist, it binds to 127.0.0.1, then tries to connect
to 10.x.y.z. It works if I replace 127.0.0.1 with my public ip

I changed that on my laptop because the socket would not even bind otherwise. On the server where it fails originally, the code is

 

char *addr = "10.134.82.199";

int port = 18000;

char *source_addr = "10.134.82.201";

I should also say that I'm running this inside docker and trying to bind to a non-local address (enabled in sysctl), see here for more details if that's relevant (https://groups.google.com/forum/#!topic/redis-db/ZLOKATn6vZc). I can connect to the other node (*.199) just fine with redis-cli and issue commands there. I can also connect to the cluster bus port with telnet.

I don't really understand sockets and networking so I might be doing stupid things here, but I'd be really helpful for any clues.

[Tomas Kramar]

It seems that IP_TRANSPARENT option has to be set on the socket to use non-local bind. Just setting this option on socket made it work:

https://gist.github.com/kremso/0bb8edffd58720b5d492#file-gistfile1-c-L107

So if I understand it correctly, this just invalidates the recommended workaround for running redis cluster in docker or in a NAT network :(

Sorry for thread hijack, this was a different error after all. Maybe the OP was also attempting non-local bind?

[Bharath Krishnan]

On the slave, there is no bind interface defined. 

It looks like the redis version the master is running is irrelevant, it fails with the same error on both 2.8 and 3.0 masters.

[Bharath Krishnan]

So it looks like having bind 127.0.0.1 in the config was the issue. Removing it lets the slave sync to master. When I tried this earlier, there was a problem between the chair and the keyboard that got in the way.

If I understand this correctly, with redis 3.0, you cannot have the server listen just on the local host and expect it to slave itself to another server. In redis 2.8.* this works somehow.

[jan....@siteone.cz]

Hi,

I had same issue with Redis 3.

I changed "bind 127.0.0.1 10.234.1.10" to "bind 10.234.1.10 127.0.0.1" and it works!

So, it looks, that "127.0.0.1" cannot be first in the list.

[Salvatore Sanfilippo]

Hello, this is fixed in Redis 3.0.3, could someone having this problem
kindly confirm? Thanks.

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.



--

Salvatore 'antirez' Sanfilippo
open source developer - Pivotal http://pivotal.io

"If a system is to have conceptual integrity, someone must control the
concepts."
— Fred Brooks, "The Mythical Man-Month", 1975.

[Bharath Krishnan]

Yes, I can confirm that this works now in 3.0.3.

Thanks,

-bharath

[Salvatore Sanfilippo]

Great! Thx

open source developer - Redis Labs https://redislabs.com