Issue with Sentinel links in virtualized ecosystems (was: Redis HA using redis sentinel)

[Leo Volinier]

Hi all!

From a previous thread,

2014-03-01 5:39 GMT-03:00 Salvatore Sanfilippo <ant...@gmail.com>:

On Fri, Feb 28, 2014 at 8:13 PM, Leo Volinier
<leonardo...@gmail.com> wrote:

> Running in a virtualized OpenStack environment, nowadays we have 3 boxes,
> 1-server 1-sentinel each. Both slaves are attached to the master. Sentinels
> are
> configured with quorum=2.
>
>
> When the box with the master fails (network wise: the processes are still
> working
> on each box, but the boxes cannot reach each other), the remaining 2
> sentinels
> just see the sdown state, but don't vote for a new master. Also both slaves
> detects
> the "master link down" state. But nothing reacts. Just stays the same way.

For a Sentinel to try to get voted, ODOWN must be reached, otherwise
the failover can't be triggered.
If it is not reached, usually it is a problem of the Sentinels not
able to communicate since port 26379 is closed.
But it can be a bug as well, the system is every week more mature but
not immune to bugs at all...

 

I think a found the issue here: the quorum is not reached because 

Sentinels cannot see each other. Why? Is not a matter of port, but 

of IP address though.

In virtualized ecosystems (like Amazon EC2 or custom OpenStack),  

the network configuration is kind of special: the system will have 

configured an IP address that is only know by the current virtual 

instance; most of the systems call it the "private IP" and is associated 

to the system network interface (i.e.: eth0). 

This IP is unreachable from other virtual instances in the same 

environment.

So, how traffic can be sent from one virtual instance to another? 

By the IP that are called "Public IP". This IPs are defined from a 

different IP Network and each host will have one. These Public 

and Private IPs are managed by the Environment Orchestrator; 

usually through NAT translation tables.

How this behavior affects Redis? Sentinels announce the 

existence of each other with a given name or IP. When a name

is given, it's traslated to IP; and this translation ends with the 

Private IP, what is useless.

Most of the systems that offers this node data propagation 

allows you to customize the IP address with which other nodes 

can use in order to connect to a specific node (and this IP is different 

to the bind address)

I didn't have the chance to test the new Redis Cluster behavior (just 

analyzed Redis Sentinel) , but definitively, if the IP translation is 

done in the same way, this mechanic can affect the same way.

Cheers,

L.

[Greg Andrews]

Leo Volinier wrote:

In virtualized ecosystems (like Amazon EC2 or custom OpenStack),  

the network configuration is kind of special: the system will have 

configured an IP address that is only know by the current virtual 

instance; most of the systems call it the "private IP" and is associated 

to the system network interface (i.e.: eth0). 

This IP is unreachable from other virtual instances in the same 

environment.

For Amazon, this statement is too sweeping.  My Amazon virtual instances at the same site (like Tokyo Japan) can reach each other's private IP addresses just fine.  It's probably because I've made them part of the same Security Group, with Group settings that grant open access between the members.  I don't use Amazon's VPC, but the documentation I've seen very strongly suggests that instances within the same VPC can also have open access to each other's private ip addresses.  I haven't used OpenStack.

It's just a matter of firewall rules, isn't it?

  -Greg

[Leo Volinier]

Virtualized System use NAT rules in order to map from public IPs to private IPs (the concepts of "public/private" are not in the Internet sense, but in relation to the virtual/physic host). The point is that the firewall rules are OK (because is part of a "bigger thing") and, from the perspective of the application, you need the ability to configure: which IP will be for listening and which IP will be used to announce the application existence. (Examples of other clustered solutions that allows to configure I'm describing: Cassandra[1] and Hazelcast[2]).

But this issue applies to systems that have private/public NAT rules and Sentinel information must go across networks

Cheers,

L.

---

[1] Cassandra: listen_address config param: http://www.datastax.com/docs/1.0/configuration/node_configuration#listen-address 

[2] Hazelcast: public-address config param: not in the official docs, but here is a reference to the parameter: https://github.com/hazelcast/hazelcast/pull/234

[Salvatore Sanfilippo]

Leo, I think that this is the same issue Matt Stancliff fixed for Redis Cluster.
Is it enough if outgoing connections, and the announcement itself, is
limited to the IP address you bind in sentinel.conf via the "bind"
directive?

Salvatore

On Sun, Mar 2, 2014 at 11:49 PM, Leo Volinier

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Salvatore 'antirez' Sanfilippo
open source developer - GoPivotal
http://invece.org

To "attack a straw man" is to create the illusion of having refuted a
proposition by replacing it with a superficially similar yet
unequivalent proposition (the "straw man"), and to refute it
-- Wikipedia (Straw man page)

[Leo Volinier]

Salvatore Sanfilippo wrote:

Leo, I think that this is the same issue Matt Stancliff fixed for Redis Cluster.
Is it enough if outgoing connections, and the announcement itself, is
limited to the IP address you bind in sentinel.conf via the "bind"
directive?

I'm still with 2.8.6. The bind is not enough because the IP is not know by the running Operating System, but is know by the networking systems that forwards traffic to it.

That's why the systems I sampled previously have both configuration params.

Cheers,

L.

[Salvatore Sanfilippo]

On Wed, Mar 5, 2014 at 5:18 PM, Leo Volinier
<leonardo...@gmail.com> wrote:
> I'm still with 2.8.6. The bind is not enough because the IP is not know by
> the running Operating System, but is know by the networking systems that
> forwards traffic to it.

Ok I'll read the thread more carefully in the next days to understand
what to do about it.
I can just provide the ability to directly specify the exact address a
Sentinel should broadcast its presence.

Thanks for the help about this,
Salvatore

[Leo Volinier]

I've read Matt's pull request[1], and it seems to me that is not the same issue.

What I understood from Matt's comments, it's just a convention to take the first bind_addr to announce to other cluster nodes where they can find the current node. But if you cannot bind to that address, Redis will shutdown.

In the case I'm describing you're not able to bind to that address since the system doesn't know it.

On other hand, what I'm asking for should include the case that Matt is trying to tackle.

Cheers,

L.

[1] correct me if I'm wrong, but I assume that Matt's issue is this one: https://github.com/antirez/redis/pull/1567

[Matt Stancliff]

On Mar 5, 2014, at 4:52 PM, Leo Volinier <leonardo...@gmail.com> wrote:

> What I understood from Matt's comments, it's just a convention to take the first bind_addr to announce to other cluster nodes where they can find the current node. But if you cannot bind to that address, Redis will shutdown.
>
> In the case I'm describing you're not able to bind to that address since the system doesn't know it.

Hi. Matt here.

I think I see what you’re saying. Your actual addresses are hiding behind a NAT/forwarding/masquerading/VIP translation layer. You need the Redis equivalent of an “X-Forwarded-For” setting (except originated by the peer, not by the intermediate networking layer).

We could create a config parameter to define “fixed-source-address x.x.x.x” instead of using internally bound IPs for advertising Sentinel communication.

For Cluster communication, it will take a few more changes. Cluster nodes never send their own IP address—the IP of the remote instance is assumed to be the IP on the connecting socket. We’d have to change the way the Cluster instances say hello to each other for fixed addressing to work properly. [Note: allowing Cluster to say hello this way makes my bind patch unnecessary too.]


-Matt

[Leo Volinier]

Matt Stancliff wrote:

We could create a config parameter to define “fixed-source-address x.x.x.x” instead of using internally bound IPs for advertising Sentinel communication.

Just a recomendation: could we stick to its concept and name it simple like: listen-address? Is there any other concept in Redis named Listen Address?

Best regards,

L.

[Matt Stancliff]

On Mar 6, 2014, at 10:19 AM, Leo Volinier <leonardo...@gmail.com> wrote:

> Just a recomendation: could we stick to its concept and name it simple like: listen-address? Is there any other concept in Redis named Listen Address?

What would ‘listen-address’ do differently from the exiting ‘bind’ option?

I could imagine users getting confused between the wording of ‘bind’ and ‘listen-address’ since ‘bind’ already listens on multiple IPs.

I think what we’re trying to fix is Redis lacking the option to say “this is the official IP address of this instance.”

Right now, ‘bind’ has three scenarios: undefined and listen on all interfaces, listen on a single IP address, or listen on up to 16 unique IP addresses. Redis currently trusts the system to just “do the right thing” when connecting to other hosts. The issue we have is Redis re-advertises IP:Port combinations it passively discovers to other hosts. Redis uses IP addresses discovered from socket connections to advertise Redis services to different hosts, but the other hosts may not be able to access the advertised hosts because of system or network configuration issues (since they were passively discovered and not directly announced as “I am reachable as instance 4.2.2.2:35000”).


-Matt

[Leo Volinier]

My answers/thoughts between lines:

Matt Stancliff wrote:

Leo Volinier <leonardo...@gmail.com> wrote:
> Just a recomendation: could we stick to its concept and name it simple like: listen-address? Is there any other concept in Redis named Listen Address?
  What would ‘listen-address’ do differently from the exiting ‘bind’ option?

The difference is that you cannot bind to that address since the box don't know it. As I understand, Redis tries to bind the process to each IP that 'bind' config param have. 

 

  I could imagine users getting confused between the wording of ‘bind’ and ‘listen-address’ since ‘bind’ already listens on multiple IPs. 

From Redis process perspective, you are going to bind to a certain IPs (using bind parameter). From cluster/replication perspective, you want to know in what address other nodes can find the current node.

 

  I think what we’re trying to fix is Redis lacking the option to say “this is the official IP address of this instance.” 

Right, rephrasing: "this is the IP that I want other nodes to know/reach me" 

 

  Right now, ‘bind’ has three scenarios: undefined and listen on all interfaces, listen on a single IP address, or listen on up to 16 unique IP addresses.  Redis currently trusts the system to just “do the right thing” when connecting to other hosts.  The issue we have is Redis re-advertises IP:Port combinations it passively discovers to other hosts.  Redis uses IP addresses discovered from socket connections to advertise Redis services to different hosts, but the other hosts may not be able to access the advertised hosts because of system or network configuration issues (since they were passively discovered and not directly announced as “I am reachable as instance 4.2.2.2:35000”). 

It seems that, for Redis master/slave replication, Redis is consuming "source-address" field from TCP header (using 'getpeername' call at 'anetPeerToString' function), what is not happening for Sentinels discovery or cluster joining (i.e. Sentinels announce themselves with explicit data).

I think you can implement anetPeerToString strategy... looking forward: what will happen if you implement a kind-of gossip protocol where the information is being forwarded by peers though? And I'm still thinking that having via configuration the explicit IP could be a simple solution for this case.

Best Regards,

L.

[Dara Kong]

Hello.

I believe I'm encountering the issue described - sentinel instances in AWS are not able to communicate with one another because the announced address is a private IP address.

I'm using AWS EC2 instances.  There are multiple redis master/slave instances in different regions: California (us-west-1) and Virginia (us-east-1).  I have also sentinel instances in both regions.

The sentinel instances are started with the same config.  Each sentinel instance can detect the redis masters and slaves.  The issue is when it detects other sentinel instances, it immediately detects "sdown".  Here is the log snippet from us-west-1 sentinel instance.

[6317] 16 Jun 18:57:32.341 * +sentinel sentinel 172.XX.XX.XX:6373 172.XX.XX.XX 6373 @ redis1 54.241.XX.XX 6371

[6317] 16 Jun 18:57:32.397 * +sentinel sentinel 172.XX.XX.XX:6373 172.XX.XX.XX 6373 @ redis2 54.209.XX.XX 6372

[6317] 16 Jun 18:57:37.400 # +sdown sentinel 172.XX.XX.XX:6373 172.XX.XX.XX 6373 @ redis1 54.241.XX.XX 6371

[6317] 16 Jun 18:57:37.400 # +sdown sentinel 172.XX.XX.XX:6373 172.XX.XX.XX 6373 @ redis2 54.209.XX.XX 6372

The EC2 instances have two IP address.  Public is in the range 54.XX.XX.XX.  Private is 172.XX.XX.XX.   According to the logs, sentinel is announcing itself with its private IP address.  EC2 instances in separate regions cannot communicate with each other with that address.

Versions used: redis 2.8.9 with Ubuntu 12.04.4

Is there a recommended workaround to configure sentinel to use a different announce IP address?

Thanks.

Dara.

[Matt Stancliff]

Hey Dara,

On Jun 16, 2014, at 3:11 PM, Dara Kong <dara...@gmail.com> wrote:
> The sentinel instances are started with the same config. Each sentinel instance can detect the redis masters and slaves. The issue is when it detects other sentinel instances, it immediately detects "sdown". Here is the log snippet from us-west-1 sentinel instance.
>
> [6317] 16 Jun 18:57:32.341 * +sentinel sentinel 172.XX.XX.XX:6373 172.XX.XX.XX 6373 @ redis1 54.241.XX.XX 6371
> [6317] 16 Jun 18:57:32.397 * +sentinel sentinel 172.XX.XX.XX:6373 172.XX.XX.XX 6373 @ redis2 54.209.XX.XX 6372
> [6317] 16 Jun 18:57:37.400 # +sdown sentinel 172.XX.XX.XX:6373 172.XX.XX.XX 6373 @ redis1 54.241.XX.XX 6371
> [6317] 16 Jun 18:57:37.400 # +sdown sentinel 172.XX.XX.XX:6373 172.XX.XX.XX 6373 @ redis2 54.209.XX.XX 6372
>
> The EC2 instances have two IP address. Public is in the range 54.XX.XX.XX. Private is 172.XX.XX.XX. According to the logs, sentinel is announcing itself with its private IP address. EC2 instances in separate regions cannot communicate with each other with that address.
>

> Is there a recommended workaround to configure sentinel to use a different announce IP address?

We have an in-progress fix at https://github.com/antirez/redis/pull/1708 — Note: the fix requires you specify the required IP address as the first `bind` address for each instance in your config file. You should have proper firewall rules in place prohibiting any servers not under your control from accessing your Redis and Sentinel ports.


-Matt

[Dara Kong]

Hi Matt.

On Monday, June 16, 2014 12:18:51 PM UTC-7, Matt Stancliff wrote:

  We have an in-progress fix at https://github.com/antirez/redis/pull/1708 — Note: the fix requires you specify the required IP address as the first `bind` address for each instance in your config file.  You should have proper firewall rules in place prohibiting any servers not under your control from accessing your Redis and Sentinel ports.

I tried your branch mattsta:sentinel-bind-source-address and updated my sentinel.conf with my public IP:

bind 54.XX.XX.XX

But sentinel fails to start:

[7523] 16 Jun 20:28:12.602 # Creating Server TCP listening socket 54.XX.XX.XX:26379: bind: Cannot assign requested address

Which probably makes sense since the EC2 instance has no knowledge of the public IP address.

Leo described something similar:

I've read Matt's pull request[1], and it seems to me that is not the same issue.

What I understood from Matt's comments, it's just a convention to take the first bind_addr to announce to other cluster nodes where they can find the current node. But if you cannot bind to that address, Redis will shutdown.

In the case I'm describing you're not able to bind to that address since the system doesn't know it.

On other hand, what I'm asking for should include the case that Matt is trying to tackle.

Would it make sense to create a separate config option for specifying sentinel source address instead of "first bind entry"?

Thanks again.

Dara.

[Matt Stancliff]

On Jun 16, 2014, at 5:00 PM, Dara Kong <dara...@gmail.com> wrote:

> I tried your branch mattsta:sentinel-bind-source-address and updated my sentinel.conf with my public IP:
>
> bind 54.XX.XX.XX
>
> But sentinel fails to start:

You’ll have to configure Linux to allow you to bind address that don’t exist on your machine.

Check out this comment: https://github.com/antirez/redis/issues/1667#issuecomment-39751990 along with http://linux.die.net/man/8/setcap.

> Would it make sense to create a separate config option for specifying sentinel source address instead of "first bind entry”?

Yeah, that’s currently just a limitation of how Redis networking has worked forever. It’s currently being reviewed for potential improvements. I’d love to have new behavior for specifying network addresses starting with Redis 3.0 since it’s a major version bump anyway and that’s a great time to throw in maybe backwards incompatible changes.

Thanks for bringing up your issue. The more people we have reporting the same problem makes issues get resolved faster. Feel free to comment or watch the issue on GitHub to keep updated with progress as well.


-Matt

[Dara Kong]

Hi Matt,

On Monday, June 16, 2014 2:11:14 PM UTC-7, Matt Stancliff wrote:

On Jun 16, 2014, at 5:00 PM, Dara Kong <dara...@gmail.com> wrote:

> I tried your branch mattsta:sentinel-bind-source-address and updated my sentinel.conf with my public IP:
>
> bind 54.XX.XX.XX
>
> But sentinel fails to start:

  You’ll have to configure Linux to allow you to bind address that don’t exist on your machine.

  Check out this comment: https://github.com/antirez/redis/issues/1667#issuecomment-39751990 along with http://linux.die.net/man/8/setcap.

That setting did fix the startup issue.  However, using the latest version of Redis (2.8.13) which includes your changes still does not work in my AWS test setup (1 sentinel in California and 1 sentinel in Virginia).  Specifically, sentinel is using the bind address to announce itself to other sentinels.  And other sentinels cannot reach it via that address.

In my case, the bind address fluctuates between 127.0.0.1 and 172.1.2.3 (which is a private IP address) for the CA machine.  But the VA machine cannot communicate with it.  See log:

18740:X 01 Aug 03:39:29.449 * +sentinel sentinel 172.1.2.3:26379 172.1.2.3 26379 @ mymaster 54.1.2.3 7381

18740:X 01 Aug 03:39:31.434 * -dup-sentinel master mymaster 54.1.2.3 7381 #duplicate of 127.0.0.1:26379 or f133e4e6c73218679f06e40e9068f501c7093bf6

18740:X 01 Aug 03:39:31.434 * +sentinel sentinel 127.0.0.1:26379 127.0.0.1 26379 @ mymaster 54.1.2.3 7381

18740:X 01 Aug 03:39:31.677 * -dup-sentinel master mymaster 54.1.2.3 7381 #duplicate of 172.1.2.3:26379 or f133e4e6c73218679f06e40e9068f501c7093bf6

<repeats over and over>

 

> Would it make sense to create a separate config option for specifying sentinel source address instead of "first bind entry”? 

  Yeah, that’s currently just a limitation of how Redis networking has worked forever.  It’s currently being reviewed for potential improvements.  I’d love to have new behavior for specifying network addresses starting with Redis 3.0 since it’s a major version bump anyway and that’s a great time to throw in maybe backwards incompatible changes.

I implemented a quick patch that appears to resolve the issue.  It adds another config option (sentinel announce <ip>) to allow users to specify the announce address.  In my case, I specify the public IP of the EC2 instance allowing both machines to communicate with one another.

https://github.com/dkong/redis/commit/9b6eb0e7d4b879873d46e6d0fac2d9cb4fab4868

Are there any caveats with this approach?

Thanks again for the help.

Dara.

[Matt Stancliff]

On Aug 1, 2014, at 12:09 AM, Dara Kong <dara...@gmail.com> wrote:

> In my case, the bind address fluctuates between 127.0.0.1 and 172.1.2.3 (which is a private IP address) for the CA machine. But the VA machine cannot communicate with it. See log:

Thanks for the report!

> I implemented a quick patch that appears to resolve the issue. It adds another config option (sentinel announce <ip>) to allow users to specify the announce address. In my case, I specify the public IP of the EC2 instance allowing both machines to communicate with one another.
>
> https://github.com/dkong/redis/commit/9b6eb0e7d4b879873d46e6d0fac2d9cb4fab4868
>
> Are there any caveats with this approach?

You found a spot I missed in the force-binding setup. I've simplified your fix to use the existing pattern of “first ‘bind’ address is the server address:" https://github.com/antirez/redis/pull/1908


Thanks for tracking down the problem!

-Matt