Channels/Redis fault tolerance with Twemproxy?

[Filbert]

Going to have 10 or more EC2 instances running Channels 2.0.  I really don't want to add the complexity of Redis Cluster or Redis sentinel.  Is a decent fault tolerant solution to implement Twemproxy, instead.  I don't care about data loss if an instance crashes, but I do care that the application continues to operate on the remaining instances.

Is this a workable solution?

[Andrew Godwin]

I'm afraid I've never used Twemproxy, so I can't really comment on it, but if you don't care about data loss, it's probably fine?

Andrew

On Fri, Jul 20, 2018 at 3:30 PM Filbert <tim...@gmail.com> wrote:

Going to have 10 or more EC2 instances running Channels 2.0.  I really don't want to add the complexity of Redis Cluster or Redis sentinel.  Is a decent fault tolerant solution to implement Twemproxy, instead.  I don't care about data loss if an instance crashes, but I do care that the application continues to operate on the remaining instances.

Is this a workable solution?

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d1d50730-c1ba-4674-8ef8-54294158d0f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Filbert]

And just to be clear as to my options, using Channels 2.0 I can use Redis cluster if I have to to get failover and fault tolerance, correct?

[Andrew Godwin]

I don't think redis-cluster supports BLPOP properly still so I don't think you can use it.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/42fe264e-7f1f-426a-82c6-36aed944a923%40googlegroups.com.

[Filbert]

Thanks Andrew,

So with a load-balanced ten web server implementation using Channels (and of course ten instances of redis) do I have a HA solution then?  RIght now Django settings simply has a list of those servers in CHANNEL_LAYERS, what if (say) three of them crash, will Channels continue to try and hash to the dead servers? Again, I'll accept a brief set of errors, but if this effectively causes Channels to continue to error that is not my definition of HA.

Thanks.

[Filbert]

Well in digging through posts I see Sentinel may be an option, does that mean if my CHANNEL_LAYER would have a single host entry?

[Andrew Godwin]

The Sentinel option was removed in the 2.0 rewrite as it didn't have someone to help maintain it (I didn't have the time then and don't right now), so you are right, there's currently no easy HA option for Redis with Channels.

If you're interested in one, I can give you some tips on what code needs to be written, but it's not something I really want to maintain myself as part of the greater Channels effort as HA solutions tend to be quite workload-heavy to develop and test and end up being specific to certain deployment scenarios. Contributions of code that could help are welcome, though.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/ae01076f-732c-4574-b878-ccfd7d9ff20d%40googlegroups.com.

[Filbert]

Andrew,

Just to be clear, there is no reason I can't use haproxy to front end redis sentinel to effectively hide all this from channels 1 or 2 right?  Very surprised there isn't anyone running this in production that doesn't need some form of HA.  Buried trying to get a product out the door right now, but as I get time and more familiar with channels I'll probably dig in deeper on this.

Thanks.

[Andrew Godwin]

Yup, any sort of TCP-level proxy with failover will work for this sort of thing, but you'll likely see some errors whenever you failover as Channels loses its blocking connection that it's using to wait for messages - as always, please test a failover before you rely on it!

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/fac4a285-610e-4814-9621-0246ed82ccb4%40googlegroups.com.

[Filbert]

Wow, I thought I had this working using haproxy in front of redis-sentinel.  haproxy fails over when redis-sentinel fails over within 30 seconds.

Problem is Daphne (Channels 1.X), reports over and over:

Error trying to receive messages: Error running script (call to f_3640886a0c8901ca9188f5f7a5f7a346145b9c5f): @user_script:3: @user_script: 3: -READONLY You can't write against a read only slave.

Which I assume means Daphne doesn't detect a problem and just keeps the socket open on the (old) master that is now the slave.

[Andrew Godwin]

Channels 1 and its Redis bindings have different behaviours to 2, so this might be improved there (but I can't guarantee it).

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/ca3c7d0a-435d-4d69-968f-acd1d4364d2f%40googlegroups.com.

[Tim Nelson]

For sure, in order to support haproxy/redis-sentinel asgi_redis would have to handle redis-py's   ReadOnlyError and reopen the connection. Way above my pay grade to inject this into 1.X.  It seems 2.X might suffer from the same issue as I don't see it handling this condition either.  Still baffles me no one is using some for of fault-tolerant Redis in production yet. Surely others must have been bitten by this already.

I'll have to weigh the merits (and pain) of migrating to Channels 2 this late in the game, we don't have many consumers, but we do some funky things with session data since we are multi-tenant.

You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/3_IKqTn-SVE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAFwN1ur%2B%3DSg6Z5Nj0oTQ5W3AXXF8NLhAooCt74D4%2By%3DbzaAwBg%40mail.gmail.com.