Django channels deployment

[Sgiath]

Hi I am working on some app with Django Channels and I am in the phase when I am looking for the best way how to deploy the Channel project (with scaling and smooth update in mind).

I have this setup:

1. Nginx server handling SSL certificate, headers, etc. it will proxy to ->
2. Daphne server - question: I can run Daphne with just two files - asgi.py and settings.py (which will match the Django app settings)?
   
3. Redis Channel backend server
4. Redis Cache server
5. MySQL database
6. Server with actual Django code running the workers (few for http.* few for websocket.* channels and few universal).

If I would like to scale I can:

• Run multiple Daphne servers and load-balance them on the Nginx server
• Run multiple Redis Channel servers in one shard
• Run multiple servers with workers

If I would like to update to the new Django code:

• Just load new code on the worker servers and restart the workers (in case I wouldn't change the CHANNEL_LAYERS settings).

Will it work? Are there the only three conditions? 1. All Daphne servers has to see the Redis, and all workers has to see the Redis too. 2. The Redis has to looks like one instance. 3. Daphne and workers has to have the same CHANNEL_LAYERS settings.

Right now I am in the proof of concept phase and I all test it with the docker-compose (every part in my architecture is separate Docker image) and it is working (only difference is that I am running Daphne and workers on the same machine but I am in the middle of separate them) but I want to be sure that it will work on actual separate servers (I want to use Google Cloud platform) before I will present this idea to my boss :) .

Thanks for your help

Sgiath

[Andrew Godwin]

Hi,

Yes, you can run daphne with just an ASGI file that matches the Django settings. It's easier to just use the Django project for this, but you also don't need to update/restart it unless the CHANNEL_LAYERS setting changes.

The conditions you put down are mostly correct, apart from 2), having one redis instance. You can supply the Redis channel layer with multiple Redis hostnames and it will automatically shard between them - however, if you do this you must have enough workers to consume from all of the shards or you may miss messages. I recommend sticking with one for now, as a single Redis can handle a LOT of throughput. You also can't use redis-cluster, as it doesn't support some of the operations we use (like BLPOP)

Andrew

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/c1eac5c2-b359-4e67-96a8-86fba12f9a07%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sgiath]

And if I would like to separate some of the infrastructure - for example dedicate some resources just for one customer. Can I run some workers for the request based on the, for example, path?

Example:

I want 10 workers consume just messages from path example.com/customer1 and all messages from example.com/customer1 will be consumed by those 10 workers.

I guess I can separate the interfaces servers - I will filter it on the Nginx server.

Can I also run (in the distance future with millions of requests :) ) separate Redis server for the 1 customer? Can I dedicate some workers and interface servers to just talk to specific Redis server?

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

[Andrew Godwin]

You can separate workers by channel name, but not by path. You're probably better off doing it further upstream with nginx as you said.

You also can't just run one redis server per customer unless you move to separate installs for bigger customers, with their own Daphne, Redis and workers.

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/d6376196-ce62-4bf1-8094-4032b525fdb3%40googlegroups.com.