Django Chat Application using channels and SQS

[Varun Jain]

Problem Statement: Build a scalable chat infra using django/python, do assume that you will have to scale this horizontally. 

I am still to get the other specifics about the product expectations but here is what I understand matters: 

a. Acceptable Latency between messages/events 

b. Number of open connections/user possible per server

c. Size of messages/payload (more on this in a minute). 

This is what my solution needs to look like (generic, not specific to django/python)

Client Web Socket Component  (CWSC)

A group of servers that can: 

a. Accept an incoming user's connection request over web socket protocol and initiate a sticky session. 

b. Accept messages/events from a user 

c. Receives messages/events received for the connected user and passes it on to the frontend client

d. Stores the messages in a persistent db (PostgreSQL) 

e. Broadcasts the message to a  pub/sub style channel for other servers to listen for. 

Central Redis User-Server Map

Since we are assuming a distributed system the two users might not be on the same server. In this case we need to know the receiver user's server (so that it can added as meta-data to the message broadcasted by CWSC). This meta data can be used the receiving server to filter messages meant for it as the pub/sub channel broadcasts to all server's subscribed to a particular channel. 

This db would hold the a mapping of user to server. Whenever a new connection request happens (a) above, it would also add an entry to the redis db. 

Pub/Sub Channel 

This is where messages/events are broadcasted. Assuming simplicity of only one type of event it would only have channel, that all servers would have subscribed to. Other types of channels could be (message read event, picture sending event etc)

Here is my understanding of how this can be done using django-channels: 

1. Django channels is the CWSC component here. It provides for a handy framework to write the implementations 

2. Pub/Sub channel is the channel layer concept within django-channels that uses channel_redis  as the backend.

3. Central redis user-server map is just something I came up with for my design and can be a very simple EC2 server running the same. 

Here are my questions:

1. How does using django channels compare to using socket module directly in python? In terms of performance/scalability. I am not looking for a silver bullet here and understand that it is all a question of product specifics but is it reasonable to expect it to scale to half a million users without us having to open up the package and make modifications to it? My understanding of django-channels is that it is built on socket in python but in a django-friendly manner. 

2. Is channel_redis using redis pub/sub underneath?

3. Can I use something like Amazon SQS instead of channel_redis? If so any relevant resources/packages would be great. 

4. Is there something else you would want me to know?

[Adam Johnson]

Hi!

I think you've found the wrong mailing list for this post. This mailing list is for the development of Django itself, not for support using Django. This means the discussions of bugs and features in Django itself, rather than in your code using it. People on this list are unlikely to answer your support query with their limited time and energy. Read more on the mailing lists at https://www.djangoproject.com/community/

For support, please use the django-users mailing list, or IRC #django on Freenode, or a site like Stack Overflow. There are people out there willing to help on those channels, but they might not respond if you don't ask your question well. Stack Overflow's question guide can help you frame it well: https://stackoverflow.com/help/how-to-ask .

Also if you haven't read it, please take a look at Django's Code of Conduct: https://www.djangoproject.com/conduct/ . These are our "ground rules" for working well as a community, and will help you get the most out of Django and our fantastic community.

Thanks for your understanding,

Adam

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/35606039-9d10-4ed2-9926-46dd9febf765%40googlegroups.com.

--

Adam