Push notifications in a Django site

[Ram Rachum]

Hi guys,

I need some advice.

We're building a site that's going to require push notifications. (e.g. user gets a real-time notification from the server in the browser.) What are our options of doing this in Django?

I know that this runs counter to how WSGI works, so I'm definitely expecting that getting this to work in Django would take some amount of blood, sweat and tears, the question is how much :) I wouldn't like to abandon Django for something like Node.js, I love Django.

What are our options? Does anyone have a Django site with Push notifications sent out and can speak from experience?

Thanks,

Ram.

[oz katz]

If by "push notifications" you mean a long lived connection between server and client (i.e. websockets or even long polling), then yes, getting this to work inside a Django WSGI app is not easy. I've seen a few attempts using gevent but they were hacky and/or not well maintained.

Your best option there is to run a separate server that can handle many simultaneous connections, capable of solving the infamous c10k problem such as Twisted or even node.js with socket.io, and creating some sort of bridge between your django app and that server.

Redis pub/sub could be a good fit there.

However, if a few seconds of delay are not critical for your use case, and you don't mind having your servers hit every few seconds, many apps can happily live with standard XHR requests every 10-15 seconds per connected user. This can, of course, live inside your standard WSGI app.

Hope I helped.

Oz 

--
You received this message because you are subscribed to the Google Groups "PyWeb-IL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyweb-il+u...@googlegroups.com.
To post to this group, send email to pywe...@googlegroups.com.
Visit this group at http://groups.google.com/group/pyweb-il.
For more options, visit https://groups.google.com/groups/opt_out.

[Emanuel Ilyayev]

Another - really easy to implement solution - just for the beginning - would be to write a small middleware that will check notifications status every time the user loads a page...

It is not "the best solution" from a functional perspective but it is definitely the easiest to implement and you can always improve it after that with one of Oz's suggestions.

[Paul Walsh]

We've dealt with a similar issue quite recently, and just implemented simple XHR polling of an endpoint every 5 seconds.

This requires no additional infrastructure overhead and will get you a long way. 

When you have real issues of scale, you can swap out this implementation for a "proper" solution on Twisted or Node, using Redis for pub/sub between the Django and Node/Twisted app - as already mentioned.

[Yuval Adam]

Just wanted to mention Tornado as teh obvious pythonic choice for a real-time server

[1] - http://tornado.readthedocs.org/en/stable/

[2] - http://blog.y3xz.com/blog/2012/06/08/a-modern-python-stack-for-a-real-time-web-application/

--

[Sim Zacks]

If you _really_ need push, there is Pusher - http://pusher.com/tutorials/html5_realtime_push_notifications, which has a python/django wrapper - https://pypi.python.org/pypi/django-pusher

My advice would be doing regular polling, as others have suggested.

[Udi Oron]

Hi!

אני מקוה שאני לא חופר (טלדר), אבל הנה כמה תובנות וקישורים:

Moving from classical web site/app to real time web usually requires considering at least some design changes, even before going to implementation details.

For example: if your user opens his browser, get some info and notification, and later goes offline (leaves the office with his laptop) and then gets back online, what would you like the app to do?  Some solutions might be: show all missed notifications (chat:IM style), ignore missed notifications (chat:IRC chatroom style), require a refresh of the page (price ticker?), inform the user and request the user to reload (bank account), or a combination of all the above.  This question has to be answered before going forward with implementation.

There are also many types of notification flows: for example, in a chat room or a price ticker, a single message can be generated by the server, and later published to many subscribers, allowing to leverage simple, (and even standard) pubsub mechanisms outside the web server itself.  Personal updates, as in email clients probably won't be able to benefit from this, and will still require each message to be generated by the server itself.

When getting closer to implementation details, more thought should be put in dividing the work between the server and the client.  Obvious possibilities will be generating all at the server, while sending rendered html to very light and simple JS client.  A more modern approach will move all rendering to the client, and send data as JSON.  This may include showing an empty page at load, and pumping all data through the socket on connection (see sails.js for example).  Intermediate solutions also exist :-)

Once you figured out what you want to do, here are some solutions:

To get thingss working very quickly, you can start with pubnub: https://github.com/pubnub/python/tree/master/python

Subscribe from js, and publish from python. This scales very good, and requires extremely minimal changes to your code, and no infrastructure change at all, but costs $$$$.

If you want to some some bucks, you can implement a pubsub service yourself.  From now on you will need to introduce two new technologies to your stack:

- A non-blocking async webserver.  Tornado is the usual suspect.

- A shared memory or messaging queue to allow different tornado instances to communicate between them, allowing load (clients) to be shared among more than one process/pc.  SQL is (probably) not good enough, as it does not allow subscribing to events on data change.  The standard for this service is redis.

Here is a simple, concise yet full tutorial and example for a pubsub service, that can even be used on Heroku, implemented with flask and flask-sockets:
https://devcenter.heroku.com/articles/python-websockets

If you want to go further down the road, and use sockets with your django code, you would probably need to get read from WSGI.  However, you can to this only to your websockets: you can write a small tornado app and import your django code from it.  You can use nginx routing to divert your websocket urls to your tornado app, or if possible, serve it from a different host/vhost/port.

The following article surfaces lately, allowing to reroute your django views via tornado, but I haven't tested it yet, and it seems like it might be an overkill:
https://github.com/GetBlimp/django-websocket-request
Blog post: http://jpadilla.com/post/74391616727/the-easiest-way-to-add-websockets-to-django
HN Discussion: https://news.ycombinator.com/item?id=7130498

Good luck!

Udi

--

[alonn]

+1 for pubnub as quick (but a little costly solution, also cheaper then pusher). I got good experience with it. If polling is good enough for you then you should certainly consider caching (redis probably) the answer. so you don't have to hit db every 5 seconds, just to find out there isn't a new result, but update the redis cache when a new notification is generated. seems that in the longer run, if this is a core feature (as in my current workplace) you should implement a different async server to handle this as mentioned before in this thread: tornado, node.js (probably the best way to go if you dare to venture outside of the python comfort zone) etc

[Ram Rachum]

Thanks everybody for all your suggestions!

--

[Tom Gurion]

I was looking for push notification too just a few weeks ago. In the beginnig I used pusher. It works just fine but I ended up using tornado for websockets and run django from within tornado.

Here's a post I wrote about the configuration on facebook:

התלבטויות לגבי קונפיגורציית עבודה שניסיתי ונראית חלומית בנתיים.
במקום להריץ django בקונפיגורציה שלמדנו (או דרך nginx + guincorn בפרודוקשיין). להריץ django בתוך tornado. כאן יש הסבר מלא:https://github.com/bdarnell/django-tornado-demo.
יתרונות:
1. לדבר עם django ב- http ועם tornado ב- websockets, עם כל הייתרונות המשתמעים מעבודה ב- websockets כמו push ל- client ותקשורת דו כיוונית "אמיתית" באופן כללי. זאת גם הסיבה שבגללה התחלתי עם כל הסיפור הזה ואת התוצאות אפשר לראות בלינק בסוףהמגילה.

2. אין שום בעייה להריץ פקודות ברקע ופקודות מתוזמנות. זה בדיוק שתי שורות קוד ב- tornado ואין צורך ב- Celery / Redis / RabbitMQ וכו'. הנה הסבר: http://didipkerabat.com/post/17994754092/rarely-mentioned-benefits-of-tornado-framework.

3. תקשורת בין ה- frameworks יכולה להתבצע ב- http (סטייל RestAPI) או לחילופין פשוט לטעון את המודלים של django ב- tornado ולעבוד איתם שם.
4. הרצת פרוייקט ב- development וב- production נעשית בקונפיגורציה הרבה יותר דומה לעומת django לבד, שזה תמיד טוב.
5. יתרון שולי אבל כיף מאוד שהוא קיים: לכל האמור מעלה אפשר לעשות deploy ל- heroku בגרסא החינמית. מה שבטוח אי אפשר כשרוצים כמה processים של django.

Here are two projects that I'm working on that use this configuration. 1, 2.

I'm new to web development, so if my suggestion is really bad, please tell me :-)

Tom

[Ram Rachum]

Very interesting!

--

[alonn]

Very interesting! can elaborate about this in a fully fledged blog post?

[Meir Kriheli]

Two comments:

• Won't that leave the Django app basically to a single thread execution (can handle only a single request per Tornado instance) ?
• Using Django's models etc. in Tornado's code will block the ioloop.

Cheers

2014-03-05 18:01 GMT+02:00 Tom Gurion <nagas...@gmail.com>:

--

You received this message because you are subscribed to the Google Groups "PyWeb-IL" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyweb-il+u...@googlegroups.com.
To post to this group, send email to pywe...@googlegroups.com.
Visit this group at http://groups.google.com/group/pyweb-il.
For more options, visit https://groups.google.com/groups/opt_out.

--

Meir Kriheli
http://meirkriheli.com

[Tom Gurion]

Meir and Alonn,

I hope that this weekend I will have time to research this configuration a bit more.

I will write about the findings in my blog and post it here.

Tom