Django-channels and JSON-RPC

[Fabien Millerand]

Hi everyone,

I am looking to implement a websocket server based on Django using JSON-RPC protocol.

I have been looking around for a pre-made solution without success. I am also a newbie in Django so I am a little bit lost...

Did anyone try to develop something like that?

 

[Andrew Godwin]

I haven't seen anything like that personally, but I also don't see all of the Channels stuff going on, so maybe there is one.

It would be relatively easy to implement as a single class-based consumer that dispatches to RPC handlers based on method name, though, as it matches the consumer pattern very well.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/f8fdc9b4-ef54-4e39-ad66-9f5b4d9eb042%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fabien Millerand]

Thanks a lot for your answer Andrew.

On a side note, would you be related to Mike Godwin? I just draw some moustaches on Trump's face 5 minutes before seeing your message... It is disturbing...

Le jeudi 19 janvier 2017 19:27:10 UTC+1, Andrew Godwin a écrit :

I haven't seen anything like that personally, but I also don't see all of the Channels stuff going on, so maybe there is one.

It would be relatively easy to implement as a single class-based consumer that dispatches to RPC handlers based on method name, though, as it matches the consumer pattern very well.

Andrew

On Thu, Jan 19, 2017 at 7:34 AM, Fabien Millerand <mill...@gmail.com> wrote:

Hi everyone,

I am looking to implement a websocket server based on Django using JSON-RPC protocol.

I have been looking around for a pre-made solution without success. I am also a newbie in Django so I am a little bit lost...

Did anyone try to develop something like that?

 

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

[Andrew Godwin]

On Fri, Jan 20, 2017 at 2:36 AM, Fabien Millerand <mill...@gmail.com> wrote:

Thanks a lot for your answer Andrew.

If you do end up implementing something, be sure to add it to the links page in the Channels docs (just submit a PR with the change)

 

On a side note, would you be related to Mike Godwin? I just draw some moustaches on Trump's face 5 minutes before seeing your message... It is disturbing...

Not as far as I'm aware, though I'm sure there's some common ancestry somewhere. 

Andrew

[Alexander Prokhorov]

If you are going to implement JSON-RPC based on Channels, I would be happy to participate, I suppose we will start doing this in a few week anyway.

пятница, 20 января 2017 г., 13:36:49 UTC+3 пользователь Fabien Millerand написал:

[Fabien Millerand]

Hi,

I will start implementing something this week (starting today).

I will let you know how things go.

Alexander, I would be happy to share my work and get some feedback/help on this project. Let me know your github username and we can colaborate

Cheers

Fab

[Alexander Prokhorov]

My github name is “prokher”. Would be happy to help.

вторник, 24 января 2017 г., 11:00:06 UTC+3 пользователь Fabien Millerand написал:

[Fabien Millerand]

I just read that on the "Docs":

One thing channels do not do, however, is guarantee delivery. If you need certainty that tasks will complete, use a system designed for this with retries and persistence (e.g. Celery), or alternatively make a management command that checks for completion and re-submits a message to the channel if nothing is completed (rolling your own retry logic, essentially).

....

[Fabien Millerand]

alexander, I uploaded some ideas.

I think our best call is to extend WebsocketConsumer with some basic Json-RPC rules. I reused some code from this module:

https://pypi.python.org/pypi/jsonrpc2

See jsonrpc_test branch on repo. If we wrap it up nicely it should be fairly easy.

The last step is to create a decorator for the consumer's method so it can send back the "Method Not Found" error if needed.

Andrew, if you have some ideas or input on that, it would be much appreciated.

Cheers,

Fabien

[Andrew Godwin]

Hi Fabien,

My basic idea for an outline would be:

* A class than inherits from JsonWebsocketConsumer

* That class' receive() method decodes incoming messages, finds a handler function, and then calls that, taking the value it returns and sending it over the reply channel

* The user of it would then subclass again and implement JSON-RPC methods either as methods on the class itself, or using a dict in the class body to map names of methods to callables (your call on that one)

The non-delivery thing mentioned is the alternative you'd want, here, too - the alternative to non-guaranteed delivery (at most once) is guaranteed-and-possible-duplicate delivery (at least once), which for RPC systems generally isn't good - in RPC, it's easy to detect if a call didn't work as you won't get a response within a certain timeout.

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/f52130e1-4ec4-48b7-b903-cadf22fb6d70%40googlegroups.com.

[Fabien Millerand]

* A class than inherits from JsonWebsocketConsumer

That is what I started to do.

 

* That class' receive() method decodes incoming messages, finds a handler function, and then calls that, taking the value it returns and sending it over the reply channel

My thoughts as well. Also RPC protocol must implement "results" but also "errors" which I plan to implement by catching exceptions.

 

* The user of it would then subclass again and implement JSON-RPC methods either as methods on the class itself, or using a dict in the class body to map names of methods to callables (your call on that one)

I was thinking about creating a decorator @channel_rpc for the methods. Those would take either their "name" or an optional argument passed to the decorator.

 

The non-delivery thing mentioned is the alternative you'd want, here, too - the alternative to non-guaranteed delivery (at most once) is guaranteed-and-possible-duplicate delivery (at least once), which for RPC systems generally isn't good - in RPC, it's easy to detect if a call didn't work as you won't get a response within a certain timeout.

I am not sure to understand. In which case can there be messages/frames lost?! Where does that happen? Between the server interface and the Django layer? I would need to know more about that... Otherwise I might need to move with uWSGI or something.... JSON-RPC in itself doesn't implement a timeout, althought the javascript client better have one...

 

Andrew

[Andrew Godwin]

I am not sure to understand. In which case can there be messages/frames lost?! Where does that happen? Between the server interface and the Django layer? I would need to know more about that... Otherwise I might need to move with uWSGI or something.... JSON-RPC in itself doesn't implement a timeout, althought the javascript client better have one...

It simply means that it's possible that you might lose an incoming frame. This is also true of implementing it in uWSGI (the process handling the socket might get OOM killed, or the server might die, etc.)

It's not a normal case, it's just that if something super bad happens, the resulting handling is to drop a message rather than play it twice. Most systems I know of that handle websockets do this.

Andrew 

[Fabien Millerand]

Ok, I start to understand now.

To be frank the docs are a bit alarming :) 

[Andrew Godwin]

Yes, it's a bit alarmist if you don't come from the background of writing distributed systems. I just don't like to hide the truth one bit!

All your software and hardware can fail in myriad ways; I have a talk I need to give about it at some point. Knowing how it fails is half the battle!

Andrew

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/97e75375-6caf-4e8d-a781-be6da421840d%40googlegroups.com.

[Fabien Millerand]

Fair enough. I understand that in distributed system. But maybe you should add a note about that, as if the whole system is not distributed over network(s), it is highly unlikely to lose frames :D

You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/b00Ie8wBPnc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAFwN1uoPw%2BTjz3dmTvLeOck%3DHfRfRycA%3DHZ_GQa%2BbYBt7oXxwA%40mail.gmail.com.

[Alexander Prokhorov]

Colleagues,

you are really fast :) How can I help you? For our project we will definitely need a JavaScript client. Quick googling led me to https://github.com/JsCommunity/jsonrpc-websocket-client it does not look mature, but such client is quite easy to implement. Do you have plans to include simple JavaScript client to the package? If so I could try to start doing one right now.

среда, 25 января 2017 г., 11:57:35 UTC+3 пользователь Fabien Millerand написал:

Fair enough. I understand that in distributed system. But maybe you should add a note about that, as if the whole system is not distributed over network(s), it is highly unlikely to lose frames :D

On 25 Jan 2017, at 09:08, Andrew Godwin <and...@aeracode.org> wrote:

Yes, it's a bit alarmist if you don't come from the background of writing distributed systems. I just don't like to hide the truth one bit!

All your software and hardware can fail in myriad ways; I have a talk I need to give about it at some point. Knowing how it fails is half the battle!

Andrew

On Wed, Jan 25, 2017 at 12:06 AM, Fabien Millerand <mill...@gmail.com> wrote:

Ok, I start to understand now.

To be frank the docs are a bit alarming :) 

Le mercredi 25 janvier 2017 08:47:06 UTC+1, Andrew Godwin a écrit :

I am not sure to understand. In which case can there be messages/frames lost?! Where does that happen? Between the server interface and the Django layer? I would need to know more about that... Otherwise I might need to move with uWSGI or something.... JSON-RPC in itself doesn't implement a timeout, althought the javascript client better have one...

It simply means that it's possible that you might lose an incoming frame. This is also true of implementing it in uWSGI (the process handling the socket might get OOM killed, or the server might die, etc.)

It's not a normal case, it's just that if something super bad happens, the resulting handling is to drop a message rather than play it twice. Most systems I know of that handle websockets do this.

Andrew 

-- 

You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/97e75375-6caf-4e8d-a781-be6da421840d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fabien Millerand]

Alexander,

What I am working on is not the client, but the server...

I am not really sure what you are looking for exactly, but I have been working with websockets and json-rpc client-server structure for a few years now, so happy to fill you in with some infos if needed.

[Fabien Millerand]

Andrew,

I have finished to develop what I called the JsonRpcWebsocketConsumer:

https://github.com/millerf/django-channels-jsonrpc/tree/master/django_channels_jsonrpc/django_channels_jsonrpc

I was thinking of creating a pypy package, there is a little bit f more work to be done for that. But if you want it for your next release it is pretty much standalone. There is an example provided and plenty of tests.

Let me know what you guys think, and if you see anything to be modified/added.

Cheers.

Le mercredi 25 janvier 2017 09:09:48 UTC+1, Andrew Godwin a écrit :

Yes, it's a bit alarmist if you don't come from the background of writing distributed systems. I just don't like to hide the truth one bit!

All your software and hardware can fail in myriad ways; I have a talk I need to give about it at some point. Knowing how it fails is half the battle!

Andrew

On Wed, Jan 25, 2017 at 12:06 AM, Fabien Millerand <mill...@gmail.com> wrote:

Ok, I start to understand now.

To be frank the docs are a bit alarming :) 

Le mercredi 25 janvier 2017 08:47:06 UTC+1, Andrew Godwin a écrit :

I am not sure to understand. In which case can there be messages/frames lost?! Where does that happen? Between the server interface and the Django layer? I would need to know more about that... Otherwise I might need to move with uWSGI or something.... JSON-RPC in itself doesn't implement a timeout, althought the javascript client better have one...

It simply means that it's possible that you might lose an incoming frame. This is also true of implementing it in uWSGI (the process handling the socket might get OOM killed, or the server might die, etc.)

It's not a normal case, it's just that if something super bad happens, the resulting handling is to drop a message rather than play it twice. Most systems I know of that handle websockets do this.

Andrew 

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

[Andrew Godwin]

A quick read through and it looks roughly how I expect; I'm not an expert in JSON-RPC though so I'd want some other people to chime in.

As for releasing it on PyPI, I think that's the best way; I don't want to roll something like this into Channels directly. I would suggest you turn it from a Django app into a simple Python module though, so you can just do:

from channels_jsonrpc import JsonRpcWebsocketConsumer

class MyConsumer(JsonRpcWebsocketConsumer):

   ....

In your own consumers file.

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/49dbc038-46e5-402c-a810-900d47a561bf%40googlegroups.com.

[Fabien Millerand]

Yes, that was my thought as well...

Will do and let you know. It could be good to add a link in your docs  as a side note...

If you get any feedback let me know...

You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/b00Ie8wBPnc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAFwN1upCc96mWBP7ZCZm9y26ZJu7YRL7Q%3DvrxXOL43xx5Zc1gA%40mail.gmail.com.

[Alexander Prokhorov]

Dear Colleagues,

I've made some fixes in the code (most of them concerns Python 3 compatibility). I also added a test showing a problem with some kind of name clash.

пятница, 27 января 2017 г., 1:07:38 UTC+3 пользователь Fabien Millerand написал:

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/49dbc038-46e5-402c-a810-900d47a561bf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/b00Ie8wBPnc/unsubscribe.

To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

[Fabien Millerand]

Thanks Alex.

I also rectified something that your test brought up.

I am modifying the structure at the moment to have a nice PyPI package.

I also remove the ability to push to master. Please make changes in a new branch and submit a push request next time ;)

But thanks anyway! I hope it will help in your project as well!

[Fabien Millerand]

Ok, module is uploaded

https://pypi.python.org/pypi/django-channels-jsonrpc/0.1

Feel free to use it.