MOSS Award to Django

[Andrew Godwin]

Hi everyone,

For those who haven't seen, Mozilla has awarded $150,000 to Django for work on Channels and request/response improvements lifted from REST Framework. More in the blog post here: https://www.djangoproject.com/weblog/2015/dec/11/django-awarded-moss-grant/

I'll be coordinating this effort for the most part, but we're still working out who to fund and the roadmap for the project, as well as work out how we can pay people for their work on a different scale to the Django Fellowship, so it might take a bit of time!

I'll be back on here with some questions for people to discuss/answer about the channels design at some point soon, but a lot of the basic design is already up at http://channels.readthedocs.org, so take a read over that if you're interested.

What I can say is that my intention is to both bake Channels into the next major release of Django, as well as hopefully release a pluggable app version that will run on 1.8 and 1.9 - I have some plans around how to do that effectively, but they involve hitherto unforged paths around Django and how we package dependencies, so I can't say we'll get there 100%.

Andrew

[Asif Saifuddin]

Awesome !!!! awesome !!! awesome!!! :D :D

[Daniel Chimeno]

Hi,

Just saying congrats to people involved and the community. Great step for Django!

[Ibrahim K]

great!!!

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/bbeea75a-1026-42f1-9cb4-56be0c8a08b2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards,

Ibrahim.k

[Mitar]

Hi!

This is really awesome news! Can I suggest to use WebRTC data channel
instead of Websockets?

I think we should make a paradigm shift and move to use WebRTC instead
of Websockets. This allows many interesting properties:
* connections can be based on SCTP
* have multi-path routing
* you can have data packets which are reliable or unreliable
* you can even have backpressure support (imagine that you could push
back on Meteor publish because client cannot handle observes as
quickly as server is pushing)

There is a great open source project already available which provides
both client server suppor: https://github.com/feross/simple-peer

Some interesting links to read:
* https://github.com/webrtcftw/goals/issues/1
* https://github.com/jbenet/random-ideas/issues/13

I opened a ticket as well: https://github.com/andrewgodwin/channels/issues/38


Mitar

> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.

> Visit this group at http://groups.google.com/group/django-developers.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-developers/CAFwN1upjXf2LLk2gGA9mzdPQn4DbWehSHNFWhF-GkEk4q54gHA%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.



--

http://mitar.tnode.com/
https://twitter.com/mitar_m

[Aymeric Augustin]

Hello,

Unless I missed something, channels aren’t about websockets. They add asynchronous processing capabilities to Django. They aim at being compatible with any bidirectional protocol.

Django should support common protocols out of the box and document a public API for third-party projects to hook to and support additional protocols.

At this point, I’m not convinced that simple-peer is sufficiently mainstream to be the recommended solution for Django websites.

Best regards,

--
Aymeric.

> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAKLmikNtJhHQntNaSWvyeiz4qhVunM4OZ%2BJM-9v4VpuArmfsdQ%40mail.gmail.com.

[Mitar]

Hi!

On Sat, Dec 12, 2015 at 6:29 AM, Aymeric Augustin
<aymeric....@polytechnique.org> wrote:
> Unless I missed something, channels aren’t about websockets. They add asynchronous processing capabilities to Django. They aim at being compatible with any bidirectional protocol.

Yes, but currently, that "bidirectional protocol" is "bidirectional
protocol without backpressure" which makes it really useless in any
serious production use with scaling. It is interesting for smaller toy
use cases, but without backpressure you simply cannot handle
congestion and your systems blow in your user's face. node.js has
learned this the hard way by reimplementing their streaming API
multiple times.

What I would propose is that Django officially exposes channels API
with backpressure and then for simpler projects (or one where you do
not care about delivery) one can use Websockets, but otherwise for
production use for heavy loads WebRTC data channels with backpressure
should be provided.

> At this point, I’m not convinced that simple-peer is sufficiently mainstream to be the recommended solution for Django websites.

No, that is just an example of the node.js module which provides
server-side WebRTC support.

And yes, this is something which is cutting edge. I would be
completely OK by supporting Websockets now on the transport level, but
the API level should understand and support backpressure. Otherwise we
will have issues in the future once this feature gets widely deployed
and started using in production.


Mitar

[Aymeric Augustin]

> Le 12 déc. 2015 à 23:10, Mitar <mmi...@gmail.com> a écrit :
>
> I would be
> completely OK by supporting Websockets now on the transport level, but
> the API level should understand and support backpressure.

Hello Mitar,

Yes, I fully agree with this way to frame the topic.

I'm don't know how this would manifest in the core API nor how much thought Andrew has given to this issue, but I'm interested in making sure we get it right. Thanks for bringing it up!

--
Aymeric.

[bliy...@rentlytics.com]

Hey,

Channels sounds a lot like celery (or celery sounds like part of channels).  Is that a fair read?  Looking for tighter REST integration either way.

Thanks,
-Ben

[Aymeric Augustin]

Hi Ben,

Celery and channels don’t tackle the same problem.

Celery is an asynchronous task queue. It is designed perform expensive work after responding to a HTTP request. At that point there is no possible communication with the browser.

Channels is an asynchronous message framework. It enables asynchronous messaging between the browser and Django, going beyond the current request-response structure.

Best regards,

-- 

Aymeric.

-- 
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/299ede52-6ae4-46d3-8667-40e7fa6a89a6%40googlegroups.com.

[Ben Liyanage]

So a channel must result in a response to a browser?  It seems both from the config (routing things into functions, or queues) and the underlying tech (using redis or whatever) sounds very similar to celery.

From the docs:

• Interface servers, which communicate between Django and the outside world. This includes a WSGI adapter as well as a separate WebSocket server - we’ll cover this later.
• The channel backend, which is a combination of pluggable Python code and a datastore (a database, or Redis) and responsible for transporting messages.
• The workers, that listen on all relevant channels and run consumer code when a message is ready.

I get that the goal of this is for asynchronous web requests, but if it's generalized right it seems like it could cover doing any kind of work without the pressure of completing a web transaction in a timely fashion.

-Ben

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/3pTMfHa8SFE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/D5557CBB-03D8-4DB2-AAFA-B4063E227D8A%40polytechnique.org.

For more options, visit https://groups.google.com/d/optout.

--

Ben Liyanage | Software Engineer | Rentlytics, Inc.

Phone: (410) 336-2464 | Email: bliy...@rentlytics.com
1132 Howard Street, San Francisco CA 94107

Visit our Website | Watch our Video

[Shai Berger]

On Wednesday 16 December 2015 01:07:16 Ben Liyanage wrote:
>
> I get that the goal of this is for asynchronous web requests, but if it's
> generalized right it seems like it could cover doing any kind of work
> without the pressure of completing a web transaction in a timely fashion.
>

No, because Channels are designed to be non-reliable (each message is
delivered at most once), and if I understand correctly this is a key point in
the design. Celery gives you much stronger delivery guarantees,

Shai.

[Ben Liyanage]

And, beyond that, there are plenty of non-critical tasks that applications could easily offload until after a response has been sent - like saving things into a cache or thumbnailing newly-uploaded images.

http://channels.readthedocs.org/en/latest/concepts.html#concepts

I mean this example sounds like celery to me.  You don't want an image to maybe have a thumbnail generated.  And you have no return value to the client when the thumbnail is generated.

-Ben

[Curtis Maloney]

On 16/12/15 12:05, Ben Liyanage wrote:
> And, beyond that, there are plenty of non-critical tasks that
> applications could easily offload until after a response has been
> sent - like saving things into a cache or thumbnailing
> newly-uploaded images.
>
> http://channels.readthedocs.org/en/latest/concepts.html#concepts
>
> I mean this example sounds like celery to me. You don't want an image
> to maybe have a thumbnail generated. And you have no return value to
> the client when the thumbnail is generated.

There are certainly some cases, AIUI, where channels can obviate the
need for Celery... like sending emails, or thumbnailing uploaded image,
and so on.

However, given the different reliability profiles, there's still room
for both.

--
Curtis

[Markus Holtermann]

From my point of view the major non-functional difference is the interaction with a client in *some way*. While a celery task runs without any form of client, a channels method would mostly do that. Picture the following example:

A user uploads an image. This could happen through the common request-response cycle. When the upload is done the server creates a celery task to generate a bunch of thumbnails and returns a http response. At this point the image is already available online but the thumbnails aren't. Every client could start a websocket connection through channels to get notified once a thumbnail is available. This notification is send through a server side callback called by the celery task.

/Markus

[Shai Berger]

On Wednesday 16 December 2015 08:08:59 Markus Holtermann wrote:
> From my point of view the major non-functional difference is the
> interaction with a client in *some way*. While a celery task runs without
> any form of client, a channels method would mostly do that. Picture the
> following example:
>
> A user uploads an image. This could happen through the common
> request-response cycle. When the upload is done the server creates a
> celery task to generate a bunch of thumbnails and returns a http response.

But here is Ben's point: Why would you introduce an additional moving part
here (Celery), when the Channels documentation specifically mentions this as a
use-case?

If I get it right -- Curtis' description is spot-on; some tasks will still
need Celery, Channels will take care of many others.

Shai.

[Andrew Godwin]

Channels is indeed a replacement for Celery if you're OK with not having delays, retry, or the other advanced options Celery has - Channels just requires an event to respond to, and this can be triggered by the user during a view much like a Celery task.

But yes, channels deliberately has an at-most-once delivery style, which will work for operations that aren't critical, or if people build their own retry logic (but at that point, maybe use Celery).

Andrew

[Markus Holtermann]

On December 16, 2015 6:01:58 PM GMT+10:30, Shai Berger <sh...@platonix.com> wrote:
>On Wednesday 16 December 2015 08:08:59 Markus Holtermann wrote:
>> From my point of view the major non-functional difference is the
>> interaction with a client in *some way*. While a celery task runs
>without
>> any form of client, a channels method would mostly do that. Picture
>the
>> following example:
>>
>> A user uploads an image. This could happen through the common
>> request-response cycle. When the upload is done the server creates a
>> celery task to generate a bunch of thumbnails and returns a http
>response.
>
>But here is Ben's point: Why would you introduce an additional moving
>part
>here (Celery), when the Channels documentation specifically mentions
>this as a
>use-case?

Because my "generate a thumbnail" might me some video decoding or another long running process that would block possible further requests, or simply because I could easier reuse that particular task in other parts of the website as well.

>If I get it right -- Curtis' description is spot-on; some tasks will
>still
>need Celery, Channels will take care of many others.

For me it's more a question of "is a client directly involved".

/Markus

[Andrew Godwin]

On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann <in...@markusholtermann.eu> wrote:

>If I get it right -- Curtis' description is spot-on; some tasks will
>still
>need Celery, Channels will take care of many others.

For me it's more a question of "is a client directly involved".

I don't agree; I think it's more "do you have enough workers to handle this". Channels makes all workers serve all requests, so if you're doing e.g. video encoding you can't segregate it from request handling. However, you could have multiple channel backends and thus multiple worker pools to achieve this.

Andrew 

[Javier Guerra Giraldez]

On Wed, Dec 16, 2015 at 6:22 AM, Andrew Godwin <and...@aeracode.org> wrote:
>
>
> On Wed, Dec 16, 2015 at 9:52 AM, Markus Holtermann
> <in...@markusholtermann.eu> wrote:
>>
>>
>> >If I get it right -- Curtis' description is spot-on; some tasks will
>> >still
>> >need Celery, Channels will take care of many others.
>>
>> For me it's more a question of "is a client directly involved".

as i read it:

celery has an specific use pattern: the Django process emits a
message to some other process. It doesn't alter the request/response
cycle in any way. the most common use is to initiate a 'background
process' from the request-bound Django process to another, non-Django
process.


The "channels" design makes Django a processor of messages, the first
example is a 'normal' http request, which would be just a message from
the user. another example is asynchronous request/responses,
typically WebSockets, SSE, etc. finally, it could also allow a
Django process on the receiving end of an intra-server message. A
future version of Celery could use that as a new infrastructure
option.


--
Javier

[Samuel Bishop]

I'm uncertain how popular the suggestion will be but ... are "channels" the right solution to our async/concurrency problems? ( I'm all for django-channels the project and the work to solve these issues)

All the talk of channels, workers, queues and back ends in the explanations I'm reading as I follow along with the progress of 'Django Channels", just feels "less good" than solving this problem via Actor Model Concurrency (championed by Erlang, Elixir, Stackless Python, etc). 

There is an excellent actor model concurrency library for python that already supports 'async django'. "Pulsar" (http://pythonhosted.org/pulsar/index.html) and the documentation here http://pythonhosted.org/pulsar/apps/pulse.html shows two different ways to use actor model concurrency to attack the issue of using Django with asynchronous web stuff like Websockets, etc.

Some of what's being described in the explanations of Django channels sounds very similar to what they provide to work around the limitations of django's blocking request processing cycle other parts sound much more complicated, for instance the need for various back ends in-memory/redis/etc.

I know Pulsar is currently Python 3.4 and newer only, but in the past it supported 2.7, so anything we can learn from their concurrency approach can work on Python 2.7 to maintain our backwards compatibility going forward. You can see an example of Pulsar using python 2 if you look on the benchmark page for their python powered 'redis clone' - https://gist.github.com/lsbardel/8068579 - One of the benchmarks is using pypy so the code definitely worked in a python 2.7 environment.

Would the work to integrate channels as part of Django improve the ability to run Django in a completely async mode and enhance its performance when run under Pulsar? or would it just be something that needed to be turned off if we wanted to use something like Pulsar to power our "async django". Is there anything to be gained from using a more actor driven concurrency model or adopting any of the other methods used in their 'pulse' django app?

I suppose I'm just trying to work out how to compare using Django-Channels with using Pulse from the Pulsar project.

[Andrew Godwin]

Hi Samuel,

You'll notice if you look carefully that I avoid saying "async" almost anywhere in the Channels announcements or documentation, and when I do, it's qualified very carefully. Channels makes Django event-driven, but it does not provide full "in-process" async like Twisted, asyncio, or something like Pulsar.

Channels is actually kind of related to the actor model in some ways, but it's simpler; imagine it as a pool of identical actors that are all stateless, apart from the interface servers, which have the incoming socket and thus the state.

The key thing is that Channels does nothing to make Django asynchronous. All the Django code will remain synchronous (the ORM, templating, etc.), and run in synchronous processes that just process one message after the next. The only part of the system that will require an async framework is the interface servers, as to be anywhere near efficient they need to serve more than one socket at once; right now we're using Twisted and asyncio for those, purely as they're incredibly mature and well-supported by the community, but you could swap out anything there that achieved the same goal (greenlets, for example, though I personally dislike their implicitness).

Channels essentially takes the worker-prefork model that's been used in webservers for years and applies it to a lower-level message concept rather than HTTP requests (http requests are just one kind of message). This not only has the advantage of a lot of previous work and research into architecting and scaling these kinds of systems, it also means the actual amount of work in core Django isn't crazy; it's more about providing nice places to wrap up the worker loop and nice utilities around authentication and session management.

If you read through the docs there's a bit more about this, but basically, the fundamental design goal here is that Channels be no harder to use than normal Django; indeed, it's almost exactly the same view concept, but instead of a request you get a message, and the newer developers I've interviewed about it seem to be comfortable with the extension of the current abstraction straight away. I don't ever want people to have to write asynchronous code if they don't want to; it's currently so easy to shoot yourself in the foot in almost any Python async solution that I don't think that makes for a good developer experience.

The "complexity" of the message backends you describe is because they're network-transparent; there is indeed a simple, in-process one that uses threads too, but the whole point of this design is that you run Django over multiple processes on many machines, and in order to get something like that you need a place to funnel messages through (another potential backend design could use direct connection and process discovery, perhaps, or an Erlang-like process spawn-and-track design with an overarching controller process with access to many pre-selected machines).

The idea of making an asynchronous Django is tangential to making Channels; this is more about changing the design abstraction of Django so we can fit the concept in, and then providing a neat solution that drops into the gap. There's nothing stopping someone making the worker processes for Channels async-aware as well, so they can run multiple message consumers at once, but that's such a complex and risk-fraught task that I don't think it's worth it when we can get 90% of the benefits with a much simpler and easier-to-design-and-understand system. If you want to see an example of actual async Django, see Amber's talk from Django Under the Hood this year, where she managed to get it working decently well under Twisted.

I expect Channels to smooth out Django's performance mostly as a side-effect of the worker model providing demand-based load balancing by design, but I doubt it will be a huge increase until people start using the primitives it provides to offload tasks to other workers.

I hope that somewhat long-winded explanation helps - if not, let me know what's still unclear and I'll try to tackle it. Clearing up what we're trying to do and communicating it is important to me; I want to make sure everyone knows so we can get good feedback on this stuff. For what it's worth, I did consider the actor model along with several other kinds of concurrency approaches (I've been prototyping bits of this for years); this approach ended up being the nicest design and the one I have the most confidence in that we can scale and shard to large work volumes.

Andrew

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1d6526d3-dd55-4719-b1ba-4804d4eb2dff%40googlegroups.com.

[Samuel Bishop]

Hi Andrew, 

Thanks for the detailed response. I'm glad to hear you have tried a few different ways to implement this, and having read through your reply, and the channels docs again, I have more questions :-)  

Your point about asynchronous code introducing needless complexity is very well founded, its one of the reasons I like Actor Model Concurrency over other concurrency models, it scales well computationally, and *mentally*, as a programmer, I am trying break down my real world goals into discrete steps of computations, comparisons, iterations and by combining these steps, I construct my program. Actor Model Concurrency, is no more complicated than saying "This step can be done without affecting anything else thats going on", which makes the 'actor/task' much easier to introduce to existing synchronous code than most other concurrency methods. 

One thing that sticks with me as I think about the current channels design, relates to the idea I've raised in another discussion ( https://groups.google.com/forum/#!topic/django-developers/-BOsuVdPFXc ).

If I put my architect hat on and think about the future a bit, a 'more' actor based design to channels might simplify later additional async work. Since all the websocket stuff is new, I'll stick to the existing HTTP cycle in this example. In version 1 a 'vanilla' Django task/actor would be a full HTTP request response cycle, like how Django and WSGI currently work. Our Django task is effectively our WSGI task, much like Pulsar Pulse. If all of Django is contained inside our new task/worker/supervisor Actor layer, then it becomes fairly trivial to do things like create separate request handler, database query, and template rendering task/actors and chain their calls. Since in the previously linked to discussion thread, I'm talking about decoupling the ORM from the 'upper' layers of Django, its hard to not also think what opportunities that would open up for async efforts. How do you see the architecture of Django channels accommodating such improvements in future? At the moment your focused on keeping things compatible and avoiding the need for any of the existing synchronous Django code to change, is this introducing any design tradeoffs that you are aware of with regards to hypothetical future developments?

Django Channels feels like it contains a possible asynchronous replacement for signals but is 'holding back', many cases where I can see myself using a channel, I'm using them with a signal handler, can you elaborate a bit on your thoughts here? I assume its a compatibility decision, but is there more to it than just that? 

Regards

Sam

[Andrew Godwin]

On Mon, Dec 21, 2015 at 6:05 AM, Samuel Bishop <lucra...@gmail.com> wrote:

Hi Andrew, 

Thanks for the detailed response. I'm glad to hear you have tried a few different ways to implement this, and having read through your reply, and the channels docs again, I have more questions :-)  

Your point about asynchronous code introducing needless complexity is very well founded, its one of the reasons I like Actor Model Concurrency over other concurrency models, it scales well computationally, and *mentally*, as a programmer, I am trying break down my real world goals into discrete steps of computations, comparisons, iterations and by combining these steps, I construct my program. Actor Model Concurrency, is no more complicated than saying "This step can be done without affecting anything else thats going on", which makes the 'actor/task' much easier to introduce to existing synchronous code than most other concurrency methods.

I agree - I'm a big fan of actors in general and it's usually what I turn to to write complex simulations when I'm mucking around with game backends.

 

One thing that sticks with me as I think about the current channels design, relates to the idea I've raised in another discussion ( https://groups.google.com/forum/#!topic/django-developers/-BOsuVdPFXc ).

If I put my architect hat on and think about the future a bit, a 'more' actor based design to channels might simplify later additional async work. Since all the websocket stuff is new, I'll stick to the existing HTTP cycle in this example. In version 1 a 'vanilla' Django task/actor would be a full HTTP request response cycle, like how Django and WSGI currently work. Our Django task is effectively our WSGI task, much like Pulsar Pulse. If all of Django is contained inside our new task/worker/supervisor Actor layer, then it becomes fairly trivial to do things like create separate request handler, database query, and template rendering task/actors and chain their calls. Since in the previously linked to discussion thread, I'm talking about decoupling the ORM from the 'upper' layers of Django, its hard to not also think what opportunities that would open up for async efforts. How do you see the architecture of Django channels accommodating such improvements in future? At the moment your focused on keeping things compatible and avoiding the need for any of the existing synchronous Django code to change, is this introducing any design tradeoffs that you are aware of with regards to hypothetical future developments?

So, what you propose obviously isn't feasible in the short term - I'm doing as much as I can to decouple Channels from "rewrite all of Django" - but notably, channels is a subset of the actor model. In particular, you could see consumers as stateless, single-action actors - but, this means that any future abstraction would be able to build on that.

 

Django Channels feels like it contains a possible asynchronous replacement for signals but is 'holding back', many cases where I can see myself using a channel, I'm using them with a signal handler, can you elaborate a bit on your thoughts here? I assume its a compatibility decision, but is there more to it than just that? 

It's entirely a compatibility decision - Channels needs to ship in a way where code written for 1.9 will work the same, barring a few edge cases where people dip down to raw WSGI.

You can entirely replace signals with custom channels, and I want to ship example code that does so (it's two lines of code, pretty much), but they work in a different way - for example, right now the pre_save signal can actually prevent a save from occurring, whereas if it were on a channel we'd have to then link the save consumer up to be run as a result of pre_save, which starts down a path of compatibility nightmares.

Channels should be flexible enough that if you want to pursue the actor pattern you can - the underlying library basically just has send() and receive_many([channel_name, ...]) methods, so you can see how that could be used to make different task-oriented processes rather than the generic worker model that's going in now - it's just a case of keeping a major addition like this as simple and backwards-compatible as possible. Rewriting everything from scratch would be lovely, but unfortunately part of maintaining a framework is not doing that.

Part of me, though, would be very interested in such a project being undertaken to see how far it could go, though I'm not sure you could then call the end result Django - at least, not without getting there incrementally over many releases.

Andrew

 

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/a827b7de-d873-4f31-b153-1fa39f88386e%40googlegroups.com.

[Samuel Bishop]

Once again I have to thank you for taking the time to keep replying. These longer explanations are proving very helpful (at least to me) in understanding Django Channels. :-)

I definitely understand the compatibility reasons, I'm also pleased to see that signalling via a channel will be as simple as I had hoped, at least simple as long as we put aside compatibility issues with Django's built in signals. I'd still be pleased with only the ability to make my custom signals asynchronous.

Digging into the implementation details, you've sparked my curiosity. For some reason the simplicity of the having just send() and receive_many() didn't fully dawn on me until now. Once I've made progress with my decoupling, I think I'll have to try experiment with cleaving the major parts of the entire request/response cycle into a chain of asynchronous actors.

I'll likely test how hard it is to 'wrap' each part of the cycle in a way that allows it to be called by synchronously or asynchronous code. I'd like to avoid creating "yet another async django, that isnt really django anymore", keep the existing synchronous code pathway seems like a good start at a "useful experiment".

As for 'could you call the end result Django'... I think "Django" is whatever we make of it ;-) 

I don't see why we wont still call the combination of all the parts, the ecosystem of compatible libraries, etc, "Django", regardless of how much we might change between now and far future versions like 4.0 or 9.0. 

Regards

Sam

[Andrew Godwin]

On Sun, Dec 27, 2015 at 7:53 AM, Samuel Bishop <lucra...@gmail.com> wrote:

Once again I have to thank you for taking the time to keep replying. These longer explanations are proving very helpful (at least to me) in understanding Django Channels. :-)

I definitely understand the compatibility reasons, I'm also pleased to see that signalling via a channel will be as simple as I had hoped, at least simple as long as we put aside compatibility issues with Django's built in signals. I'd still be pleased with only the ability to make my custom signals asynchronous.

Yes, the built-in signals have to remain as they are now; it is possible we could hook something into the existing Signal code to provide an option to run them via Channels, but I think the API to just make a new channel and send events to it is actually easier and better anyway.

(I've never really liked the implicitness of channels, and the "come-from" rather than "go-to" nature so you can't tell what code a signal invocation might send without searching, whereas channel registrations are explicit in a big dict)

 

Digging into the implementation details, you've sparked my curiosity. For some reason the simplicity of the having just send() and receive_many() didn't fully dawn on me until now. Once I've made progress with my decoupling, I think I'll have to try experiment with cleaving the major parts of the entire request/response cycle into a chain of asynchronous actors.

Yup; in fact, I'm working on a spec, alongside development efforts, to exactly lay out what the low-level API will be that everything runs on, which is currently nicknamed "ASGI": http://channels.readthedocs.org/en/latest/asgi.html

Django will run directly on top of that, with a small bit of code that transforms send/receive_many into the documented consumer functions, but it means there's flexibility to use the low-level API more directly in some fashion if you want; since Channels is going to result in multiple projects all cross-communicating already, I figured having a spec would be Very Useful.

 

I'll likely test how hard it is to 'wrap' each part of the cycle in a way that allows it to be called by synchronously or asynchronous code. I'd like to avoid creating "yet another async django, that isnt really django anymore", keep the existing synchronous code pathway seems like a good start at a "useful experiment".

I am curious to see what you could achieve with more atomic breakdown of Django view processing; things like template rendering can definitely be done separately, while I'm less sure about the ORM (as you can't collect results from multiple channels at once in Channels for performance reasons; all you can do is listen for a single event on a single channel and react to it before you get the next event).

 

As for 'could you call the end result Django'... I think "Django" is whatever we make of it ;-) 

I don't see why we wont still call the combination of all the parts, the ecosystem of compatible libraries, etc, "Django", regardless of how much we might change between now and far future versions like 4.0 or 9.0. 

Well, "Django" is also something that's agreed upon by the community, the Technical Board, and keeps to its promises of ease of use, backwards compatibility, good documentation and so forth. Any radical change that might end up happening would have to happen in slow steps over many releases, and only with the buy-in of the community - I don't even fully have that yet, and I don't expect to get it until Channels can show that it works, is backwards-compatible, and solves issues without making anything worse.

 

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/cdf23837-6d82-418a-a034-aec2b6d2b1c2%40googlegroups.com.