Django Async DEP

[Andrew Godwin]

Hello everyone,

While the ASGI patch (https://github.com/django/django/pull/11209) is maybe the first small step in a long road to async, it's the only real one we could do without a DEP as it purely pulls on existing specs and bugs.

To that end, I have drafted a DEP (provisionally #0009), "Async-capable Django". It is a summary - and I use that word very lightly - of the rest of the work to be done to make a version of Django that one could call "async capable".

You can view it here: https://github.com/andrewgodwin/deps/blob/async/draft/0009-async.rst

It is approximately 7,000 words long - while it could probably do with some editing, the subject matter means it has to be quite in-depth. Even then, it does not propose an exact plan to follow; instead, it proposes the overall strategy and the high-level ideas about what needs to be done.

If you are short on time, please read the Foreword and the High-Level Summary, and then jump around using the table of contents to the sections that interest you.

Feedback on this is very much appreciated; you can either reply here or, if you have comments that would benefit from precise location on the diff, use the pull request here: https://github.com/django/deps/pull/56

This is a very complex topic, and I appreciate feedback might take a while; at least a couple of weeks to get everyone's thoughts. If you want a more private channel to discuss concerns or design questions you're not happy to talk about publicly, feel free to email me directly.

This proposal does not quite fit the DEP template - we never really planned for something of this scale - but I have done my best to make it work; I think we still benefit from discussing and voting on it in the DEP format, but suggestions on how to make it work better are welcome, provided they fit within how Django operates as a community now.

Finally, if there is positive feedback on this, we have the separate question of funding it that I will discuss separately, if and when we get that far. The DEP covers some of the topics, but it's notable that the implementation speed of this varies considerably; if we could get, say, me and someone else on it full-time (which is not a cheap or likely prospect), we can probably be done inside of a single release. If we rely just on volunteer time, it may take years. There is hopefully somewhere between those two that works.

Thanks for taking the time to read through!

Andrew

[Patryk Zawadzki]

Hey Andrew,

Great work on the DEP, the task at hand is humongous.

Do you think it's worth it to try and make the ORM async? It contains tons of magic that is inherently incompatible with explicit I/O that is required for async/await to work, things like silently fetching relations on first access, silently fetching deferred fields, queryset laziness etc. Those need to be explicitly awaited now and having to await regular field access for the sake of making deferred fields work is (I believe) not worth it.

Conveniently, all of these magic properties are things that silently break performance of applications leading to the n+1 queries problem so I'm more than happy to get rid of them ;)

Seeing that we can't get rid of anything in the old synchronous ORM, one idea would be to just depend on a separate, asyncio-based ORM and configure it to use the same tables and relations that the old ORM uses. It could work like model forms or formset factories, give it a "classic" model class, a list of overrides (we can't hope to automatically match old field types to anything understood by another engine) and receive an asyncio-capable table. There's an obvious candidate that Tom Christie is working on.

We could even improve the querying syntax to use the SQL Alchemy's filtering operators with the hope of eventually reaching full static type safety[2] (one can only dream).

The new ORM would only be usable in new async views and other features could be implemented gradually. You don't need to async forms (but with Tom Christie's typesystem[3] you don't really need async forms) and async template loading to achieve great things.

You can already use things like requests-async[4] to access external resources, you can use libraries like Ariadne[5] to implement a GraphQL API with WebSocket-based subscriptions without having any of those (disclaimer: I work on Ariadne). It's okay if some things can't be ported initially or if certain features are only available in their sync flavor. The community will pave the way and provide many missing pieces before we get to implementing them. It's okay if new async counterparts have different APIs if these APIs are either better or are a closer match to how async/await works.

[1] https://github.com/encode/orm

[2] https://docs.sqlalchemy.org/en/13/core/tutorial.html#operators

[3] https://github.com/encode/typesystem

[4] https://github.com/encode/requests-async

[5] https://github.com/mirumee/ariadne

PS: If Django provided abstractions for common event loop operations such as "await this collections of futures" and "ensure this task is executed even if not explicitly awaited" then I think it may be possible to avoid explicitly depending on asyncio.

All the best

[J. Pic]

Hi Patryk,

I'm not sure but for me the "What is Django" section answers the question. For me Django is full of philosophy that seeds a great ecosystem of apps of all sorts with a growing user base nonetheless, and a bunch of brilliant hackers to look up to and inspire for more. Of course if you're into fixing a particular technical issue such as "I want to make an efficient GraphQL server", then even Django works even if it's not necessarily the most fit for X reason. But for "I want to get a web page going with some forms and a database and see where that goes" then Django is definitely enjoyable, and at this point in time where tons of more sophisticated frameworks are born it's clear that Django is still in the game and will be for the next 10 years. So, why not also get more features out of it while we've not been abandoned by all the talented contributors ? It's not because we like Django that we cant haz nice things ;)

Great move Andrew and very exciting, keep it up and best luck to you !

[J. Pic]

Oops too fast, if it's possible to split the DEP and delay the ORM as advised by Patrick it could make it a lot easier to distribute the work, I don't know really sry. Have a great day !

[Patryk Zawadzki]

I'm not sure but for me the "What is Django" section answers the question. For me Django is full of philosophy that seeds a great ecosystem of apps of all sorts with a growing user base nonetheless, and a bunch of brilliant hackers to look up to and inspire for more. Of course if you're into fixing a particular technical issue such as "I want to make an efficient GraphQL server", then even Django works even if it's not necessarily the most fit for X reason. But for "I want to get a web page going with some forms and a database and see where that goes" then Django is definitely enjoyable, and at this point in time where tons of more sophisticated frameworks are born it's clear that Django is still in the game and will be for the next 10 years. So, why not also get more features out of it while we've not been abandoned by all the talented contributors ? It's not because we like Django that we cant haz nice things ;)

I'm arguing the opposite, that by limiting the scope of the MVP to asynchronous views (even if middleware stays synchronous for now) and using an existing ORM we can make Django a viable solution for many new applications. And some of them don't need the full stack.

I'd go as far as arguing that fewer and fewer applications benefit from Django providing all of the pieces (as many new applications only implement an API and render in the client) so treating them all as blockers here may not be beneficial.

Best regards

[Andrew Godwin]

I would agree with both of you - I think the most important thing is to get the view layer async-capable, as that then lets sites use any manner of asynchronous libraries that already exist to get experiments and unique things going. Tom Christie, for example, has already started work on an asynchronous ORM. Some of Django's biggest sites don't use the ORM - for example, Instagram.

That said, I also think it's important to allow the ORM to support both modes in the long term. I truly believe the best way to be able to write async code is to _have the choice to write it_, rather than being made to all the time; if we make people use a separate, async ORM, then we force them to write every view asynchronously, with all the extra danger and thinking that requires. It's much better for Django to do the hard work, and say "hey, if you want to write asynchronously or synchronously, that's fine - it takes literally zero extra effort to go either way".

This is why I propose in the DEP that we do the view layer first, and then move onto the ORM as a second wave.

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/314d7b71-1bf1-441b-ae49-e4fbeb5e1891%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[J. Pic]

In one project I really enjoyed using channels just as a background worker instead of celery, and the DEP does not talk much in the Email section: use the async variant, can be tackled separately, low priority.

For now the ORM works, but if sending an email fails (ie. SMTP down) it will raise an exception that can end up in a 500 error, it would be great if Django could retry so that emails leave when its SMTP is up again, would that maybe weight in priority ?

Also, you would probably like to view the queue in the admin, maybe make an unmanaged Model for tasks. So, as simple as Email would look like, I think efforts in it could benefit in a much wider way too.

[Patryk Zawadzki]

That said, I also think it's important to allow the ORM to support both modes in the long term. I truly believe the best way to be able to write async code is to _have the choice to write it_, rather than being made to all the time; if we make people use a separate, async ORM, then we force them to write every view asynchronously, with all the extra danger and thinking that requires. It's much better for Django to do the hard work, and say "hey, if you want to write asynchronously or synchronously, that's fine - it takes literally zero extra effort to go either way".

Slightly off-topic but once we have an async ORM, making it synchronous is not impossible (I believe either Channels or Daphne already have shims that use a worker thread to spin the event loop until a future is fulfilled).

[Aymeric Augustin]

Hello Andrew,

Thanks for your work putting together this plan. Within our constraints, it's a good plan.

Regarding templating, I would say it isn't a priority because a developer who knows how to parallelize I/O bound operations will prefer (or at least accept) to perform these operations in the view, not in the template.

I'm on the fence about the convention for async APIs. I'm not super excited by spraying async code with _async prefixes. The namespacing approach would allow for cleaner async code. Most Python modules should be either sync or async, not a mix of both styles. But I might be underestimating the importance of explicitness...

I don't have much else to say on the DEP. It makes a lot of sense. I'm happy to see this happening!

Cheers,

-- 

Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1uqav3JXQA3m%2Bajf0Bd15QQX3JTZTfmH3Hc3ECWyRR3CVg%40mail.gmail.com.

[Andrew Godwin]

> Slightly off-topic but once we have an async ORM, making it synchronous is not impossible (I believe either Channels or Daphne already have shims that use a worker thread to spin the event loop until a future is fulfilled).

This is in fact the proposed way to make the ORM work with both sync and async worlds once the interior is fully-async.

> In one project I really enjoyed using channels just as a background worker instead of celery, and the DEP does not talk much in the Email section: use the async variant, can be tackled separately, low priority.

> 

> For now the ORM works, but if sending an email fails (ie. SMTP down) it will raise an exception that can end up in a 500 error, it would be great if Django could retry so that emails leave when its SMTP is up again, would that maybe weight in priority ?

If you want guaranteed email delivery, that's a task for something like Celery or a third-party API; any method of sending emails in the background in the same process, be it threads or async coroutines, is going to be unreliable as the server may die any time. I don't think it's sensible for Django to try and solve this problem internally, at least not as part of this async push.

Andrew

On Thu, May 9, 2019 at 12:56 PM Patryk Zawadzki <pat...@gmail.com> wrote:

That said, I also think it's important to allow the ORM to support both modes in the long term. I truly believe the best way to be able to write async code is to _have the choice to write it_, rather than being made to all the time; if we make people use a separate, async ORM, then we force them to write every view asynchronously, with all the extra danger and thinking that requires. It's much better for Django to do the hard work, and say "hey, if you want to write asynchronously or synchronously, that's fine - it takes literally zero extra effort to go either way".

Slightly off-topic but once we have an async ORM, making it synchronous is not impossible (I believe either Channels or Daphne already have shims that use a worker thread to spin the event loop until a future is fulfilled).

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/402dbe77-44fc-485b-b058-e619a1576ddb%40googlegroups.com.

[Andrew Godwin]

On Thu, May 9, 2019 at 1:04 PM Aymeric Augustin <aymeric....@polytechnique.org> wrote:

Hello Andrew,

Thanks for your work putting together this plan. Within our constraints, it's a good plan.

Regarding templating, I would say it isn't a priority because a developer who knows how to parallelize I/O bound operations will prefer (or at least accept) to perform these operations in the view, not in the template.

I'm on the fence about the convention for async APIs. I'm not super excited by spraying async code with _async prefixes. The namespacing approach would allow for cleaner async code. Most Python modules should be either sync or async, not a mix of both styles. But I might be underestimating the importance of explicitness...

I am also not a fan of the approach, but I did err towards being explicit. Jannis suggested what I think might be a nicer approach on Twitter, which is to add an async "proxy object" to access methods with, e.g.:

cache.get("foo")

cache.async.get("foo")

This is still explicit but looks less ugly, and gives us a way in future to make some modules transition to async-by-default (by e.g. supplying cache.sync.get). What do you think?

Andrew

[Patryk Zawadzki]

I am also not a fan of the approach, but I did err towards being explicit. Jannis suggested what I think might be a nicer approach on Twitter, which is to add an async "proxy object" to access methods with, e.g.:

cache.get("foo")

cache.async.get("foo")

This is still explicit but looks less ugly, and gives us a way in future to make some modules transition to async-by-default (by e.g. supplying cache.sync.get). What do you think?

I think I'd personally prefer to have to do `from django.asyncore.cache import cache` and if the synchronous cache raised an exception when attempting to use it with an event loop active. Like Aymeric, I don't see myself mixing async and sync code within one module.

[Aymeric Augustin]

> On 9 May 2019, at 22:06, Andrew Godwin <and...@aeracode.org> wrote:
>
> Jannis suggested what I think might be a nicer approach on Twitter, which is to add an async "proxy object" to access methods with, e.g.:
>
> cache.get("foo")
> cache.async.get("foo")
>
> This is still explicit but looks less ugly, and gives us a way in future to make some modules transition to async-by-default (by e.g. supplying cache.sync.get). What do you think?

I don't think it makes a significant difference from a readability perspective in this example. It does have some advantages, though:

- It could be easier to implement a sync version based on the async one, or vice-versa, if each one has its own class. It will probably be more DRY.
- It could also be a bit more usable by developers, especially in IDEs

Also it could provide an alternative solution for the async attribute access problem: `model_instance.async.related_field` could be a Future that, when awaited, resolves to the related field. So you could `await model_instance.async.related_field` instead of having to `select_related`. To be honest I didn't investigate all the consequences of this idea. It seemed worth mentioning, though, even if it turns out to be impractical :-)

We've always considered that implicit queries on attribute access were an intractable problem. I said it on stage an DjangoCon US 2013. I'm now wondering if I was wrong all along! In an async ORM context, every time we traverse a relation, we could create a Future that would execute the query and resolve to its result. This would require one await per SQL query so we'd still get the benefit of making queries explicit, although to a lesser extent than with an explicit select/prefetch_related.

--
Aymeric.

[J. Pic]

Hi Andrew,

On Thu, May 9, 2019 at 10:04 PM Andrew Godwin <and...@aeracode.org> wrote:

If you want guaranteed email delivery, that's a task for something like Celery or a third-party API; any method of sending emails in the background in the same process, be it threads or async coroutines, is going to be unreliable as the server may die any time. I don't think it's sensible for Django to try and solve this problem internally, at least not as part of this async push.

I'm a bit confused here, what benefit are you getting from async emails if you're already retrying emails in the background in production ?

Thanks for your kind reply,

Have a great day

--

∞

[J. Pic]

Nevermind my question you will get a lot more out of the workers, that Django 3.0 is going to be really blazing fast like channels that calls for a celebration xD

[Patryk Zawadzki]

W dniu czwartek, 9 maja 2019 22:47:48 UTC+2 użytkownik J. Pic napisał:

I'm a bit confused here, what benefit are you getting from async emails if you're already retrying emails in the background in production ?

Anything that uses I/O should be async to unblock the worker to process other things while for example waiting for an SMTP server to return. Network I/O like that often requires tens to hundreds of milliseconds, in this time you could process another request from another browser.

[Patryk Zawadzki]

I don't think it makes a significant difference from a readability perspective in this example. It does have some advantages, though:

- It could be easier to implement a sync version based on the async one, or vice-versa, if each one has its own class. It will probably be more DRY.
- It could also be a bit more usable by developers, especially in IDEs

Speaking about convenience, it would be great if we could make the sync versions raise exceptions when used with an event loop attached and the async ones fail when there isn't a loop running. I assume synchronous views would be executed within a worker thread with the event loop explicitly set to None. 

[KimSia Sim]

Hi Andrew, 

I joined this group and chat because I saw the twitter post you made about this DEP. I  find that interesting.

On Friday, May 10, 2019 at 3:49:30 AM UTC+8, Andrew Godwin wrote:

 Tom Christie, for example, has already started work on an asynchronous ORM. Some of Django's biggest sites don't use the ORM - for example, Instagram.

I have a question that strays slightly away from the main topic. I have looked at Tom's repos. Is it encode/databases  https://github.com/encode/databases that you're referring to?

Or do you mean Tom's working on an async ORM that works within Django?

Thank you. I think Django will be even more awesome with async features

[Andrew Godwin]

I don't think it makes a significant difference from a readability perspective in this example. It does have some advantages, though:

- It could be easier to implement a sync version based on the async one, or vice-versa, if each one has its own class. It will probably be more DRY.
- It could also be a bit more usable by developers, especially in IDEs

Right. It could also be a sub-class on the class in question to make the implementation easier - e.g. "core.cache.cache.sync" is a literal passthrough, and "core.cache.cache.async" is the async wrapper implementation.

 

Also it could provide an alternative solution for the async attribute access problem: `model_instance.async.related_field` could be a Future that, when awaited, resolves to the related field. So you could `await model_instance.async.related_field` instead of having to `select_related`. To be honest I didn't investigate all the consequences of this idea. It seemed worth mentioning, though, even if it turns out to be impractical :-)

I think this might be feasible? I'd want to probably push it to have its own research and proposal, probably, but it might be a nice way out of the initial thing of requiring select_related. I just don't know enough about how that might cascade down the ORM internals to judge it at this point!

Andrew

[Andrew Godwin]

My ASGI patch already does this with the @async_unsafe decorator all over the ORM so people can't screw up by accident. It would be quite easy to extend this to enforcement on both the sync and async versions - there's maybe an edge case that you can call an async function from a thread you have not started an event loop in _yet_, but I'd rather see if and when that happens and provide a workaround, maybe.

Andrew 

[Andrew Godwin]

On Thu, May 9, 2019 at 2:44 PM KimSia Sim <Kim...@oppoin.com> wrote:

I have a question that strays slightly away from the main topic. I have looked at Tom's repos. Is it encode/databases  https://github.com/encode/databases that you're referring to?

Or do you mean Tom's working on an async ORM that works within Django?

It is indeed encode/databases!

Andrew 

[Patryk Zawadzki]

We've always considered that implicit queries on attribute access were an intractable problem. I said it on stage an DjangoCon US 2013. I'm now wondering if I was wrong all along! In an async ORM context, every time we traverse a relation, we could create a Future that would execute the query and resolve to its result. This would require one await per SQL query so we'd still get the benefit of making queries explicit, although to a lesser extent than with an explicit select/prefetch_related.

In theory there's nothing stopping you from turning prefetch_related int on explicit fetch_related:

queryset = Foo.objects.filter(Foo.publish_date <= datetime.date.today())

queryset = fetch_related(queryset[:10], Foo.author)  # with a future

results = await queryset.all()

fetch_related(results, Foo.category)  # with a result set, returns identity

[Asif Saif Uddin]

I have a separate question. Is it possible to get the django 3.0 asgi things into a different package to use with django 2.2?

Thanks for the great work.

[Andrew Godwin]

On Fri, May 10, 2019 at 3:51 AM Asif Saif Uddin <auv...@gmail.com> wrote:

I have a separate question. Is it possible to get the django 3.0 asgi things into a different package to use with django 2.2?

Thanks for the great work.

Unfortunately not - this is covered a little in the section in the DEP that says why we can't do it in a separate package, but basically, the changes required to Django are too deep to do separately (or even as a long-running fork). 

Andrew

[Tobias Kunze]

Hi Andrew (and everybody following the discussion, of course),

First off, thank you for your work here. DEP9 is an excellent technical
document, and it was as easy and pleasant to read as a document of this
scope and depth can be.

Especially the Motivation section was very insightful – it might be
worth moving it up a bit, as I found myself dropping about a third of my
notes and questions from the first half of the DEP after reaching the
motivation section.
Also, discussing the Why first and the How later is a bit better
argumentatively – if I just spent half an hour reading about the How,
I'll start to see the Why as a given, so it's harder to reason about
alternatives upon reaching the motivation section. Maybe that's just me,
though.

The Sequencing section is equally helpful to get a feeling for the
implementation work. It might be worth including a note on additional
future DEPs there, as those are only mentioned in the high level summary
(and in the templating section, I think).

I have a couple of questions and comments – none of which are meant to
criticise the direction of the DEP or arguments. It's just niggles,
really. I also left some comments on the PR that concern only some
phrasing.

The DEP doesn't really include a discussion of potential downsides. Even
if that's not in scope of a DEP, I'd like to ask this here:

- Is there a potential negative impact for Django users who just
continue to use Django? The only mention of this says "Running code in
threads is likely not going to increase performance - the overhead
added will probably decrease it very slightly in the case where you're
just running normal, linear code". Do you have any details on that? I
would have expected some (even just a sentence or two) discussion of
potential downsides in the Rationale section next to the alternatives.
- Is there a potential negative impact on Django if work on this
proposal takes a long time/is abandoned?


Less general remarks:

> Every single feature that is converted to be async internally will also present
> a synchronous interface that is backwards-compatible with the API as it stands
> today (in 2.2), at least for the normal deprecation period.

As it stands, this will sound a lot like "we'll deprecate lots of
synchronous interfaces soon" to people who are afraid of exactly that.
It's probably worth to clarify this here, or to choose different
phrasing, unless that's what you're planning to do.

> This general overview works on nearly all features on Django that need to be
> async, with the exceptions mostly being places where the Python language itself
> does not provide async equivalents to features we already use.

Can you give examples for this? They don't need to be in the draft, I
think, I'd just like to understand this part better.

> Asynchronous views will continue to be wrapped in an ``atomic()`` block by
> default - while this reduces immediate performance gains, as it will lock all
> ORM queries to a single subthread (see "The ORM" below), it is what our users
> will expect and much safer. If they want to run ORM queries concurrently, they
> will have to explicitly opt out of having the transaction around the view using
> the existing ``non_atomic_requests`` mechanism, though we will need to improve
> the documentation around it.

By default, Django's views are not wrapped in ``atomic()`` blocks. This
is only the case if ``ATOMIC_REQUESTS`` is ``True``, which it isn't by
default. Not sure if an off-by-default feature is worth an entire
paragraph here, but in any case, please make it clear that not every
async view will be wrapped in an ``atomic()`` block (unless I'm mistaken
and they will be?).

> In some ways, this
> will end up looking more like Django 1.0 era middleware again from an internal
> perspective.

It might be worth to make it clear that the middleware interface doesn't
change on the user-facing side, though.

> Whenever a
> ``new_connections()`` block is entered, the transactions do not persist inside,
> but transactions can be made inside the ``new_connections()`` block and run
> against those connections.

I think this was the most complicated sentence in the entire document.
It took me several tries to parse it in a way that could be correct.
Could you try to clarify? I think the missing reference for the "the
transactions" and "those connections" probably led to my confusion.

> This means that, at some point, the ``valid`` methods and ``save``, at
> minimum, need to be able to be called in an async fashion.

The ``valid`` methods? Did you mean the ``clean`` (and ``clean_*``)
methods, or am I missing something?

> While Python is not a perfect asynchronous language, and there are some flaws
> in the core design of ``asyncio``,

This leads me to a question I haven't really seen discussed so far: How
stable is the asyncio API by now? In the 3.x releases so far, asyncio
API has shifted quite a bit. I looked through the 3.8 release notes, but
didn't see any major changes (except maybe task names). This is mostly
relevant to figure out how much work we'll have with the support of
future Python version, and/or keeping backwards compatibility.

You're mentioning `asgiref` a lot – do you expect it to become a
dependency of Django? Do you expect any other new dependencies to be
introduced?

Thank you again for your work (and making it through this “light reading”)
Tobias

[Andrew Godwin]

On Mon, May 13, 2019 at 4:31 AM Tobias Kunze <r...@rixx.de> wrote:

Hi Andrew (and everybody following the discussion, of course),

First off, thank you for your work here. DEP9 is an excellent technical
document, and it was as easy and pleasant to read as a document of this
scope and depth can be.

Especially the Motivation section was very insightful – it might be
worth moving it up a bit, as I found myself dropping about a third of my
notes and questions from the first half of the DEP after reaching the
motivation section.
Also, discussing the Why first and the How later is a bit better
argumentatively – if I just spent half an hour reading about the How,
I'll start to see the Why as a given, so it's harder to reason about
alternatives upon reaching the motivation section. Maybe that's just me,
though.

The ordering that's there is defined by the DEP template, alas - though maybe that's just because of historical reasons. I agree it might be better to swap them.

 

The Sequencing section is equally helpful to get a feeling for the
implementation work. It might be worth including a note on additional
future DEPs there, as those are only mentioned in the high level summary
(and in the templating section, I think).

I've added another paragraph in there to flesh it out a bit.

 

I have a couple of questions and comments – none of which are meant to
criticise the direction of the DEP or arguments. It's just niggles,
really. I also left some comments on the PR that concern only some
phrasing.

The DEP doesn't really include a discussion of potential downsides. Even
if that's not in scope of a DEP, I'd like to ask this here:

- Is there a potential negative impact for Django users who just
  continue to use Django? The only mention of this says "Running code in
  threads is likely not going to increase performance - the overhead
  added will probably decrease it very slightly in the case where you're
  just running normal, linear code". Do you have any details on that? I
  would have expected some (even just a sentence or two) discussion of
  potential downsides in the Rationale section next to the alternatives.
- Is there a potential negative impact on Django if work on this
  proposal takes a long time/is abandoned?

These are both good questions - I have added in explainers about them to the "Motivation" section. (In short: Yes, but we'll keep it to a small impact; No, as long as we're good about code committing and being sustainable).

 

Less general remarks:

> Every single feature that is converted to be async internally will also present
> a synchronous interface that is backwards-compatible with the API as it stands
> today (in 2.2), at least for the normal deprecation period.

As it stands, this will sound a lot like "we'll deprecate lots of
synchronous interfaces soon" to people who are afraid of exactly that.
It's probably worth to clarify this here, or to choose different
phrasing, unless that's what you're planning to do.

Good point, I've made this a lot clearer.

 

> This general overview works on nearly all features on Django that need to be
> async, with the exceptions mostly being places where the Python language itself
> does not provide async equivalents to features we already use.

Can you give examples for this? They don't need to be in the draft, I
think, I'd just like to understand this part better.

The classic one is attribute access - Django uses "model_instance.related_field.name" to do work in the background. Attribute access can't be async, though, so we can no longer call the database.

It also exists for length - doing "len(queryset)" does not pass down an async context, so you can't do blocking work down there in a subthread. The pattern continues for most things that Python makes objects supply __special__ methods for; unless serious work is done to Python as an async language, these will likely stay the same indefinitely.

 

> Asynchronous views will continue to be wrapped in an ``atomic()`` block by
> default - while this reduces immediate performance gains, as it will lock all
> ORM queries to a single subthread (see "The ORM" below), it is what our users
> will expect and much safer. If they want to run ORM queries concurrently, they
> will have to explicitly opt out of having the transaction around the view using
> the existing ``non_atomic_requests`` mechanism, though we will need to improve
> the documentation around it.

By default, Django's views are not wrapped in ``atomic()`` blocks. This
is only the case if ``ATOMIC_REQUESTS`` is ``True``, which it isn't by
default. Not sure if an off-by-default feature is worth an entire
paragraph here, but in any case, please make it clear that not every
async view will be wrapped in an ``atomic()`` block (unless I'm mistaken
and they will be?).

I have updated this. I have kept the paragraph about it because I think it's an important illustrative point about the whole problem.

 

> In some ways, this
> will end up looking more like Django 1.0 era middleware again from an internal
> perspective.

It might be worth to make it clear that the middleware interface doesn't
change on the user-facing side, though.

Done!

 

> Whenever a
> ``new_connections()`` block is entered, the transactions do not persist inside,
> but transactions can be made inside the ``new_connections()`` block and run
> against those connections.

I think this was the most complicated sentence in the entire document.
It took me several tries to parse it in a way that could be correct.
Could you try to clarify? I think the missing reference for the "the
transactions" and "those connections" probably led to my confusion.

I reworked this quite a bit to be a lot clearer about transactions and a point about isolation levels that I think is important to bring up.

 

> This means that, at some point, the ``valid`` methods and ``save``, at
> minimum, need to be able to be called in an async fashion.

The ``valid`` methods? Did you mean the ``clean`` (and ``clean_*``)
methods, or am I missing something?

I did mean that. It has been a bit too long since I wrote form code!

 

> While Python is not a perfect asynchronous language, and there are some flaws
> in the core design of ``asyncio``,

This leads me to a question I haven't really seen discussed so far: How
stable is the asyncio API by now? In the 3.x releases so far, asyncio
API has shifted quite a bit. I looked through the 3.8 release notes, but
didn't see any major changes (except maybe task names). This is mostly
relevant to figure out how much work we'll have with the support of
future Python version, and/or keeping backwards compatibility.

It is mostly-stable, and everyone I talk to (myself included) is slightly unsatisfied with it but generally thinks it does the job, which makes me believe it's probably an alright API. 

It's certainly more of a moving target than Python itself is, though. If Django adopts async in a big way and becomes a large user of asyncio, we may have a knockback effect on its development, for better or worse.

 

You're mentioning `asgiref` a lot – do you expect it to become a
dependency of Django? Do you expect any other new dependencies to be
introduced?

I do expect it to, and in fact the ASGI patch already does this, and causes `async_timeout` to be required too (as asgiref needs this). I expect no other dependencies. I consider `asgiref` acceptable as it is a Django project, though I have considered just copying over ("vendoring") the code we need from async_timeout to remove that extra dependency.

 

Thank you again for your work (and making it through this “light reading”)

And thank you for your feedback and your GIF :)

Andrew 

[Andrew Godwin]

Just a heads up that feedback on this draft DEP has slowed down, and so I have merged it into the deps repository.

This is an invitation for any additional feedback before I take the DEP to the Technical Board to get their opinion in a week or two. In the meantime, I am going to start work on writing up a funding plan for this, including various options for how we can pay people for their work.

Andrew

[Pkl]

Hello,

I'm a little late to the party, thanks for the big overview on this complex matter.

There are lots of thinks I still don't understand though, for example regarding "what it unlocks". What asyncio structures would allow to run several DB queries concurrently safely and easily, that can't be achieved with some threadpool-like mechanism in nowadays' cod ?

Most importantly, I'm not getting why gevent/eventlet-style solutions are systematically being dismissed in favor of asyncio. And the more I read/talk about it, the less I understand what all that hype around asyncio is. This is a whole new language, which forces people to trashbin half of the Python ecosystem, and remake it with similar code but filled with incompatible async/await statements.

For sure, having so obstrusive keywords everywhere can be handy when writing highly concurrent code without mutexes. But who writes highly concurrent code in webservers ? The very principle of webservers is imho precisely to never interfere with other requests, never modify process-global structures, and at worst delegate some heavy or concurrent tasks to some dedicated executor.

Greenlet-style parallelism would bring long-running requests and high parallelism to Django without having to touch the bulk of the code code, just the I/O parts (DB connections...) ; and thread-local is already greenified by Gevent for exemple, according to docs. Why would it bring "much higher risk of race conditions and deadlocks without careful programming" ? Preemptive threads are much more dangerous than the implicit but deterministic context-switching which occurs when greenlets reach socket/disk/sleep operations, and yet race conditions seem to be the least of the problems of the huge majority of Django programmers.

I'd love to be wrong, but I have the feeling that with a fraction of the work required by django-asyncio, it would be possible to greenify the whole of Django, fix lots of current limitations and corner-cases of gevent (lack of builtin executor to offload tasks to real threads, lack of support in python package X/Y/Z...), and without having to recode any of the "middle" parts of existing modules. How can we say "third-party support for this style of concurrency is much weaker", whereas through monkey-patching, about ANY python module (except those using blocking C extensions) can be used in a Gevent project ?

Granted, I have little experience with Geven and Asyncio, but all experience feedbacks I've read so far mainly insist on minor limitations of greenlets, and on the fact that "people are mainly going with asyncio" (a self-fulfilling prophecy?). Considered the dramatic difference in workload between the two, I'd really love to understand what killer-features justify to go for the "recode everything" solution (or what greenlet limitations would be show-stoppers on the long term).

Thanks in advance everyone for your feedback on this issue.

regards,
Pascal Chambon

[Andrew Godwin]

On Thu, Jun 6, 2019 at 3:20 PM Pkl <chambon...@gmail.com> wrote:

Hello,

I'm a little late to the party, thanks for the big overview on this complex matter.

There are lots of thinks I still don't understand though, for example regarding "what it unlocks". What asyncio structures would allow to run several DB queries concurrently safely and easily, that can't be achieved with some threadpool-like mechanism in nowadays' cod ?

Python threads perform badly, as they force context-switches even when no work is pending; you can't sensibly run more than 20/30 threads before the overheads seriously start eating into your performance.

 

Most importantly, I'm not getting why gevent/eventlet-style solutions are systematically being dismissed in favor of asyncio. And the more I read/talk about it, the less I understand what all that hype around asyncio is. This is a whole new language, which forces people to trashbin half of the Python ecosystem, and remake it with similar code but filled with incompatible async/await statements.

...

Granted, I have little experience with Geven and Asyncio, but all experience feedbacks I've read so far mainly insist on minor limitations of greenlets, and on the fact that "people are mainly going with asyncio" (a self-fulfilling prophecy?). Considered the dramatic difference in workload between the two, I'd really love to understand what killer-features justify to go for the "recode everything" solution (or what greenlet limitations would be show-stoppers on the long term).

I can't give you a full answer about why they are being dismissed, but the move is clearly systematic. My own personal experience is that writing with gevent in particular ends up being very difficult, as it is not very explicit about what is async, what causes a context switch, and so you end up wrapping a lot of your code in locks to even try and get code that isn't susceptible to nasty race conditions.

I suspect asyncio fits much more with the Zen of Python - you know exactly when a context switch might occur (when you see an await), and modules explicitly add support for it rather than having it monkey-patched in. In addition, most people I know in the Python community who are actively working on async libraries are doing so against asyncio.

You could make the same argument with trio - it's arguably a better, cleaner async implementation. But, again, it's not where Python is at. This change is already big enough that it's very important we keep consistency with Python core to lower the workload. If Python core turned around and blessed greenlets and gevent as the chosen async solution, I'd change my mind, but I haven't seen any evidence of that over many years.

Andrew

[John Obelenus]

I wonder about the end-result payoff of this approach. In general, Django/Python code is not going to be I/O bound, which is where asynchronous approaches are going to get the bang for your buck. Even when it comes to DB access—the DB is a lot faster than the python and django code running against the result set. And too much context-switching (as you noted) has painful ramifications for performance.

I can absolutely see why creating a layer that handles asgi, websockets, and http requests asynchronously is going to pay off. Bit time. But I'm less certain that the ORM access will benefit from an asyncio approach. Do we have anything that approaches a hard number that would tell us re-doing the ORM layer in asyncio would get us X% performance benefit?

I'm basing my thoughts off this well-reasoned look at performance: https://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/

[Andrew Godwin]

On Fri, Jun 7, 2019 at 9:19 AM John Obelenus <jobe...@gmail.com> wrote:

I wonder about the end-result payoff of this approach. In general, Django/Python code is not going to be I/O bound, which is where asynchronous approaches are going to get the bang for your buck. Even when it comes to DB access—the DB is a lot faster than the python and django code running against the result set. And too much context-switching (as you noted) has painful ramifications for performance.

To the contrary, I have found that as you scale up, a large amount of your time becomes I/O (either HTTP calls to other components/hosted serviecs or database calls). Our APM at work shows me that it's around 80% of request time.

Obviously we don't design Django just for large use cases, which is why it's not going to be the default, but with the massive growth of hosted services, I suspect this trend will continue to trickle down to smaller deploys too. And ultimately, for smaller deploys performance is rarely a concern anyway.

 

I can absolutely see why creating a layer that handles asgi, websockets, and http requests asynchronously is going to pay off. Bit time. But I'm less certain that the ORM access will benefit from an asyncio approach. Do we have anything that approaches a hard number that would tell us re-doing the ORM layer in asyncio would get us X% performance benefit?

I'm basing my thoughts off this well-reasoned look at performance: https://techspot.zzzeek.org/2015/02/15/asynchronous-python-and-databases/

I do not personally have hard numbers that I am allowed to share, unfortunately, but I would encourage you to look at results on benchmarks that include database access - like this one (https://twitter.com/_tomchristie/status/1005001902092967936) using Python asyncio/ASGI - and see that it does make a difference. Obviously it doesn't matter for all deploys, but I believe it matters for the majority of site architectures as they scale up.

Andrew

[Pascal Chambon]

Hello, 

There is something a little scary for me, in changing all the core of Django to async, when this really helps only, imho, a tiny fraction of users : websocket/long polling services, and reddit-like sites with thousands+ hits per second. For most webpages and webservices, async artillery sounds quite overkill.

Are cpython threads inefficient ? As far as I know they are only kernel threads constrained by the Gil, so they shouldnt wake up when they are blocked on io syscalls/mutexes (or do they?), and context switches remain acceptable compared to the slowness of python itself.

We used to provide provisioning and automatic authentication for 20 million users, with partner webservices tar-pitting us for sometimes 1mn. The nightmare scenario. But with 2 machines, 1 process by core, and 800 threads by process, it did the job, enough for us to answer millions of hits a day. Without even relying on other no-recoding optimizations like pypy or gevent.

Async would certainly have been a relevant techno if we had known in advance that our partners would be so slow, but avoiding the extra complexity burden of this style (where a single buggy dependency can block all requests in a process, where all modules have to be recoded for it) was also a huge benefit. And the limited thread pool also protected our DB from unbearable loads.

It's very nice if a proper async ecosystem emerges in python, but I fear lots of people are currently jumping into it without a need for such performance, and at the expense of lots of much more important matters like robust ess, correctness, compatibility... like it happened for docker and microservices, transforming into fragile bloatwares simple intranets, which just needed a single django codebase deployed in a single container.

A few days ago I audited a well used django module, the current user was stored in a global variable (!!!). People might eventually fix that ticket, use threadlocals, and then switch to a future django-async without realizing that the security issue has come back due to the way async works.

Still I hope I'm wrong, that the performance gains will prove worth the software fragmentation and complexity brought by asyncio, but I still dont understand them for 99% users... Especially as long as key-in-hand solutions like greenlets exist for power users.

Regards, 

Pascal

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/5CVsR9FSqmg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1urJVeamhLWLLDpkz_-e1fAM59x6b6rzyAV%3DZAUgg9qrvQ%40mail.gmail.com.

[Andrew Godwin]

On Sat, Jun 8, 2019 at 3:14 AM Pascal Chambon <chambon...@gmail.com> wrote:

Hello, 

There is something a little scary for me, in changing all the core of Django to async, when this really helps only, imho, a tiny fraction of users : websocket/long polling services, and reddit-like sites with thousands+ hits per second. For most webpages and webservices, async artillery sounds quite overkill.

Are cpython threads inefficient ? As far as I know they are only kernel threads constrained by the Gil, so they shouldnt wake up when they are blocked on io syscalls/mutexes (or do they?), and context switches remain acceptable compared to the slowness of python itself.

It's fine when you only at 5/10 threads - which, notably, is what most WSGI servers run at. When you get to the hundreds, though, you start losing a large proportion of your execution time (tens of percent, in some cases). 

 

We used to provide provisioning and automatic authentication for 20 million users, with partner webservices tar-pitting us for sometimes 1mn. The nightmare scenario. But with 2 machines, 1 process by core, and 800 threads by process, it did the job, enough for us to answer millions of hits a day. Without even relying on other no-recoding optimizations like pypy or gevent.

Async would certainly have been a relevant techno if we had known in advance that our partners would be so slow, but avoiding the extra complexity burden of this style (where a single buggy dependency can block all requests in a process, where all modules have to be recoded for it) was also a huge benefit. And the limited thread pool also protected our DB from unbearable loads.

Please remember that even after this change, Django will still expect you to write synchronously by default, and not impose any of that extra complexity on you. We will only swap out the "native" implementation of things if the performance matches (within ~10%) or exceeds that of the synchronous version when there's a couple of threads going; it's expected this will largely be the case due to the direct benefits of idling less. 

But - the plan is not to make it more complex by default (you only have to interact with the async if you want to) or slower.

 

It's very nice if a proper async ecosystem emerges in python, but I fear lots of people are currently jumping into it without a need for such performance, and at the expense of lots of much more important matters like robust ess, correctness, compatibility... like it happened for docker and microservices, transforming into fragile bloatwares simple intranets, which just needed a single django codebase deployed in a single container.

A few days ago I audited a well used django module, the current user was stored in a global variable (!!!). People might eventually fix that ticket, use threadlocals, and then switch to a future django-async without realizing that the security issue has come back due to the way async works.

Still I hope I'm wrong, that the performance gains will prove worth the software fragmentation and complexity brought by asyncio, but I still dont understand them for 99% users... Especially as long as key-in-hand solutions like greenlets exist for power users.

I agree with you that there's a chance this is all useless and doesn't bear fruit, in which case I will be the first person to pull the plug and say that Python async isn't ready. However, I've been working with it for the last four years, including on several very large deployments, and there are some direct benefits that I believe we can get without making things a lot more complex, even inside Django.

Andrew 

[Andrew Godwin]

Hi everyone,

After a long and involved vote, I can announce that the Technical Board has voted in favour of DEP 0009 (Async Django), and so the DEP has been moved to the "accepted" state. 

As some may have seen, I've started work on adding async support to views (https://github.com/django/django/compare/master...andrewgodwin:async_views) - this is, as the DEP states, the last "blocking" change before we can open up lots of parallel work on making other parts of Django async, and so now the DEP is approved the next step is to work out funding and organisation for future async work.

If you are interested in helping with fundraising, then please get in touch with me directly; I have some ideas about how to structure it, but I could do with some people to help out. Otherwise, stay tuned for more information about how to get involved contributing and what to work on!

Andrew

[Jacob Kaplan-Moss]

Congratulations, and great news!

I hope the TB will consider sharing details and/or a summary of the "long and involved vote"; I'll bet there's a bunch the broader community could learn from the specifics. 

Jacob

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1up%2BRzR3FcA8xBPPSfpY_5%2B5bfyZbJ%3DN9G0Ohy9kOGb53A%40mail.gmail.com.

[Andrew Godwin]

I'll ask permission and then summarise the points raised back out here!

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAK8PqJHBvv-Gd-%2BQR95NB52zr8meGhwgyFTysgqc1KWdvwP9KA%40mail.gmail.com.

[Ehigie Aito]

Django 3.0?

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upqMLyZgtmcqgmYV8whH43SCXJimTCZGJxA4p1D5PRYTA%40mail.gmail.com.

[Andrew Godwin]

On Sun, Jul 21, 2019 at 1:11 PM Ehigie Aito <aitoe...@gmail.com> wrote:

Django 3.0?

Django follows time-based releases; what's in Django 3.0 will depend on when we can get it landed. At the moment I am optimistic something will make it in, but I make no promises!

Andrew 

[Andrew Godwin]

OK, here is some of the feedback from the Technical Board, aggregated together:

* There were questions around contextvars and if they might supplant the need for a threading.local override - I clarified why this doesn't work in the DEP.

* Several board members queried around how we might distinguish async versions of functions/method from sync ones, and if we could get core Python to implement a better, language-level way of having both under the same name. For now, the conclusion is to pursue just having them as separate names, likely under a sub-object (so, for example, cache.get and cache.async.get), and ensure the documentation is very clear when we must present both

* Templating was queried by multiple people, with one board member referencing the streaming template work that happened in the past (https://github.com/django/django/pull/11157), and another wondering if we should just say that Django templates are sync forever and potentially look at Jinja2 for async support if users desire. My personal takeaway from this is that we should address async templates on its own, with a team who can do some experimentation and come back with a recommendation.

* There was a concern over the fact that asgiref is now a core dependency of Django, joining tzdata, and that there had already been some negative user reactions about this in the wild. As part of helping out this problem, I removed asgiref's own dependency on async_timeout, so it's now just a single flat package.

* There was concern about users being able to opt out of async mode should it cause a performance hit (like we allow with localisation), and another member mentioned the djangobench benchmark project. I stated that if adding async to the core flow impacted synchronous use by 10% we would find a way to make it strictly opt-in, as well as my personal belief that this may actually make things faster even for synchronous code and projects. I also said that I'd like to revive regular benchmarking and add an async one in to both help stop potential slowdown caused by adding async, as well as illustrating the advantage of async code for certain workloads.

* A board member mentioned that the behaviour of transactions is already tricky, and adding this in might make it worse. I agree with them, but I've already started looking at ways to keep transactions and threads tied strictly together and put some explicit safeguards to stop them leaking across threads. That said, transactions and the ORM are my own personal biggest worry about this first phase of the project.

* It was pointed out that while Django does not ship a threadlocal for request, this is a very common pattern and we need to make sure it works. I said that asgiref.local.Local is intended to be a drop-in, backwards-compatible replacement for threadlocal for this very reason; most projects will merely need to change a single import to get the correct behaviour.

* There was some questioning about how the debug server should run and if it should enable the asyncio debug mode. I think it should, but I'm not sure at what stage we will make an async runserver the default, considering we will likely have to bring in Daphne or Uvicorn as a dependency to do so.

* The final query I'm going to pull out here was non-technical - that the Django project has lost many contributors over the years, is essentially in a maintenance mode, and that we likely do not have the people to staff a project like this. I agree with the observation that things have substantially slowed down, but I personally believe that a project like async is exactly what Django needs to get going again. There's now a large amount of fertile ground to change and update things that isn't just fixing five year old bugs.

Hopefully this gives you some idea of the conversation we had. In my years on the Board, this is by far the most detailed a vote has ever gotten, and I can only apologise to the incoming board for springing this on them right after an election!

If anyone on this list would like to continue to talk about the above, or if a Board member wants to bring their conversation out here, you are all more than welcome.

Andrew