A Django Async Roadmap

[Andrew Godwin]

Hello everyone,

For a while now I have been working on potential plans for making Django async-capable, and I finally have a plan I am reasonably happy with and which I think we can actually do.

This proposed roadmap, in its great length, is here:

https://www.aeracode.org/2018/06/04/django-async-roadmap/

I'd like to invite discussion on this potential plan - including:

 - Do we think async is worth going after? Note that this is just async HTTP capability, not WebSockets (that would remain in Channels)

 - Can we do this in a reasonable timeframe? If not, is there a way around that?

 - Are the proposed modifications to how Django runs sensible?

 - How should we fund this?

There's many more potential questions, and I really would love feedback on this. I'm personally pretty convinced that we can and should do this, but this is a decision we cannot take lightly, and I would love to hear what you have to say.

Andrew

[Rigel]

Hello Andrew.

I like your proposal and want to help out. In particular, what I'm really fond of is the approach of making async available only if you want it, while keeping Django backwards-compatible. Django is brilliant in that, like the Python language, it's easy and fun to pick and learn, but flexible and powerful at the same time.

Rigel.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1up4%2BP%2B4bCSSiG6t845idoM8dvktnuTs4dH2R5aAuCz3iA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Christie]

> Do we think async is worth going after?

I think this is hugely important. Django's advantage in the field currently is in it's developer productivity and maintainability. As frameworks based on Node or on languages with co-operative concurrency built-in continue to mature it's going to face more of a squeeze there, while being less resource efficient than many alternatives.

Although performance characteristics are often over-stated or over-valued, it's still a substantive point against a team choosing Django in some circumstances. It's particularly relevant in super successful high-scale cases, and I want Django to continue to be a great choice both for single person building out an MVP on a couple of instances, all the way up to large teams working on massive systems, with many running services.

If we don't attempt to tackle this I think we may end up very slowly starting to write Django out as a contender for some teams.

> Can we do this in a reasonable timeframe? If not, is there a way around that?

I can't speak to the ORM work, as I don't know the internals well enough, but I think how everything else is proposed makes sense. The minimal first step, of tackling the request/response path is already potentially valuable for any teams that include an API gateway as one of their services - being able to have a Django-based app that can make non-blocking requests out to other services would immediately benefit a decent chunk of real-world use cases.

> How should we fund this?

It's a pretty serious amount of funding you'd want to see, so I guess ideally not as a single backer.

A joint combination of Mozilla's MOSS program, PSF grant, and corporate sponsorships all orchestrated together? (Waves hands magically)

Wrt. corporate sponsorships I think we might well want to set realistic expectations about how much developer time money actually buys, and a set a decently high minimum tier. What I think works especially well is tying in contributions, so. eg. an agreed-in-principle grant that's conditional on also achieving a certain amount of corporate sponsorships might be a great motivator for companies to make a shared, collaborative investment. ??

Thanks Andrew - *really* exciting proposal.

[Tom Forbes]

Hey Andrew, thank you for the very exciting proposal and the ongoing work on Django channels! There is a lot to consider here and some very interesting problems to work through - I’d love to help out wherever I can.

Do we think async is worth going after?

I think this is very much worth doing. Async interfaces to the ORM and templates would be a huge, huge improvement if we can get it right. I think Tom Christie summed it up really well however I would add that I think teams excluding Django as a contender may happen sooner rather than later if it’s not happening already.

Are the proposed modifications to how Django runs sensible?

I had a few random thoughts while reading your proposal:

I think getting rid of the related field references could be a big issue here and cause a lot of headaches for existing applications. Could we do this in a backwards compatible way at all? I wonder if PEP 567 could help here, could we define some kind of ‘Django async context’ and do different things depending on if this is true or false?

Regarding exposing an async interface alongside a synchronous one: are you envisaging something like appending _async to methods or having some kind of wrapper class that could be optionally included to go from async->sync? I guess it would have to be appending _async, as a wrapper class could be used in different contexts.

Async templates seem particularly powerful if we work out the details. We could eventually render different parts of the template concurrently, i.e example each iteration of a for loop could be it’s own future resolved independently, but this is likely a pipe dream.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

[Curtis Maloney]

On 06/05/2018 07:54 AM, Tom Forbes wrote:
>> Are the proposed modifications to how Django runs sensible?
>
> I had a few random thoughts while reading your proposal:
>
> I think getting rid of the related field references could be a big issue
> here and cause a lot of headaches for existing applications. Could we do
> this in a backwards compatible way at all? I wonder if PEP 567

> <https://www.python.org/dev/peps/pep-0567/> could help here, could we

> define some kind of ‘Django async context’ and do different things
> depending on if this is true or false?

One thought about the related field stuff... whilst having it makes it
easy to get started, any site at scale will sensibly obviate their need
in almost all cases by using select_related, for performance reasons.

--
C

[Andrew Godwin]

I think getting rid of the related field references could be a big issue here and cause a lot of headaches for existing applications. Could we do this in a backwards compatible way at all? I wonder if PEP 567 could help here, could we define some kind of ‘Django async context’ and do different things depending on if this is true or false?

Unfortunately even if you have context variables, you simply can't await inside of an attribute reference because there's a synchronous call in your stack. I even chatted to some Python core devs at PyCon US about this and we couldn't really think of a way out of this problem without some very serious changes to the language.

What will work, though, is attribute access from a sync context - the related field references will only error out of they know they're in an async loop, and we can detect that by looking for an active event loop on the current thread (no context varaible needed).

 

Regarding exposing an async interface alongside a synchronous one: are you envisaging something like appending _async to methods or having some kind of wrapper class that could be optionally included to go from async->sync? I guess it would have to be appending _async, as a wrapper class could be used in different contexts.

I'm not quite sure on this one. For backwards compatibility, we have to keep sync methods working with the same name, but there are several options as to how to separate them - keyword arguments (like "my_method(..., async=True)"), suffixes ("my_method_async()", as you suggest), automatically changing based on if you're in an async thread (which is a bit... magic) or different module namespaces ("from django.db.async import foo").

I want to sketch out what all of these look like as part of this project and then work out which is best for Django.

 

Async templates seem particularly powerful if we work out the details. We could eventually render different parts of the template concurrently, i.e example each iteration of a for loop could be it’s own future resolved independently, but this is likely a pipe dream.

That would very much be a long-term thing, and honestly something I might consider handing off to something like Jinja (which already has full async support, I learnt yesterday)

Andrew

[C. Kirby]

Andrew,

I don't know enough about async web to comment directly on your proposal, but like most of your proposals it looks well thought out and comprehensive. Kudos to you.

As you state, the scope of this project is one that can raise the profile of developers in the community - that appeals to me in ways that bug fixing has not, mostly because I have not encountered bugs in django.

Towards that end, can you point to any of your preferred references to get up to speed on async concepts to help understand the scope and scale of this undertaking? Even if I don't work on the project I think intuitively that I could make good use of it.

Thanks,

Chaim

[Mattia Procopio]

I'm really excited that the django community is starting to think of async-friendly django. I think this is needed for the community and will be a strenght point for the framework itself for next couple of years if this will take place.
Now that asyncio is a thing in the Python world I think more and more devs will try to use it, I personally tried to use it within django a couple of times and I ended up hacking lot of things,I really would like to have an available async loop
that I can use to run my coroutines and do other stuff. These are just hacks of course but I think that if django won't be more async friendly lots of people will end write their own hacks ending with a mess, while if
async will be baked into it there will be no need of custom async-like code. Adding initial support for coroutine views will be a very cool and interesting starting point for experiments.
Really hope this plan will become real

[Tom Forbes]

> Unfortunately even if you have context variables, you simply can't await inside of an attribute reference because there's a synchronous call in your stack. I even chatted to some Python core devs at PyCon US about this and we couldn't really think of a way out of this problem without some very serious changes to the language.

I’m just spitballing here but could we not conditionally return a future/promise if some conditions are met, then implement some kind of chaining based on the attribute accesses? So `instance.related_field.other_related_field` breaks down into the `related_field` returning some kind of awaitable instance, and we return a nested instance when `other_related_field` is accessed? This last future would resolve to ’the right thing’. Perhaps just throwing an exception if we are in an async loop is an acceptable and simpler trade off however just as long as the existing synchronous implementation works as-is.

>  keyword arguments (like "my_method(..., async=True)"), suffixes ("my_method_async()", as you suggest), automatically changing based on if you're in an async thread (which is a bit... magic) or different module namespaces ("from django.db.async import foo”).

If we go with a `_async` suffix we could reduce a lot of boilerplate with a bit of magic. We could potentially auto-generate the sync versions of the methods with a decorator, seeing as they would just call the async version and wait for the result? I’m not sure if we could do the same nicely with a different namespace, but that might be cleaner.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFwN1upPVQCjHTxgJmzJSUzaevsfQ7evwi89hFt4OvHmMe5m2A%40mail.gmail.com.

[Tom Christie]

I wonder if a sensible first tack onto this might be:

Supporting an ASGI/asyncio request-response cycle in 2.2 might be a perfectly achievable goal with or without funding. Users wouldn't be able to  use the ORM, or much of the default middleware, but for some gateway-type services I can still see it being useful.

A proof of concept there might help validate (or otherwise) both the motivation (run some actually tests against a service that makes outgoing HTTP calls and then returns a response - what sort of throughput can we achieve there vs. the threaded case?) and the feasibility (what aspects that we haven't considered do we bump into along the way to implementation?).

Perhaps we could work on the basis of (1) pulling together a proof of concept there, and then (2) attempting to get that into the 2.2 alpha freeze, without being blocked on getting together the rest of the plan & funding for the more substantial parts such as the ORM work?

That wouldn't necessarily preclude also getting asnycio ORM work into 2.2, or achieving funding early on, but would at least give us something valuable to start working towards immediately?

[Jordan Eremieff]

The proposal looks great, thanks for putting it together and pushing async in Django forward.

I'm not sure the best approach to take here, but I've just started experimenting on a fork: https://github.com/erm/django/tree/async-experiment-2. I am unsure how much of the more complex items I may be able to find solutions for, but it seems like a good first step is to get basic ASGI support running alongside the current behavior and then bring in the middleware from that point. Open to any suggestions for other directions to go in, otherwise I'll just be digging away at it this way for now.

[Andrew Godwin]

Chaim: For references to get up to speed on async concepts, there's not a lot of great beginner-level stuff out there. I have a practical guide to async functionality at https://www.aeracode.org/2018/02/19/python-async-simplified/, but it's not great at going into detail on Web stuff and why it has big advantages. I'd love if other people could chime in with resources that helped them (rather than me, who just started writing async code for fun 8 years ago and have sort of kept up with things that way)

Tom F:

>  I’m just spitballing here but could we not conditionally return a future/promise if some conditions are met, then implement some kind of chaining based on the attribute accesses?

We could, but as you can see from the length of the paragraph you've written this is incredibly _tricky_ to do correctly, and I'm not a fan of lazy objects in the first place. Something we should investigate maybe, but ultimately I'd prefer a simpler solution with less magic.

>  If we go with a `_async` suffix we could reduce a lot of boilerplate with a bit of magic

Sure, though honestly I wouldn't say it needs magic, just:

    def function_async(self, foo):

       ...

    function = async_to_sync(function_async)

Tom C:

> Supporting an ASGI/asyncio request-response cycle in 2.2 might be a perfectly achievable goal with or without funding. Users wouldn't be able to  use the ORM, or much of the default middleware, but for some gateway-type services I can still see it being useful.

Indeed. The time I personally spend each week on Channels (around 2 days fulltime-equivalent) could easily be partially or wholly diverted into this project, as Channels is pretty stable right now.

That said, my time for that project has been recompensed by the Mozilla OSS grant they gave us a while back up until now - the funds for that are finally drying out, and while I could do the work for free, I take a paycut from my normal work to get the time that I use to work on Django, so funding would be nice to have - and I don't expect anyone else to work for free. I suspect the funding level needed would not be too great for the initial work.

Given we have some MOSS funds left, I may use the last of those to start getting an async pathway ready for 2.2 once this whole thing gets approved. It's still within the scope of the original grant.

Jordan:

I'm not sure how we're going to develop this yet, but it'll likely be a branch in the main Django repo once this proposal is approved (there's still a little while before that happens - there needs to be consensus here and/or a technical board vote).

Andrew

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/7a958407-7dcf-4fd8-b09f-67e28346cb65%40googlegroups.com.

[Josh Smeaton]

I think most of what you've laid out sounds great and that pursuing async Django is in the projects best interests. The sync to async and async to sync wrappers that have come out of channels give me much more confidence that this project is doable in a reasonable amount of time with backwards compatibility being preserved.

The only real concern I have at the moment is around your comments regarding on demand foreign key traversal. If existing code running synchronously is going to be impacted, that's going to be very difficult for a lot of people. If it's only asynchronous traversal that'll have issues, then I have no real concern, as on demand foreign key fetching is usually a bug anyway.

Having a brief read through the psycopg asynchronous docs[0], it looks like a number of features will be impossible or troublesome to use, like transactions, executemany, and named cursors (.iterator() with server side cursors). We'd also need to investigate how pgbouncer would work in async mode, as most large sites using postgres are also using pgbouncer. I would expect support can only further improve, especially if there is a driver like django pushing. Fallback would just be to run inside a thread pool though, so it's not a blocker for the rest of the proposal.

Very exciting times ahead!

[0] http://initd.org/psycopg/docs/advanced.html#asynchronous-support

[Ran Benita]

> Loading a lazy ForeignKey/RelatedField attribute on a model instance

Also deferred fields.

[Andrew Godwin]

On Fri, Jun 8, 2018 at 11:31 PM Josh Smeaton <josh.s...@gmail.com> wrote:

The only real concern I have at the moment is around your comments regarding on demand foreign key traversal. If existing code running synchronously is going to be impacted, that's going to be very difficult for a lot of people. If it's only asynchronous traversal that'll have issues, then I have no real concern, as on demand foreign key fetching is usually a bug anyway.

It would only be for asynchronous contexts - it would still work exactly as today for synchronous ones. Backwards compatability is VERY important to me during this whole thing.

 

Having a brief read through the psycopg asynchronous docs[0], it looks like a number of features will be impossible or troublesome to use, like transactions, executemany, and named cursors (.iterator() with server side cursors). We'd also need to investigate how pgbouncer would work in async mode, as most large sites using postgres are also using pgbouncer. I would expect support can only further improve, especially if there is a driver like django pushing. Fallback would just be to run inside a thread pool though, so it's not a blocker for the rest of the proposal.

Yes, there's a lot of specifics that need to be worked out for each driver (e.g. does it even make sense with SQLite at all?), but I think providing that space and framework to experiment in is going to be very beneficial. The aiomysql package implemented everything in pure Python, as well - it's possible that similar would be needed for full async PostgreSQL support, where the non-compiled speed reduction might be outweighed by the parallelism improvements.

Andrew

[Emil Stenström]

I just wanted to add that I'm really excited for this change.

On PyCon Sweden 2015 I held a "Why Django Sucks"-presentation where I talked about where the web is heading, and why django is lagging behind. The main point was lack of support for async. After the presentation lots of people came forward, and made clear that:

1. They would really like async support in Django
   
2. Django would NEVER be what I'd hope for. "That would need a separate deamon process", "That would be too much work", "Nah, Django should just leave the web stuff for Javascript and become an API".

That Andrew is pushing the project in this direction, makes me more excited for Django than I've been in a long time.

[charettes]

Regarding the lazy loading of deferred fields and foreign keys
I wanted to mention I've been working on a third-party application
that allows overriding the default behavior[0].

The project works by tainting objects retrieved from "sealed"
querysets and having fields descriptors lookup whether or not
the object is "sealed" on attribute access and warn the developer
about it if it's the case. Warnings can be elevated to errors
using `filterwarnings` when deemed appropriated (e.g. CI, staging).

It has been an useful tool to assist in figuring out where
`select_related()` and `prefetch_related()` should be used
to adjust complex projects database interactions.

I assume a similar pattern could be used to mark objects retrieved
from `QuerySet.__aiter__` to prevent non-async queries from being
performed on attribute accesses; on `Model._state.async = True`
field descriptors would error out.


Cheers,
Simon

P.-S.

While the project might look complex most of the code takes care
of the delicate tasks of replacing fields descriptors once models
are configured which could be significantly simplified if it was
part of Django core.

[0] https://github.com/charettes/django-seal

[Curtis Maloney]

In general this sounds like a tremendously useful tool... I'm caused to
wonder, however... what, if any, are the performance impacts?

--
Curtis

> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com

> <mailto:django-develop...@googlegroups.com>.

> To post to this group, send email to django-d...@googlegroups.com

> <mailto:django-d...@googlegroups.com>.

> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/django-developers/0b3b92b9-eee7-4909-82d6-c0138e8b5760%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/0b3b92b9-eee7-4909-82d6-c0138e8b5760%40googlegroups.com?utm_medium=email&utm_source=footer>.

[charettes]

AFAICT the performance hit is minimal unless you are doing something

slow when logging warnings (e.g. -Walways with a slow I/O py.warnings handler).

The Python/Django instrumentation simply adds a value to Model._state

and look it up on attribute accesses.

Cheers,

Simon

> <mailto:django-developers+unsub...@googlegroups.com>.

[James Addison]

On Monday, June 4, 2018 at 6:18:23 AM UTC-7, Andrew Godwin wrote:

For a while now I have been working on potential plans for making Django async-capable, and I finally have a plan I am reasonably happy with and which I think we can actually do.

Andrew,

Out of curiosity, how would these kinds of changes interact with gunicorn running with a `gevent` worker? I also currently use `psycogreen.gevent` for postgresql async behaviour.

Thanks,

James

[Ray Marceau]

James Addison,

 I also currently use `psycogreen.gevent` for postgresql async behaviour.

How did you go about using psycogreen.gevent for async postgres?  I tried using various tools and was unsuccessful in seeing any performance improvements.  I was following the info here: https://medium.com/@bfirsh/squeezing-every-drop-of-performance-out-of-a-django-app-on-heroku-4b5b1e5a3d44

Have any suggestions/tips?  django blocking while waiting for postgres is a huge bottleneck for my app.

Thanks!

How did you go about using psycogreen.gevent for postgresql async behavior?  I'm 

[James Addison]

If you're following that 'medium' article, then you'll see it uses https://github.com/jneight/django-db-geventpool, which has instructions for using `psycogreen` with gunicorn... I believe what it states is correct.

[Shaggy]

and how it is going ?

is there some interest from django devs?

On Monday, 4 June 2018 15:18:23 UTC+2, Andrew Godwin wrote:

[Chris Barry]

Hey all,

Just wondering what the future of this is looking like?

CB

[Andrew Godwin]

The DEP is drafted and in the DEPs repo, and awaiting approval by the freshly-elected Technical Board once I submit it. In the meantime, we landed the ASGI patch, as well.

Andrew

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6ea76507-4041-4850-ac6c-bb13a09af941%40googlegroups.com.