Poor Performance for Form Rendering In Django 1.11

[John]

My site has a navbar with an advanced search widget (beside the search field), which renders on every page. For each request, a context_processor creates the form so it can be available on that page in the navbar. This form has about a dozen selects with a total of several hundred options. Most of those options are for the currency and country selects, along with about 80 other options. There is an even larger list for "stores" but it is loaded via AJAX so it should not be a factor here.

Performance was fine on Django 1.8, but after upgrading to 1.11 I noticed with NewRelic that over 500 ms are now being used on my most frequent request between the following:

• Render/django/forms/widgets/select_option.html
• Render/django/forms/widgets/select.html
• Render/django/forms/widgets/attrs.html

This seems to be related to 1.11's change to Template-based Widget Rendering (docs), however the only pages I could find talking about related problems were about Django Toolbar which I do not run in production.

I am and already using the Cached Template Loader (which is now default), however I don't know if this helps here. I cannot easily cache this form because as you can see in the code, I set a number of defaults based on the request.

Why is my form suffering so badly from this change? Eliminating two of the bigger selects helps, but surely several hundred options should not take this long to render so it seems to me there is an underlying problem that the quantity is merely exacerbating.

Here are links to to code for the full form and html. (I will include snippets in the question later when we identify the problem, for future readers).

• Search Form
• Search HTML
• Live Site

Thank you in advance for your help!

[Simon Charette]

Hello John,

Did you try switching to the Jinja2 form renderer[0] to see if it helps? I've not

tried it myself but I heard it makes a significant difference.

Cheers,

Simon

[0] https://docs.djangoproject.com/en/2.1/ref/forms/renderers/#jinja2

[John Lehmann]

Hi Simon, thanks for the response.  

I am still hoping however for someone to explain to me why the default renderer cannot handle my use case, such as that a few hundred inputs is too many, or that I am doing something else improperly.  Surely this kind of a change would not be made to the framework that would cause such an unacceptable performance (500+ms added). Kind of disappointing to spend a whole week upgrading only to end up wanting to downgrade again.

I think your idea is a good workaround to explore if I cannot get any answers, and I appreciate it. I did briefly try to set this up by setting the default_renderer = renderers.Jinja2, but ran into some problems so that will take more looking into.

--
You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/uhe7ExBJoxg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b4a2a1d1-63e2-4a14-a5fa-2e75a018dadf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Bennett]

On Mon, Dec 10, 2018 at 5:31 AM John Lehmann <john.l...@gmail.com> wrote:

I am still hoping however for someone to explain to me why the default renderer cannot handle my use case, such as that a few hundred inputs is too many, or that I am doing something else improperly.  Surely this kind of a change would not be made to the framework that would cause such an unacceptable performance (500+ms added). Kind of disappointing to spend a whole week upgrading only to end up wanting to downgrade again.

The Django template language is not really optimized for speed of rendering. Even with a caching loader, rendering a template hundreds of times will be an expensive enough operation to show up in your profiling.

The solutions are to use something faster (Jinja) if you absolutely must be rendering hundreds of things, or find a way to make the form more efficient (generally, when a form has hundreds of options to display in a select, that's a sign of issues with the design of the form).

[John Lehmann]

Hi James,

Thank you for the input!  

So my takeaway from what you are saying is that no one is running a production site with a Django select field to render a country or currency option (e.g., django-countries or django-money), because either of these would have well over a hundred entries (and my form happens to have both).  Instead they would be loading these options via AJAX or using a different widget all together like an auto-complete.  (I get it that a long list is not the best UX but that's a bit of a different discussion).

It seems like a regression of this magnitude should be called out in the Django documentation for the default rendering option?  I'm also surprised neither of those mainstream libraries would mention something about this.  Adding 500+ms isn't in the realm of "tweaking" performance, for most serious sites it would be a complete fail.  So I guess I'm just a little skeptical that the real answer here is that I shouldn't expect this to work anyway. I feel like this kind of change would have affected a lot of other folks in similar significant fashion, which makes me think there is something else going on (some other kind of boneheaded thing I may be doing).

thanks,

John

--

You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/uhe7ExBJoxg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAL13Cg9amLPan9-pj%2BHpHc7vns6_5EJcvNHvF0VPakf651%3DT1g%40mail.gmail.com.

[James Bennett]

On Mon, Dec 10, 2018 at 6:29 AM John Lehmann <john.l...@gmail.com> wrote:

So my takeaway from what you are saying is that no one is running a production site with a Django select field to render a country or currency option (e.g., django-countries or django-money), because either of these would have well over a hundred entries (and my form happens to have both).  Instead they would be loading these options via AJAX or using a different widget all together like an auto-complete.  (I get it that a long list is not the best UX but that's a bit of a different discussion).

Have you profiled a country select on its own?

Your original post suggested you had multiple large (hundreds of options) selects in a single form that you were rendering. Whether any specific individual select in the form is the sole culprit, or whether it's the combination of them, is something you still haven't pinned down, and it's a bit combative to jump to "nobody must have a country select" from that.

In general, a select with a large set of options is going to be slower to render. It may turn out that this has nothing whatsoever to do with your problem, but we don't have enough information to precisely diagnose your problem; all we can do here is give you general advice like "avoid rendering a template hundreds of times if you can, or use a faster-rendering template option if you can't".

[John]

Yes, the form has about a dozen selects, with two in particular being larger - the country and the currency fields - that probably have a couple hundred options each.  I have since removed those two fields which leaves about 10 selects with about 80 options total.  While these numbers fluctuate, the 12 field form was taking a little over 750ms, and the new form with 10 fields totals 375ms. So the two larger fields accounted for about half the time, but even the reduced form is still a major problem.

Regarding the reduced version, I cannot say how much the performance issue relates to having 10 selects, or to have 80 options. But it seems like 375ms is an unreasonable amount of time for this amount of work.  (Just as it seems unreasonable to spend 400ms drawing those other two fields).

You can see the form by visiting the site and clicking on "options" next to the search bar.

I'm sorry and didn't mean to sound combative, was trying to point out that if this is the performance we expect for the default renderer, then I would have expected to see a lot more people reacting to 1.11.  Which is why I believe something else must be going on.

[Matthew Pava]

I vaguely remember having this issue when I moved to Django 1.11.  It was frustrating.  I have thus far refused to move over to Jinja, and I just use autocompletes for my large select widgets.

 

There is also the possibility of using template fragment caching.  Wrap the select in a cache block, and it will loaded into memory once.  See https://docs.djangoproject.com/en/1.11/topics/cache/#template-fragment-caching.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/7e723ec8-be37-49cf-969e-5b484e3e5b65%40googlegroups.com.

[John]

Matthew, thanks for the note.  Yes I do use fragment caching in places, though it's a bit tougher here because some of these inputs, while simple, are both populated (option sets) and initialized (defaulted) based on the user and the URL. That being said, it's probably still the easiest workaround.  Despite the large number of permutations this could create based on all the parameters, I expect that the vast majority of requests could be satisfied from a couple of cached templates.

These are the two reasons why this change hit me harder than most perhaps. First, due to the its ubiquity of the form (being present on every page of my site) and second due to its dynamic configuration (thus being harder to cache).

That being said, no Django site has more Python than mine... except maybe IG and Pinterest... though this issue is slowing it to a slither. ;)

To post to this group, send email to djang...@googlegroups.com.