Slow page load performance with a large number of formsets (over 25)

[Jason]

We are new to the Django framework and are currently building an application in PythonAnywhere. We have a couple of html pages where the volume of formsets that we are loading up varies from 25 to 100. We find that when we use a small number of formsets, the page loads very quickly. However, performance seems to degrade as the numbers increase. 

It's not an issue with our SQL queries as when we load the same data into a HTML table instead of using formsets, it loads very quickly. 

We need to load the data using formsets because we expect updates to each row from the user.

Has anyone encountered this problem? If so, is there a way to improve the performance of the formsets or are formsets not intended to be used with large volumes of data? If not, are there alternative options that we can use that allow users to modify the data in multiple rows returned by the query? Note - we don't want to send the user to a 'Pop-Up' or another screen to update data. Note 2 - We are using "Model" formsets, in particular "inlineformset_factory".

[bobhaugen]

We have had the same problem, but with a large number of forms. I have profiled the situation and the load time is dominated by form rendering on the server. 

I remember vaguely something about caching forms so they don't need to be re-rendered, but can't find it now. And I don't understand the tradeoffs anyway. (Maybe somebody does?)

We're looking at moving away from Django forms to passing Json back and forth between Javascript thingies. Recently used http://handsontable.com/ to replace a formset. Much faster, much more flexible. You lose some of the niceties of model forms, need to do a lot of javascript validations which can be cranky, and of course need to do all of your updating in procedural code in views.  As always, tradeoffs.

[Collin Anderson]

Yes, if you want speed, using javascript and ajax is probably your best bet. It will probably also reduce merge-conflicts.

[Steven Smith]

Did this issue ever get resolved?  I'm experiencing the same thing.  Once it hits 100 forms or so it gets pretty slow.

[Abdulrahman Abbas]

Try hard reload it's little bit faster

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/48609393-cb83-4eea-845b-d353e41af1e7n%40googlegroups.com.

[Steven Smith]

What is hard reload?

[Ryan Nowakowski]

Steven, are you using ModelFormSet? If so, Is it the initial query
that's slow? The form rendering? Or is it the POST back to the server
that's slow? You can use django-debug-toolbar[1] to profile your page
and get these metrics.

Once you figure out what part is slowest, then you can optimize.

Hope this helps!

Ryan N

[1] https://django-debug-toolbar.readthedocs.io/en/latest/

[Jacob Greene]

If you're using model choice fields, it's very likely you are making an SQL query for each form set. It should be pretty obvious in the log with debug turned on. 

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/20220225222926.GH11627%40fattuba.com.

[Steve Smith]

If in fact I am….how do I avoid this?  I have been playing with alternatives for a week and I’m starting to lose it…lol

 

Sent from Mail for Windows

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAF-Y%3De5Fp7kDtLq47Z3Y4GEjXMne2PcTLYxOJ8St5gkKCkpUpw%40mail.gmail.com.

 

[Jacob Greene]

You would need to cache the queryset and pass it to each form instance that the formset creates. I've done this a few different ways in the past. I've never understood why there isn't a more straight forward way to do this. Here's a post with a few legit options depending on your use case:

https://stackoverflow.com/questions/8176200/caching-queryset-choices-for-modelchoicefield-or-modelmultiplechoicefield-in-a-d

Hard to be more specific without knowing anything about your code.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CH0PR18MB42923C9432F56E071FA9716CD23F9%40CH0PR18MB4292.namprd18.prod.outlook.com.

[Steve Smith]

Thanks for the suggestion.  I really appreciate it.

Steve

---

From: django...@googlegroups.com <django...@googlegroups.com> on behalf of Jacob Greene <jacobgr...@gmail.com>
Sent: Friday, February 25, 2022 9:43 PM
To: django...@googlegroups.com <django...@googlegroups.com>

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAF-Y%3De4E3zFuzmK_UOf9exaEc5pM5FNun7xJXhn_yVbHyWqiWg%40mail.gmail.com.

[Antonis Christofides]

However, caching the queryset solves only part of the problem.

If you have a formset with 100 forms, and each of these forms has a choice field with 100 choices (say 50 bytes each), then you have altogether 10k choices or 500k bytes of data. While, depending on the particular case, this may be manageable, it can get quickly out of hand if the forms become 200 or the choices become 200 or more etc.

A technical solution to that problem could be to retrieve the choices only once and have some JavaScript fill in the choices in the forms. But this departs from the Django formset paradigm and might require significant development.

The alternative I see would be to redesign the UI so that it doesn't need a formset (at least such a big formset). While I don't know the specifics of this particular case, I would hack my brains to see whether I can present this information in a more simple manner to the user (because 100 forms would be overwhelming I guess). Once I have a saner UI, the implementation details would follow naturally.

Antonis Christofides
+30-6979924665 (mobile)

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAF-Y%3De4E3zFuzmK_UOf9exaEc5pM5FNun7xJXhn_yVbHyWqiWg%40mail.gmail.com.

[Carlos R. Pacheco F.]

The solution is to use select2 with ajax

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAF-Y%3De5Fp7kDtLq47Z3Y4GEjXMne2PcTLYxOJ8St5gkKCkpUpw%40mail.gmail.com.