Django performance issue that's troubling

[Jim Illback]

I have a Django app in development that performed perfectly with about 20 client records. Each client could have one or more support records and there are 40 potential support records, connected to the client with a ForeignKey (client_id). Then, I loading about 37K client records. Search queries perform almost instantaneously. A paginated list of support records with 40 pages and 10 clients per page comes back in half a second. Great performance.

However, when I click on any record in the list to go to update a specific support record for that specific client (it uses the PK in the URL so is a direct call to the UpdateView routine), I get about 10 second response times and these kind of statistics from Django-debug-toolbar:

SQL: 5 queries, 20ms (so clearly not a DB related performance issue)

TIME:  Resource usage:

User CPU time 8827 msec

System CPU time 137 msec

Elapsed time 9003 msec

Context switches 1 voluntary, 9639 involuntary

Browser timing:

domLoading 9026 (+52130) msec

CACHE: 0 calls in 0.00ms (so clearly not a cache issue either)

Here are some details: Django 2.1, python 3.6.3

Hardware: iMac with 32GB, i7 quad core

Installed apps: Bootstrap3 v9.1.0, crispy forms v1.7.0, easyPDF v0.1.1, registration-redux v2.2, Pillow v4.3.0

Here’s one of the specific UpdateView routines being executed - there are only 5 fields in the AddictionForm plus the prime key (no image or file attachments).

class AddictionUpdate(PermissionRequiredMixin, UpdateView):
    model = Addiction
    form_class = AddictionForm
    template_name = 'addiction_update.html'
    permission_required = 'clientapp.change_addiction'

The slow response appears to be on UpdateView pages only, even though all updates are called with the prime key (ID) in the URL. Can anyone give me guidance where to look for improving the performance? Why all the involuntary context switches? Why the large domLoading time?

Thanks in advance -

Jim

[Vijay Khemlani]

Maybe your update form has a field related to a model with thousands of records? like rendering a select combo box with thousands of choices?

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/76F251D2-A429-4700-9F69-053673BDB91F%40hotmail.com.
For more options, visit https://groups.google.com/d/optout.

[Jim Illback]

Vijay, you are exactly correct. Thank you much!

Now, knowing the problem, I still don’t have a great solution. Here’s how it is supposed to work.

Someone chooses a client from the list or a search. All the possible support records are shown, whether they exist or not, on a screen to hyperlink to one of them. If the record exists, the URL includes the PK and transfers to the UpdateView. This is similar for an AddView if the record does not exist. However, in the Update or Add, I don’t want the primary key changed. So, I hid the field (and thus all the records were read). However, now I can’t limit that field to only one value. If I format the entire SELECT tag with only one OPTION, Django gives an invalid expression for the ID or Name because it is based on my query set I passed in that fills out the values (ID and name). If I use the form’s field value, all the fields are read. Especially on an Add, I don’t know a way to fill the client ID field in such a way that Django will take it as valid.

My thoughts on possible solutions:

1. Override the get query set method to only pass in the single record - will that work?

2. I’ve been trying to show the field in a <p> tag and override the View’s post method by aftering is_valid(), but it always returns invalid expression errors.

What am I missing?

Thanks much again -

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALn3ei2korG5V3dDcCUQuKiAhG6tzAz%3DJZssNpGJpkAOyKAXpQ%40mail.gmail.com.

[Vijay Khemlani]

To tell you the truth I couldn't understand some parts of your message, but if your AddictionForm is a ModelForm then you should be able to limit its fields to just the ones you want your users to be able to update.

https://docs.djangoproject.com/en/2.1/topics/forms/modelforms/

Regards!

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CE5F73B1-38EC-435B-A7B8-61F143B5C1F2%40hotmail.com.

[Vijay Khemlani]

Please reply to the user group

"But, in order to have the update work, it needs the ForeignKey to the parent in the form"

As far as I know, it doesn't need to (if the entry already exists in the database and you are just editing it), you just need to set up the fields that you want to be editable by the user

On Fri, Aug 24, 2018 at 2:06 PM Jim Illback <suba...@hotmail.com> wrote:

Vijay, think of this is a parent - child relationship. I have 37K parents. Now I want to update a child of one of the parents. But, in order to have the update work, it needs the ForeignKey to the parent in the form. However, to add that key, Django must then allow it to update that field and thus has to fill in all 37K entries. Is there no way to tell Django that the FK relationship can not be changed? If I leave it off of the form, the update fails because it is a required field. If I add it to the form, Django has to allow all potential update values to be shown. What’s the way out of this mess?

Thanks,

Jim

Begin forwarded message:

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALn3ei3%2B5NV3Hb5rxO-VYYGbfxRzo1Fi8-mJbXLDFFVpYTrAEg%40mail.gmail.com.

[Matthew Pava]

You can keep the field, but change its widget to a HiddenInput.

https://stackoverflow.com/questions/6862250/change-a-django-form-field-to-a-hidden-field

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALn3ei2LYt6JFBXUUCVNRAer-LdL69q9y2_Muxk-mmah1z0bAw%40mail.gmail.com.

[Jim Illback]

Matthew, thanks - that was my original code, but even hidden, it has to fill in all possible values - 37K - so it takes multiple seconds.

Vijay, I’ll try your suggestion.

Thanks much,

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/2aafdf9d6360491a83da7d5839234e68%40ISS1.ISS.LOCAL.

[Matthew Pava]

It’s important that you set the widget to a HiddenInput.  It only has to render it with the current value.  If you are keeping the widget to be a Select widget and then setting a CSS class to make it hidden, the system will still have to load all 37K values.  I guess I’m missing something in your explanation.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/25286DDB-941D-408E-96E4-85D634A9EC5C%40hotmail.com.

[Jim Illback]

Vijay, your suggestion worked. I had tried this previously but it was using a FBV and it didn’t work. But with this CBV app, it works perfectly. Thanks.

Matthew, I’ll definitely keep your suggestion in mind - maybe back on my FBV application if I refactor!

Thanks much to both of you!

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/84e8c6364ac8455285b5a13f75e0044c%40ISS1.ISS.LOCAL.

[Tim Vogt (Tim Vogt)]

Also found this perhaps helpfull

https://blog.labdigital.nl/working-with-huge-data-sets-in-django-169453bca049

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/FF1B4F5D-AE22-400E-9CB9-2820AF131749%40hotmail.com.

[Jim Illback]

Thanks much, Tim. I’ll check it out - probably will need even more tuning down the road...

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/1C757A10-580D-4ECC-B8E9-B1657BB83687%40gmail.com.