Optimizing admin change view with inlines

[Petros Moisiadis]

Hello people :)

I am struggling with optimizing an admin with inlines which causes too
many database requests.

My model structure is like this:

class ExampleA(models.Model):
...

class ExampleB(models.Model):
aexample = models.ForeignKey('ExampleA', related_name='bexamples')
cexample = models.ForeignKey('ExampleC')
dexample = models.ForeignKey('ExampleD')
eexample = models.ForeignKey('ExampleE')
...

class ExampleC(models.Model):
...

class ExampleD(models.Model):
...

class ExampleE(models.Model):
...

The admin classes:

class ExampleBInline(admin.StackedInline):
model = ExampleB
extra = 0

class ExampleAAdmin(admin.ModelAdmin):
inlines = [ExampleBInline]

admin.site.register(ExampleA, ExampleAAdmin)

As I can see with django-debug-toolbar, when rendering the admin
template for the inline formset with the forms for ExampleB objects, a
new db request is sent to db server for each related field of each
ExampleB object. Particularly, I am seeing a lot of queries like these:

SELECT ••• FROM `examplec` WHERE `examplec`.`id` = 1 LIMIT 21
SELECT ••• FROM `examplee` WHERE `examplee`.`id` = 2 LIMIT 21
SELECT ••• FROM `examplec` WHERE `examplec`.`id` = 2 LIMIT 21
SELECT ••• FROM `exampled` WHERE `exampled`.`id` = 2 LIMIT 21
SELECT ••• FROM `examplee` WHERE `examplee`.`id` = 3 LIMIT 21

The template context is this (I am using grappelli):

21 {% if field.is_readonly %}
22 <div class="grp-readonly">{{
field.contents|linebreaksbr }}</div>
23 {% else %}
the marked line => 24 {{ field.field }}
25 {% endif %}
26 {% endif %}
27 {% if line.fields|length_is:'1' %}{{
line.errors }}{% endif %}


I have tried the following optimizations:

First try:

class ExampleAAdmin(admin.ModelAdmin):
inlines = [ExampleBInline]

def get_queryset(self, request):
qs = super(ExampleAAdmin, self).get_queryset(request)
return qs.prefetch_related('bexamples',
'bexamples__cexample', 'bexamples__dexample', 'bexamples__example')

Second try:

class ExampleBInline(admin.StackedInline):
model = ExampleB
extra = 0

def get_queryset(self, request):
qs = super(ExampleInline, self).get_queryset(request)
return qs.select_related('cexample', 'dexample', 'eexample')

Third try:

class BaseExampleBFormSet(BaseInlineFormSet):
def __init__(self, *args, **kwargs):
super(BaseExampleBFormSet, self).__init__(*args, **kwargs)
qs = self.queryset
self.queryset = qs.select_related('cexample', 'dexample',
'eexample')

ExampleBFormSet = inlineformset_factory(ExampleA, ExampleB,
formset=BaseExampleBFormSet)

class ExampleBInline(admin.StackedInline):
model = ExampleB
extra = 0
formset = ExampleBFormSet

Unfortunately, none of the above works.

So, I would be really grateful if anyone could help on this by giving
any hint or pointing out what could be possibly missing. Or, could I
have hit a restriction of django admin's internals?

Thank you in advance,

Petros

[Petros Moisiadis]

Hello again,

I would like to clarify that the big number of db requests is not caused
by fetching all the objects for the related model to populace select
boxes for the related fields in the line, as it is often the case. To
avoid that, I have included the related fields in raw_id_fields. I have
also used grappelli's convenient autocomplete lookup feature, but that
does not add extra queries. So, the inline class actually looks like this:

class ExampleBInline(admin.StackedInline):
model = ExampleB
extra = 0

raw_id_fields = ['cexample', 'dexample', 'eexample']
autocomplete_lookup_fields = {
'fk': ['cexample', 'dexample', 'example'],
}

The only optimization that works is adding the related fields in
readonly_fields in combination with a queryset changing method with
select_related(), like this:

class ExampleBInline(admin.StackedInline):
model = ExampleB
extra = 0

raw_id_fields = ['cexample', 'dexample', 'eexample']
autocomplete_lookup_fields = {
'fk': ['cexample', 'dexample', 'eexample'],
}
readonly_fields = ['cexample', 'dexample', 'eexample']

def get_queryset(self, request):
qs = super(ExampleBInline, self).get_queryset(request)

return qs.select_related('cexample', 'dexample', 'eexample')

However, I am not satisfied with that solution because I do want to be
able to edit the related fields inline.

[Petros Moisiadis]

So the question merely is why the inline class right above produces so
many queries if the 'readonly_fields' line is commented out?
For my data, according to debug toolbar, the view causes 11 queries with
readonly_fields and 115 queries without readonly_fields.

[Collin Anderson]

Hi Petros,

I think the raw_id_fields still display the __str__/__unicode__ of the selected object so therefore need to get queried. I wonder if select_related would possibly help?

Collin

[Petros Moisiadis]

Hello Collin,

On 12/29/14 07:25, Collin Anderson wrote:

Hi Petros,

I think the raw_id_fields still display the __str__/__unicode__ of the selected object so therefore need to get queried. I wonder if select_related would possibly help?

I do have select_related in the queryset (look at the last version of ExampleBInline in my previous post).

The thing is that if the same fields that are included in raw_id_fields are also included in readonly_fields, the queries are reduced dramatically, despite the fact that the same amount of information is displayed in both cases. __str__()/__unicode__ is still called when the fields are in readonly_fields, so it is really very strange why there is difference between adding and commenting out the readonly_fields line.

Very strange...

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/778f1f33-3c34-491e-a062-9527ec703bfd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Collin Anderson]

Hi Petros,

I think it's because "value_from_object" only passes the primary key of the object and then raw_id_field re-queries it.

https://github.com/django/django/blob/40ccef16/django/contrib/admin/widgets.py#L195

https://github.com/django/django/blob/40ccef16/django/forms/models.py#L151

https://github.com/django/django/blob/40ccef16/django/db/models/fields/__init__.py#L844-L848

Basically, even if you use select_related(), django still queries your object again. It looks like a spot where there could be some optimization.

Collin