DRF ModelViewSet ignore prefetch/select_related?

[Dedi Purwanto]

I'm using django-debug-toolbar and see there are thousands of queries made to related objects. Basically there are about 2k rows in the model, and it takes about 10 seconds for DRF's API page to load.

I've tried using all possible combinations of prefetch/select_related but they don't work. Here's my code:

    class A(models.Model):
        b = models.ForeignKey(B)


    class B(models.Model):
        pass


    class MyViewSet(viewsets.ModelViewSet):
        serializer_class = ASerializer
        queryset = A.objects.all().select_related('b')
        pagination_class = pagination.PageNumberPagination


        def get_queryset(self, *args, **kwargs):
            return super(MyViewSet, self).get_queryset(*args, **kwargs).select_related('b')


    class ASerializer(serializers.ModelSerializer):


        class Meta:
            model = A

The biggest culprits are the pagination class and the serializer. However, I couldn't find anything suspicious in DRF's built-in pagination class, while waiting for the page to load, I put some print statement on all places in the pagination class and they are all printed immediately, so definitely that's not the bottle neck.

Next suspicion is the ModelSerializer, I am not exactly sure what's happening. Originally, debug-toolbar shows there are around 1k queries for every page with 8 items per page, and then I did this:

    class MyViewSet(viewsets.ModelViewSet):
        serializer_class = ASerializer
        queryset = []
        pagination_class = pagination.PageNumberPagination

To my surprise, although it's now showing 0 items, but debug-toolbar still shows there are ~1k queries and it still takes a long time to load. Now if I change the serializer class into some other model which has far lesser rows:

    class MyViewSet(viewsets.ModelViewSet):
        serializer_class = BSerializer
        queryset = []
        pagination_class = pagination.PageNumberPagination


    class BSerializer(serializers.ModelSerializer):


        class Meta:
            model = B

The page loads faster, and there are less queries. My only assumption now is that somehow ModelSerializer loads the "almost" the entire result set in some iteration processes, but I couldn't find it. The reason I said "almost" is because there are difference between the number of total rows (about 2k) and the number of queries reported in debug toolbar (1k). Any ideas?

[Andrew Backer]

One thing to be _very_ careful about (it has bitten me) is extra queries happening inside a model's __str__ or __unicode__ methods.  If you have code like this:

def __unicode__(self):
    return "%s - %s" % (self.company.name, self.name)

DRF will try to get the name for each one and display it.  This may be the case if you are seeing hundreds of nearly identical queries (especially for the same ID) in the debug toolbar.

~ Andrew

Dedi Purwanto

April 19, 2016 at 5:24 PM

--
You received this message because you are subscribed to the Google Groups "Django REST framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-rest-fram...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sent from Postbox

[Xavier Ordoquy]

Could also be the create / update form if you’re using the browsable API, esp if it didn’t return results.

Regards,

Xavier,

Linovia.

[Andrew Backer]

Xavier,

I should have mentioned that I saw this issue in the edit forms specifically, and sometimes in the admin.  We don't ever repr/str/unicode our objects so I didn't notice it before, but the dropdown in the browsable api create/update form would try to load 100 "company" records (for example) and then str() each one

Xavier Ordoquy

April 19, 2016 at 7:03 PM

Could also be the create / update form if you’re using the browsable API, esp if it didn’t return results.

Regards,

Xavier,

Linovia.

--
You received this message because you are subscribed to the Google Groups "Django REST framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-rest-fram...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Andrew Backer

April 19, 2016 at 6:19 PM

Sent from Postbox