django-filter slow performance

[Abdulhaq Elhouderi]

Hi

Assume we have the following FilterSet and ModelViewset

class StudentFilter(django_filters.FilterSet):

    registration_number = django_filters.CharFilter()

    class Meta:
        model = Student
        fields = ('registration_number',)

class StudentViewSet(viewsets.ModelViewSet):

    queryset = Student.objects.all()
    serializer_class = StudentSerialiser
    filter_class = StudentFilter

For a large table, `GET /api/students/?registration_number=1234` takes an extremely long time (it really depends on what is included in the serialiser). However, this is an alternative viewset:

class StudentViewSet(viewsets.ModelViewSet):

    queryset = Student.objects.all()
    serializer_class = StudentSerialiser
    # filter_class = StudentFilter

    def get_queryset(self):
 
    registration_number = self.request.query_params.get('registration_number', None)
     
    if registration_number is not None:
        queryset = Student.objects.filter(registration_number=registration_number)
    return queryset

These are examples are almost a copy of the ones in the documentation. The second viewset is extremely fast. compared to the first. In fact, for one table, the second one returns the results in 3 seconds while using django-filter I don't get anything for more than 3 minutes!! (at that point I just cancel the request). It doesn't make a difference if I add queryset = Student.objects.all() before the if statement (just like the docs)

I think django-filter somehow evaluates Student.objects.all() which takes a long time. Am I correct about this? If so, is there a way to stop this behaviour and if not, why is django-filter slower?

[Xavier Ordoquy]

Hello,

The first step to fix performance issues is to get some facts.

In particular, it would be helpful to log the queries done with the django filter.

Your second view is totally valid. The speed improvement is possible because there are assumptions relating to the expected filtering.

It would really be helpful to see the SQL requests in order to improve the django filters.

Regards,

Xavier,

Linovia.

--
You received this message because you are subscribed to the Google Groups "Django REST framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-rest-fram...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Carlton Gibson]

Hi.

This is curious. If you can pin down where the slow query is coming from (and it’s related to Django Filter) please open an issue on GitHub, so I can review.

https://github.com/alex/django-filter

Thanks

Regards,

Carlton

[Abdulhaq Elhouderi]

 

I solved the issue, It doesn't have any direct relation to django-filter but it is interesting!

So why two similar requests differ in response time, 135 seconds compared to 0.3 seconds: ...drum rolls...: it happens because 'DEFAULT_FILTER_BACKENDS' is not set!

I have no idea how the filtering works without a filtering backend, I would appreciate it if someone can explain what is happening?

Regards

Abdulhaq

[Xavier Ordoquy]

> Le 3 sept. 2015 à 23:19, Abdulhaq Elhouderi <el.inge...@gmail.com> a écrit :
>
>
> I solved the issue, It doesn't have any direct relation to django-filter but it is interesting!
>
> So why two similar requests differ in response time, 135 seconds compared to 0.3 seconds: ...drum rolls...: it happens because 'DEFAULT_FILTER_BACKENDS' is not set!

Which one has what setting ?
By saying it’s not set, you mean you set it to empty or left it to the default rest framework value ?

> I have no idea how the filtering works without a filtering backend, I would appreciate it if someone can explain what is happening?
>
> Regards
> Abdulhaq

Regards,
Xavier.

[Abdulhaq Elhouderi]

The first case is the one which requires 'DEFAULT_FILTER_BACKENDS' to be set, it's indicated in the docs 

I left it to the default drf settings which is an empty tuple.

In [15]: from rest_framework.settings import DEFAULTS
         DEFAULTS['DEFAULT_FILTER_BACKENDS']
Out[16]: ()

The second case which doesn't use django-filter isn't affected by the filter backends settings because it's not using 'generic filtering'

Regards

[Tom Christie]

> I have no idea how the filtering works without a filtering backend, I would appreciate it if someone can explain what is happening?

Answer - absolutely nothing happens when there's no filtering backend - `filter_queryset` just returns the queryset directly.

https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/generics.py#L150-L152

Seems likely that there's some other issue at play in your codebase.

[Abdulhaq Elhouderi]

Sorry about all of this. With no filtering backend, the complete queryset is returned which is why it's taking so long. I had that confused with the filtered queryset!!

Anyway, so that we don't go away empty-handed, what do you think of this 

        if not self.filter_backends and (self.filter_class is not None or
                                         self.filter_fields is not None):
            warnings.warn(
                "You have set 'filter_class' or 'filter_fields' on the view "
                "without specifying any 'filter_backends'. "
                "No filtering will be used.",
            )
        else:
            for backend in list(self.filter_backends):
                queryset = backend().filter_queryset(self.request,
                                                     queryset, self)

        return queryset

Regards

Abdulhaq

[Asif Saifuddin]

https://github.com/philipn/django-rest-framework-filters how is it?