Paginate related fields

[Deepak Prakash]

Hello people,

We are using DRF for our API & loving it so far! However, we have a need to paginate relationship fields that return multiple items - couldn't find a way to do this.

To demonstrate using examples similar to those in the documentation:

class TrackSerializer(serializers.ModelSerializer):
    class Meta:
        model = Track
        fields = ('order', 'title')

class AlbumSerializer(serializers.ModelSerializer):
    tracks = TrackSerializer(many=True)

    class Meta:
        model = Album
        fields = ('album_name', 'artist', 'tracks')

Example serialized output for an Album:

{
    'album_name': 'The Grey Album',
    'artist': 'Danger Mouse'
    'tracks': [
        {'order': 1, 'title': 'Public Service Annoucement'},
        {'order': 2, 'title': 'What More Can I Say'},
        {'order': 3, 'title': 'Encore'},
        ...
    ],
}

This becomes problematic where there are hundreds of tracks in a single Album. Is there a way to paginate the 'tracks' in this case?

Ideally, in cases like this, the 'tracks' should point to an API URL that just returns the Tracks for this particular Album - which in turn can be paginated easily. The down side to that approach being the extra request (and hence delay, etc) required to get even the first few tracks. In our case, its important that we be able to get at least a few of the Tracks with the single request to the Album API and then dynamically load the rest of the tracks as required.

Does the DRF offer any specific feature or pattern for this? Or are there any work arounds?

==================================

In reply to the Github issue that I created for this, Tom suggested:

Hiya,

You use case isn't something that I've done myself, but it does makes sense and I think you'd be able to get that working just fine by using a paginated serializer for the tracks field.

Take a look at this section in the pagination docs... http://django-rest-framework.org/api-guide/pagination.html#paginating-querysets

class TrackSerializer(serializers.ModelSerializer):
    class Meta:
        model = Track
        fields = ('order', 'title')

class PaginatedTrackSerializer(pagination.PaginationSerializer):
    class Meta:
        object_serializer_class = TrackSerializer

class AlbumSerializer(serializers.ModelSerializer):
    tracks = PaginatedTrackSerializer()

    class Meta:
        model = Album
        fields = ('album_name', 'artist', 'tracks')

Failing that, you might have more chance of a useful response over on the mailing list... https://groups.google.com/forum/?fromgroups#!forum/django-rest-framework

Unfortunately, that doesn't work and fails with the following error:

Traceback:

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response

  111.                         response = callback(request, *callback_args, **callback_kwargs)

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/django/views/generic/base.py" in view

  48.             return self.dispatch(request, *args, **kwargs)

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view

  77.         return view_func(*args, **kwargs)

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/rest_framework/views.py" in dispatch

  393.             response = self.handle_exception(exc)

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/rest_framework/views.py" in dispatch

  390.             response = handler(request, *args, **kwargs)

File "/Users/deepak/Code/GitHub/zoomdeck_dev/zoomdeck/apps/core/api/views.py" in get

  137.         return self.retrieve(request, *args, **kwargs)

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/rest_framework/mixins.py" in retrieve

  80.         return Response(serializer.data)

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/rest_framework/serializers.py" in data

  384.                 self._data = self.to_native(obj)

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/rest_framework/serializers.py" in to_native

  274.             value = field.field_to_native(obj, field_name)

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/rest_framework/serializers.py" in field_to_native

  320.             return [self.to_native(item) for item in obj.all()]

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/rest_framework/serializers.py" in to_native

  274.             value = field.field_to_native(obj, field_name)

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/rest_framework/fields.py" in field_to_native

  106.             value = get_component(value, component)

File "/Users/deepak/.virtualenvs/zmdk/lib/python2.7/site-packages/rest_framework/fields.py" in get_component

  46.         val = getattr(obj, attr_name)

Exception Type: AttributeError at /api/v2/images/jf1mxb59oic9/

Exception Value: 'Track' object has no attribute 'paginator'

I'm guessing that the custom paginated serializer expects to find a paginator object on the object being serialized. Not sure if I'm missing something or if this approach will not work at all.

Any help or workaround will be appreciated!

Thanks a lot,

Deepak Prakash.

[Tom Christie]

Hi Deepak,

I realized I missed out a step when I posted that example up.  As it stands the nested `PaginatedTrackSerializer` is being passed a list of tracks, rather than being passed a *paginated* list of tracks.  You should be able to address that fairly easily.

In your `Album` model, you'd want to add a method that returns the first page in a paginated set of tracks...

    def paginated_tracks(self):

        page_size = 20

        paginator = Paginator(self.tracks, page_size)

        return paginator.page(1)

You'll then want to tweak your serializer definition slightly, to use `paginated_tracks` instead of `tracks`...

    tracks = PaginatedTrackSerializer(source='paginated_tracks')

I think you'd still have a little bit more work to do after that because the `next` and `previous` links will be pointing to something like `http://testserver/album/123/?page=2`, you'd need to look at creating a custom pagination serializer to handle those links differently.

> Ideally, in cases like this, the 'tracks' should point to an API URL that just returns the Tracks for this particular Album - which in turn can be paginated easily

That'd certainly be much simpler, yup :).  (There's an example in the docs of that style, here)

You will of course still need that extra endpoint in order to paginate through the remainder of the items.

This is a reasonably complex use-case you've got here, so it's entirely possible I've missed something else!

Hope that helps,

  Tom :)

[Deepak Prakash]

Tom,

Thanks a bunch!

In fact, saw your mail just as I finished implementing something similar, but putting "paginated_tracks" in the AlbumSerializer and then associating it as a SerializerMethodField. Also took care of implementing the custom pagination serializer inside this.

class TrackSerializer(serializers.ModelSerializer):
    class Meta:
        model = Track
        fields = ('order', 'title')

class PaginatedTrackSerializer(pagination.PaginationSerializer):
    class Meta:
        object_serializer_class = TrackSerializer

class AlbumSerializer(serializers.ModelSerializer):

    tracks = serializers.SerializerMethodField('paginated_tracks')

    class Meta:
        model = Album
        fields = ('album_name', 'artist', 'tracks')

    def paginated_tracks(self, obj):
        paginator = Paginator(obj.tracks.all(), 10)
        tracks = paginator.page(1)

        serializer = PaginatedTrackSerializer(tracks)
        return serializer.data

This seems to work well. Any suggestions/improvements/issues with this?

Thanks again,

Deepak.

--
You received this message because you are subscribed to a topic in the Google Groups "django-rest-framework" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-rest-framework/UtVRH8mHwmU/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to django-rest-fram...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.