Customizable serialization. (Finally?)

[Tom Christie]

Hi all,

  Over the coding sprints at djangocon.eu I started working on a proposal for customizable serialization. [1]

I've talked over the API and implementation with Russ, and I think we're broadly happy with it all,

so I'd like to open it up to wider discussion.

  What I have right now looks something like this:

1) Serialization is a two stage process - firstly serializing into native datatypes, then rendering into json/xml/whatever.

2) Serialization behavior is specified by inheriting from a Serializer class and setting various attributes such as 'fields', 'depth' etc... on an inner 'Meta' class.

3) Serialization of a given field on a model can be overridden by specifying a method on the Serialization class.

4) Behavior can also be refined by overriding various public methods on the Serializer class, if required.

5) Multiple Serialization classes can be composed together, to provide for more complex behavior.

Here's an example of the proposed API, that replicates the existing json serialization behavior, without any of the implicit assumptions about serialization structure that are currently made:

class PKOnlySerializer(Serializer):

    """

    Serializes models into their pk representation.

    """

    def serialize_model(self, instance):

        return instance.pk

class FieldsSerializer(Serializer):

    """

    Serializes only the fields on a model.

    """

    class Meta:

        exclude = ('id', 'pk')

        related_serializer = PKOnlySerializer

class DjangoSerializer(Serializer):

    """

    Serializes into the existing django style.

    """

    class Meta:

        fields = ('pk', 'model', 'fields')

    

    def model(self, instance):

        return unicode(instance._meta)

    def fields(self, instance):

        return FieldsSerializer().serialize(instance)

Which can be used like this:

    data = DjangoSerializer().serialize(queryset)

    ret = json.dumps(data, cls=DateTimeAwareJSONEncoder, indent=4, sort_keys=False)

The code as it stands is available on github [2].

You'll want to look at impl.py [3] for the implementation and views.py [4] for example API usage.

The serializer currently supports depth-limited and recursion-limited output, and I'm right now I'm looking to work on finalizing the xml related issues,

adding some slightly more concise ways of being able to describe nested models [Russ: I think this'll be closer to what you were looking for], and then

trying to refactor Django to use the new serialization code, and see what test breakage I come up against.

Would def welcome any constructive criticism of the API or implementation.

Also if anyone wants to help in contributing in order to get this (long outstanding) feature into Django, that would be very much appreciated.

(I suspect there might be quite a lot to do once I look at merging into Django!)

It's not totally impossible that we might be able to get this feature in for 1.4, which'd be pretty cool,

and there's also a bunch of serialization related tickets that we might be able to close off in the process. [5]

Feedback?

  t.

[1] https://code.djangoproject.com/wiki/SummerOfCode2011#Customizableserialization

[2] https://github.com/tomchristie/scratch

[3] https://github.com/tomchristie/scratch/blob/master/myapp/impl.py

[4] https://github.com/tomchristie/scratch/blob/master/myapp/views.py

[5] https://code.djangoproject.com/query?status=assigned&status=new&status=reopened&component=Core+(Serialization)&order=priority

[sebastien piquemal]

Hi !

Looks like a great idea to me. I had already proposed something
similar when there were talks for implementing this for SOC.
Like I had said in this thread, I have already implemented something
that works a similar way as what you propose :
http://readthedocs.org/docs/any2any/en/latest/doc_pages/djangocast.html
. And I would be happy to help with implementing that feature for
Django, considering that I have already dealt with most of the
problems that this approach bring.

A suggestion though, is to be able to declare serializers in a similar
fashion as models or forms. For example :

class FieldsSerializer(Serializer):
"""
Serializes only the fields on a model.
"""

some_fk = PKOnlySerializer() #Overrides default serializer of
field `some_fk`
some_char_field = AnOtherCustomSrz() #Overrides default
serializer of field `some_char_field`

class Meta:
exclude = ('id', 'pk')

That would things more consistent no !?

> [1]https://code.djangoproject.com/wiki/SummerOfCode2011#Customizableseri...

> [2]https://github.com/tomchristie/scratch
> [3]https://github.com/tomchristie/scratch/blob/master/myapp/impl.py
> [4]https://github.com/tomchristie/scratch/blob/master/myapp/views.py

> [5]https://code.djangoproject.com/query?status=assigned&status=new&statu...

[Tom Christie]

> A suggestion though, is to be able to declare serializers in a

> similar fashion as models or forms.

Actually that's a fair point, yup - the proposal could be made a little closer to the existing Forms and Models APIs.

Something along these lines...

class ModelPKField(SerializerField):

    def get_value(self, obj, key):

        return unicode(obj.pk)

class ModelNameField(SerializerField):

    def get_value(self, obj, key):

        return unicode(obj._meta)

class ModelFieldSetField(SerializerField):

    def get_value(self, obj, key):

        return FieldsSerializer().serialize(obj)

class RelatedPKField(SerializerField):

    def get_value(self, obj, key):

        return unicode(getattr(obj, key).pk)

class FieldsSerializer(Serializer):

    """

    Serializes only the fields on a model.

    """

    class Meta:

        exclude = ('id', 'pk')

        default_related_field = RelatedPKField()

class DjangoSerializer(Serializer):

    """

    Serializes into the existing django style.

    """

    pk = ModelPKField()

    model = ModelNameField()

    fields = ModelFieldSetField()

[mofle]

Looks really good, the current implementation for serialization is
pretty limited, so this will be a breath of fresh air :)

Using the proposed Serializations class, will it be possible to
serialize properties of a model that is not fields, for example
methods?

[Tom Christie]

Yup.

The current implementation (see the github repo) allows properties and methods that only take a 'self' argument to be serialized by adding the name to 'fields' or 'include'.

If you need anything more involved then you'd write a custom method on the Serializer class. (or a custom SerializerField if the final API uses that style instead.)

[fas]

Hi! I put your code to good use in my project and have two minor suggestions:

Make the signature of 

def get_related_serializer(self, key)

like this:

def get_related_serializer(self, key, obj): 

because the related serializer could also depend on obj (does for me).

Secondly: the naming of serialize_val(self, key) is a bit unfortunate, I was confused by it multiple times. Maybe it should be made clear that this is used for serialize_model (and not something arbitrary). Also, key should maybe be named fname to be consistent (it is not the final serialized key, after all).

Lastly:

if any([obj is elem for elem in self.stack]):

 

Didn't you just redefine the 'in' operator?

if obj in self.stack:

 

Thanks for your work!

[Tom Christie]

Cool, thanks, all good points.

Slight haitus working on this right now whilst I try to get my head around the right way to try to bring it more into line with the Forms API, but hoping to pick it up again once I've marshaled my thoughts.

Obviously I'll keep the list updated if I do make more headway, and if anyone's thinking about contributing to this / sprinting at DjangoCon US or whatever, then please do give me a shout.

And, slightly OT, but...

> Didn't you just redefine the 'in' operator?

Not quite:

>>> a={}

>>> b={}

>>> collection=[a]

>>> b in collection

True

>>> any([b is elem for elem in collection])

False