[GSOC 2012] Customizable serialization
Piotr Grabowski
My name is Piotr Grabowski. I'm last year student at the Institute of
Computer Science University of Wrocław (Poland). I want to share with
you draft of my GSOC proposal.
PS. Sorry for my poor english :/
--
Piotr Grabowski
Łukasz Rekucki
> Hi,
>
> My name is Piotr Grabowski. I'm last year student at the Institute of
> Computer Science University of Wrocław (Poland). I want to share with you
> draft of my GSOC proposal.
Hi, nice to meet you :)
>
> http://pastebin.com/ePRUj5HC
IMHO, Sending the full text to the list would make it easier to
comment on specific points.
I like your approach in overall - mainly because I've been using a
similar one for a few months now ;). Some comments after a quick read:
1) The Meta.structure things looks like a non-starter to me. It's
a DSL inside a DSL. I'm also not sure what it actually does - you
already have all those @attribute decorators, why repeat their names
in some string?
2) Meta.fk_level - I don't think this is really useful (AFAIR, the
previous GSOC had this and it wan't very well received). In 80% cases
you know exactly which FKs you want to have on the model. If the
related objects also have FKs, then either they are important or they
can be omitted. Cutting off at some arbitrary depth doesn't feel
right. In my code, I instead define "named" serializers for models and
resolved them based on the "path". This way I can get different fields
for "User related to Comment" and "User related to Task", while
avoiding an infinite loop.
3) Did you thought about splitting the serialization process in
two parts (dehydration + presentation)? This is what most REST
frameworks do. First you serialize the objects into Python native
types, then render it to any format.
--
Łukasz Rekucki
Alex Ogier
I like the project and am +1 on Łukasz's points #1 and #3.
Meta.fk_level looks very useful though. For example if models have parent/sibling relationships or otherwise relate to themselves and you can't guarantee the relationships are acyclic.
Definitely a worthy project.
-Alex Ogier
Alex Ogier
Actually, thinking about it a bit more, it seems fk_level is bound to be a significant source of untested edge-case behavior. Every time the limit is reached, there will be an unexpectedly flat relationship that a naive recursive deserializer may choke on. I would be +1 on a fail-fast version of the same parameter that throws at time of serialization.
-Alex Ogier
Piotr Grabowski
> 1) The Meta.structure things looks like a non-starter to me. It's
> a DSL inside a DSL. I'm also not sure what it actually does - you
> already have all those @attribute decorators, why repeat their names
> in some string?
Ex
<object>
<a>
<b>
<c>
<name>Django</name>
</c>
</b>
</a>
</object>
1. With Meta.structure you can do:
def name
return Django
structure="a[b[c[name__field]]]
2. Without:
def a
return BFieldSerializer
class BFieldSerializer
def b
...
You see my point? I agree that second solution is more elegant but first
is a lot faster. Question is that someone actually want/need to define
structure tree like this.
Ex2.
<object>
<model_field1>...
<model_field2>...
<special_model_fields>
<model_field_3>
<model_field_4>
</special_model_fields>
<object>
1. structure="model_field1__field model_field2__field
special_model_fields{model_field3__field model_field4__field}"
or structure="__fields special_model_fields{model_field3__field
model_field4__field}"
2. Even if model_field1/2 will be automaticaly in right place what to do
with 3/4 ?
def special_model_fields
return {'model_field_3' : model_field_3, model... }
Hmm, I want to prove that structure will be better in this case but come
up with above idea :)
If Serializers methods can returns base type objects, FieldSerializers,
[] and {} we can define anything :) And it's a lot better than structure!
Must rewrite my proposal :)
>
> 3) Did you thought about splitting the serialization process in
> two parts (dehydration + presentation)? This is what most REST
> frameworks do. First you serialize the objects into Python native
> types, then render it to any format.
>
>
Yes, in my solution anything at the end of first phaze will be Python
base type, BaseFieldSerialize subclass or BaseModelSerializer subclass
with resolved Meta.structure. If I remove Meta.structure it will be even
simplier. I can resolve Base(Model/Field)Serializer only when i know to
what format it will be serialized.
--
Piotr Grabowski
Alex Ogier
> Ex
> <object>
> <a>
> <b>
> <c>
> <name>Django</name>
> </c>
> </b>
> </a>
> </object>
>
> 1. With Meta.structure you can do:
> def name
> return Django
> structure="a[b[c[name__field]]]
>
> 2. Without:
> def a
> return BFieldSerializer
> class BFieldSerializer
> def b
If you switch to a two phase serialize you can get this for free:
def a(instance):
return {'b': {'c': {'name': instance.name}}}
and then you can exclude name if you don't want it appearing as a
naked key. You seemed to hit on this yourself later in your reply, so
forge onwards.
> Yes, in my solution anything at the end of first phaze will be Python base
> type, BaseFieldSerialize subclass or BaseModelSerializer subclass with
> resolved Meta.structure. If I remove Meta.structure it will be even
> simplier. I can resolve Base(Model/Field)Serializer only when i know to what
> format it will be serialized.
I think we mean different things by "phase." You are describing the
first phase as resolving the serializer into a python class instance.
We are considering the first phase to be serializing a model instance
into python primitive types, before serializing it to a textual
format.
-Alex Ogier
Piotr Grabowski
research, find some REST serializers. I focus more on deserialization -
it should be easy to provide data is round-trippable. I discard some
unnecessary fields and try to improve functionality.
--------
GSOC 2012 Customizable serialization
-------
Django has a framework for serialization but it is simple tool. The main
problem is impossibility to define own serialization structure and no
support for related models. Below I present a proposal to improve
current framework.
In my opinion it is not possible to create a framework completely
independent of formats that will be user to serialize objects. For
instance XML has richer syntax than json (e.g. fields can be tags or
attributes) so we must provide functions to handle it which won't be
useful in JSON serialization.
-------
Features to implement:
-------
Based on presented issues to consider, GSOC proposal from last years and
django-developers group threads I prepare a list of features that good
solution should have.
1. Defining the structure of serialized object
1.1. Object fields can be at any position in output tree.
1.2. Renaming fields
1.3. Serializing non-database attributes/properties
1.4. Serializing any subset of object fields.
2. Defining own fields
2.1. Related model fields
2.1.1. Serializing foreign keys, m2m and reverse relations
2.1.2. Choose depth of serialization
2.1.3. Handling natural keys
2.1.4. Handling objects serialized before (in other location of output tree)
2.1.5. Object of same type can be differently handled depends on location
2.2. Other fields - custom serialization (e.g. only date in datetime fields)
3. One definition can support multiple serialization formats (XML, JSON,
YAML).
4. Backward compatible
5. Solution should be simple. Easy to write own serialization scheme.
Below I have tags like (F2.1.2) - means support for feature 2.1.2.
------
Concept:
------
Make the easy things easy, and the hard things possible.
In my proposal I was inspired by Django Forms and django-tastypie.
Tastypie is great API framework for Django.
Output structure will be defined declarative using classes. For sure
there is needed class for model definition. In my solution I define also
model fields with classes. It's the simplest way to provide free output
structure.
There should be two phases of serialization. In first phase Django
objects like Models or Querysets will be write as native Python types
(F3) and then in second phase it will be serialized to chooses format.
Suppose we want to serialize this model:
class Comment(Model):
user = ForeignKey(Profile)
photo = ForeignKey(Photo)
topic = CharField()
content = CharField()
created_at = DateTimeField()
ip_address = IPAddressField()
class User(Model):
fname = CharField()
lname = CharField()
class Photo(Model):
sender = ForeignKey(User)
image = ImageField()
Below we have definition of serializer classes CommentSerializer.
If we want to serialize comment queryset:
serializers.serialize('json|xml|yaml', queryset,
serializer=CommentSerializer, **options)
If 'serializer' isn't provided we have defaults serializer for each
format (F3)
class CommentSerializer(ModelSerializer):
content = ContentField()
topic = TopicField(attribute=True)
photo = ForeignKey(serializer=PhotoSerializer)
y = YField() #(F1.1.3)
def dehydrate__datetime(self, obj): #(F2.2)
return smart_unicode(obj.date())
def hydrate__date(self, obj): #(F2.2)
return smart_unicode(datetime.combine(obj, datetime.time.now()))
class Meta:
aliases = {'topic' : 'subject'}
#fields = (,)
exclude = ('ip_address',)
relation_reserialize = FlatSerializer
field_serializer = FieldSerializer
# subclass of ModelSerializer or FieldSerializer
relation_serializer =
FlatSerializer|ModelSerializer|NaturalModelSerializer|MyModelSerializer
object_name = "my_obj"
model_name = "model"
ModelSerializer has definition of fields, methods and Meta class.
Default each field is serialized by Meta.field_serializer or
Meta.relation_serializer. ModelSerializer fields redefining this
behavior. ModelSerializer methods dehydrate__xxx redefining
serialization for type xxx, and hydrate__xxx is for deserialization.
ModelSerializer methods returns native Python types
I will explain ModelSerializer fields later
Meta Class
a) aliases - redefine field name: topic : "..." => subject : "...". Can
do 'topic' : '' - return of topic method is one level higher. There is
metatag __fields__ - rename all fields. If more than one field has same
name list is created #(F1.2)
b) fields - fields to serialize #(F1.4)
c) exclude - fields to not serialize #(F1.4)
g) relation_reserialize - using what Serializer if object was serialized
before(F2.1.4)
h) field_serializer - default field serializer
h) relation_serializer - default relation (ForeingKey or ManyToMany)
serializer. There are some build-in posibilities: (2.1)
* FlatSerialzer - only primary key - default
* ModelSerializer - Predefined serializer for related models. One level
depth, all fields.
* NaturalModelSerializer - like flat but serialize natural keys
* Custom Model Serializer
If someone want serialize also intermediate model in M2M he should wrote
custom field
i) object_name - if it isn't empty returns <object_name_value>serialized
object</object_name_value> else return serialized object. Useful with
nested serialization. Default object_name is empty. In root level if
object_name is empty then "object" is default
j) In what field of serialized input is stored model class name
ModelSerializer fields are responsible for serialization model fields
In serialization value of model field will be passed to some methods of
ModelSerializer field class and it should be able to return Python
native types. Field should be able also to deserialize model field value
from input.
If there is some ModelSerializer class field and none field of that name
in model it should be treated as custom field.
class ContentField(FieldSerializer):
def hydrate__value__(self, field):
return field.lower()
def dehydrate__value__(self, field):
return field.upper()
class YField(FieldSerializer):
def dehydrate__value__(self, field):
return 5
class TopicField(FieldSerializer):
@attribute
def dehydrate__lower_topic(self, field):
return field.lower()
def __name__(self, field):
return "value"
def dehydrate__value__(self, field):
return field
Each method represent field in serialized output. They can return python
native types, other Fields or ModelSerializers, list or dict.
Field serializer has two special methods __name__ and value__. value__
is the primary value returned by field. Each method except __name__
should be preceded by dehydrate or hydrate. First is used in
serialization, second in deserialization.
E.g.
In some model class (topic="Django")
topic = TopicField()
class TopicField(Field):
def dehydrate__value__(self, field):
return field
xml: <topic>Django</topic>
json "topic" : "Django"
But what if we want to add come custom attribute (like lower_topic above).
xml: <topic><lower_topic>django</lower_topic>Django</topic> - far i know
it's correct but it's what we want?
json topic : {lower_topic : django, ??? : Django}
We have __name__ to provide some name for field:
class TopicField(Field):
def __name__(self, field):
return "value"
def dehydrate__value__(self, field):
return field
xml: <topic><lower_topic>django</lower_topic><value>Django</value></topic>
json topic : {lower_topic : django, value : Django}
Like I say before, there are two phases of serialization.
First phase present Django models as native Python types.
At beginig each object to serialize is passed to ModelSerializer and to
each Field. Everything will be resolve to Python native types.
Rules for resolving:
1. In Serializer class:
* ModelSerializer class => {}
* If ModelSerializer class has object_name => __object_name__ :
Meta.object_name
* Fields in Serializer class => aliases[field_name] : field_value
* If aliases[x] == aliases[y] => aliases[x] : [x_value, y_value]
* If x=Field(attribute=True) => __attributes__ : {x : x_value} Fail if
x_value can't be attribute value
2. In Field class:
* If only value__ => dehydrated__value__
* If other methods presents => {mehod_name : method_value, ...}
* If __name__ => { __name__ : dehydrated__value__ }
* If method decorated @attribute => __attributes__ : {method_name :
method_value} Fail if method_value can't be attribute value
After that we have something like dict of dicts or lists. Next we must
append ModelSerializer dehydrate__type rules to output.
In dict there is special key __attribute__ contains dict of attributes
for xml
In this stage we must decide format to serialize. If it's not XML
__attribute__ must be joined to rest of the dict.
In second phase we have Python native types so we can serialized it with
some module like simplejson.dumps(our_output)
Deserialization:
It's also two phases process. In first phase we deserialize input to
Python native types (same as return in second phase of serialization),
and in second create Model objects. First phase should be simple. Second
is a lot harder.
First problem is what type of object is in serialized input. There are
two way to find it. You can pass Model class as argument to
serialization.serialize or specify in Meta.model_name what field
contains information about type.
Next all fields in Serializer should be matched with input.
'hydrate__value__' and other hydrate methods are used to fill fields in
model object.
-----
Prove of concept
-----
class PKField(Field):
def dehydrate__value__(self, field):
return smart_unicode(self.instance._get_pk_val(), strings_only=True)
def hydrate__value__(self, field):
self.instance.set_pk_val(field)
class ModelField(Field):
def dehydrate__value__(self, field):
return smart_unicode(obj._meta)
#no need of hydrate__value__
class JSONSerializer(ModelSerializer):
pk = PKField(attribute=True)
model = ModelField(attribute=True)
class Meta:
aliases = {'__fields__' : 'fields'}
relation_serializer = FlatSerializer
class XMLSerializer(JSONSerializer):
class Meta:
aliases = {'__fields__' : 'field'}
default_field_serializer = XMLFieldSerializer
default_relation_serializer = XMLFlatRelationSerializer
XMLFieldSerializer(Field):
@attribute
def name(self, name, obj):
...
@attribute
def type(self, name, obj):
...
XMLFlatRelationSerializer(Field):
@attribute
def to
...
@attribute
def name
...
@attribute
def rel
...
-----
Shedule
-----
I want to work approximately 20 hours per week. 15 hours writting code
and rest for tests and documentation
Before start: Discussion on API design, I hope everything should be
clear before I start writting code.
Week 1-2: Developing base code for Serializer.
Week 3-4: Developing first phase of serialization.
Week 5: Developing second phase of deserialization.
Week 6: Developing second phase of serialization and first of
deserialization
It's time for mid-term evaluation. I will have working Serializer except
nested relations.
Week 7-8: Handling nested ForeignKeys and M2M fields.
Week 9: Developing old serialization in new api with backward compatibility
Week 10: Regression tests, writing documentation
Week 11-12: Buffer weeks
-----
About
-----
My name is Piotr Grabowski. I'm last year student at the Institute of
Computer Science University of Wrocław (Poland). I've been working with
Django for 2 years. Python is my preffered programing language but I
have been using also Ruby(&Rails) and JavaScript.
--
Piotr Grabowski
schinckel
Tom Christie
Piotr Grabowski
It breaks the problem down into two very well defined tasks.
1. Convert native datatypes to/from data streams. (eg. Tastypie's serializers, REST framework's renderers & parsers)
2. Convert complex objects to/from native datatypes. (eg. Tastypie's hydrate/dehydate, REST frameworks serializer, form based deserialization)
In your proposal you've understandably addressed model serialization in detail, but I (personally) think it'd be worthwhile breaking things down a little bit more.For example, you might consider the following:
1. Define a base API for part (1). Is it handled by a single class with instance methods for converting in each direction, or by two separate classes?
In my solution user should (but it is not reguired) think about format in first phase:
e.g
In present django serialization we have:
For json
fields : {
first_field : ...
second_field : ...
}
<field name=first_field>...
<field name=second_field>..
I don't see how you do it in your proposal. You only specify 'fields' in Meta.fields.
In my proposal there is need for two Serializers definition but it's simple and one can inherite from other.
I specify JSONSerializer and XMLSerializer for backward compatibility.
In my solution I can provide one Serializer for all formats:
class PKField(Field):
def dehydrate__value__(self, field):
return smart_unicode(self.instance._get_pk_val(),
strings_only=True)
def hydrate__value__(self, field):
self.instance.set_pk_val(field)
class ModelField(Field):
def dehydrate__value__(self, field):
return smart_unicode(obj._meta)
class AllSerializer(ModelSerializer):
pk = PKField()
model = ModelField()
class Meta:
aliases = {'__fields__' : 'fields'}
relation_serializer = FlatSerializer
{
pk : ... ,
model : ...,
fields : {
}
}
It's what we want. But in xml:
<object>
<pk>...
<model>...
<fields>only_field_value</fields>
<fields>...
...
</object>
Ups :/
What about formats that are one direction only? (eg. You might want to parse HTTP form data, but you're unlikely to want to render it.)Are formats that are not fully reversible an issue? (eg. Some formats [eg csv?] might not have native representations of all the python datatypes, and only have string representations.)
If some information about object are losed in serialization-deserialization then I try construct model only from what I get. I won't save it. It's user problem. Mayby he wants to add some other informations from request (you mention it in point 7)
Some things I was slightly confused by in your proposal as it stands...
* JSONSerializer subclassing Serializer. (and XMLSerializer subclassing JSONSerializer.)I'd have thought that if you're using a two phase approach you'd keep the complex<->native and native<->data stream APIs totally decoupled.JSON serialization doesn't itself have anything to do with how you choose to structure the serialization of a complex object, so why should it subclass the implementation of that part?
* "dehydrate__value__", "hydrate__value__" - what's with the double underscore naming?
def some_name(...):
return "test"
Then you get field_name : { some_name : test}
But __value__ is special:
field_name : return_of__value__
or field_name : {
some_name : test,
return_of__name__ : return_of __value__
}
But this is theory. I have dehydrate__xxx for serialization xxx value. So I should have dehydrate____value__. Ugly :/ So there is value__
It should be rewrite.
* I don't get the '@attribute' decorator. I think it'd be simpler to just have 'fields', 'include', 'exclude'. If fields is None then use the 'default set of fields' + 'include' - 'exclude'. If fields is not None, use that and ignore include/exclude.
* I wouldn't consider special casing for XML serialization in the complex<->native stage. Sure, yeah, make sure there's an XML implementation that can handle the current Django XML serialization structure, but anything more than that and you're likely to end up muddying the API for a special case of data format.
* 'relation_reserialize' - Why is that needed?
sender = User
person_on_photo = User
If p.sender=p.person_on_photo - mayby we want to serialize this two times or mayby we want ony sender : {serialized_sender}, person_on_photo : 10
* 'object_name' - It's not obvious to me if that's necessary or not.
* "In what field of serialized input is stored model class name" - What about when the class name isn't stored in the serialization data?
First problem is what type of object is in serialized input. There are
two way to find it. You can pass Model class as argument to
serialization.serialize or specify in Meta.model_name what field
contains information about type.
* "dehydrate__xxx redefining serialization for type xxx." I'm not convinced about that - it's not very pythonic to rely on type hierarchy in preference to duck typing.
Thanks for your reply.
Cheers,
Tom
--
Piotr Grabowski