Django Model serializer for dojo ItemFileRead/Write store

[BenW]

Hello,

I've put together a model serializer based on Django's python
serializer for models. It puts the data the format used by
ItemFileRead/Write Store. There's a deserializer as well for getting
Django objects back out of the store ie, after a store.save() -- I'd
love to have a more permanent spot for it. Right now I just pasted on
pastebin.

http://dpaste.com/hold/76563/

Use it like so:

from myapp.models import ChatRoom
from modelstore import ModelStore, Deserializer

store = ModelStore(label='title')
data = store.serialize( Chatroom.objects.all(), exclude=
('secret_attribute',))

the serializer takes 3 args:
fields: which fields to include
exclude: which fields NOT to include
as_json: do we convert the data to Json? useful if you want datetimes
to serialize properly, otherwise you can just do it yourself with
simplejson and cls=DjangoJSONEncoder

Example View:

def my_ItemFileReadStore(request):

store = ModelStore(label='name')
json = store.serialize(ChatRoom.objects.all(), as_json=True)
return HttpResponse(json, mimetype='application/json')

def my_ItemFileWriteStore(request):

if request.method == 'GET'
store = ModelStore(label='name')
json = store.serialize(ChatRoom.objects.all(), as_json=True)
return HttpResponse(json, mimetype='application/json')

data = simplejson.loads(request.POST.get('data'))
for obj in Deserializer(data):
# do something with deserialized Django objects
# from here on out it's exactly like handling
DeserializedObject instances with Django's own
# core serializers

[klipstein]

Hey Ben,

at first sight this code looks very clean (really like it), but I need
to compare it with the implementation of:

http://code.google.com/p/django-dojoserializer/source/browse/src/dojoserializer/utils.py

Maybe yours and the approach of Fabian could be merged to the perfect
dojo<->django serializer. If would be great, if you could have a look
at the django-dojoserializer and compare it with your implementation
and point out the differences/capabilities. For you it should be much
easier doing that comparison, as you already dived into that topic.
One plus of Fabian's implementation is the serializer test (http://
code.google.com/p/django-dojoserializer/source/browse/src/
dojoserializer/tests.py).

Thanks, Tobias

[Ben Wilber]

I only now saw that Fabian's app even existed! -- otherwise I may not
have written one at all. I took a look and really like it but there
are a couple of things to mention. His seems to have a bit wider
scope in mind, with the restful view functions and whatnot whereas
mine is purely aimed at adapting Django's own Python/Json serializers
to work with dojo data stores. I don't see anywhere in his code that
deals with '_reference' items and FK/M2Ms, which I definitely find
useful. Also, using <app name>.<model>__<pk> as the identifier saves
space in the store and allows to resolve the store item into a Django
model object at the same time, instead of having separate (optional)
fields for those.

My ideal serializer would take a queryset or any iterable that yields
Django model objects and format them in ItemFileRead/WriteStore
structure including hierarchy and '_reference's (ModelStore already
does the '_reference' bit. The interface should look a lot like
Django's forms.Form implementation, where the same class produces the
store and validates/saves it. Getting objects back from json is a
tricky bit because Django's DeserializedObject instances are not
actually in the database. For instance, if you produce a store that
has three items:

{}&&
{ identifier: '_id_', label: 'name', items: [
{_id_: 'auth.user__1', name: 'Joe'},
{_id_: 'auth.user__2', name: 'Bob'},
{_id_: 'auth.user__3', name: 'Mary'}
]}

When items in this store get deserialized into DeserializedObject
instances, the are not checked for reference integrity or validation
or anything -- so you basically have to do a big loop full of
get_or_create() on the item identifier otherwise you might overwrite
other objects in your database if you just blindly call save() on the
deserialized object, which can be a major security issue. Plus, any
malicious user can arbitrarily add fields to a store item, which will
get populated by Django's deserializer -- so you still have to check
that the only fields coming back to python are the ones you put in
there in the first place, (that's where the forms.Form implementation
looks appealing.)

I'll keep working on this and see if I can get it close to what I had
in mind and get some tests going. I'd love to join forces with Fabian
to get a killer dojo datastore app for Django out there, (i even gave
a thought to implementing a new dojo store altogether with Django as
the focus. Some interesting things could definitely be done with
jsonrpc -- installing automatic connections to setValue etc. that
calls back to a Django method that updates the db realtime, rather
than after a save() or saveCustom() .. just thoughts for now.)

Ben

[klipstein]

Hi Ben,

I now found the time looking at your implementation in detail and in
my opinion it is totally amazing. Especially the fact, that you can
directly use the deserialized objects for saving. And as you've
mentioned, that these objets aren't validated, I had a look at
ModelForms and how this class can be used for validating the
deserialized objects. Here a short test that worked for me:

from myapp.models import Movie
from dojango.util import DojoSerializer
from django.forms.models import model_to_dict, modelform_factory

# serialize a queryset
s = DojoSerializer.ModelStore()
s_objs = s.serialize(Movie.objects.all())

# deserialize the previous generated serialized dataset
d_objs = DojoSerializer.Deserializer(s_objs)

# get one deserialized object and convert it to a dict
# so the data of this object can be passed to a ModelForm
one_obj = d_objs.next().object
one_obj_dict = model_to_dict(one_obj)

# create a ModelForm of the Movie model
mf = modelform_factory(Movie)

# instantiate this ModelForm with the deserialized object data and the
# original instance of this Movie object
form = mf(one_obj_dict, instance=Movie.objects.get(id=one_obj.id))
if form.is_valid():
form.save()

And this whole procedure also works for selected (or excluded) fields:

d_objs = DojoSerializer.Deserializer(s_objs, fields=['title','year',])
# or ,exclude=[]
one_obj = d_objs.next().object
one_obj_dict = model_to_dict(one_obj)
mf = modelform_factory(Movie, fields=['title','year',]) # or ,exclude=
[]

I think integrating that behaviour in your serializer should be fairly
simple and now should met your (and also my :-)) requirements. A
serializer for normal python objects is already in dojango. Maybe this
could be merged with yours somehow or we keep it separate.

Fabian's implementation of a REST interface goes too far. I would
rather use the reusable app django-piston for that.

I'm curious, if this ModelForm validation would be sufficient for you.

Regards, Tobias

> klipstein<tobias.klipst...@googlemail.com> wrote:
>
> > Hey Ben,
>
> > at first sight this code looks very clean (really like it), but I need
> > to compare it with the implementation of:
>

> >http://code.google.com/p/django-dojoserializer/source/browse/src/dojo...

[Ben Wilber]

Thanks

There are definitely some areas that could use some improvement. I
really like what you did with ModelForm and I think that is definitely
the way to move forward with the validation stuff. I also like
Fabian's class that allows you to serialize 'external' field/values
that are not part of any model field -- I would like to roll this into
ModelForm as well. I thought about adding some hooks to do
pre-processing of field values before putting them in the store, but
dojo's data drivers already have that (the 'formatter' attribute and
'typeDef') but this would be useful server-side as well.

I'm thinking about bringing Deserializer into the main ModelStore
class and implementing methods closely resembling forms.Form (or
ModelForm) -- ideally, a ModelStore view would look like this:

def MyStore(ModelStore):

objects = MyModel.objects.all()

def is_valid(self):
# do custom validation ...

from dojango.util.DojoSerializer import ModelStore
def my_modelstore(request):

if request.method == 'POST':
store = ModelStore(request.POST)
if store.is_valid():
store.save()
else:
# do something if not valid
...
else:
store = ModelStore(MyModel.objects.all(), as_json=True)

return HttpResponse(store, mimetype='application/json')

This is pretty close to what a typical forms.(Model)Form view looks
like which is what I'd like to aim for. There is a fundamental
difference though in that a Form usually deals with a single obj or
Model, whereas there is no such limitation in ModelStore. You can mix
and match your models and instances all you want all the while
preserving the fk and m2m relationships so that Dojo can take
advantage of it in the client side.

defining the objects or QuerySet declaratively in a subclass is also
nifty, especially since that allows a store to be reused etc. without
duplicating code (another forms.Form benefit.)

Your thoughts are welcome!

Ben

[BenW]

I've finally gotten around to digging into the serializer code again.
I decided that a better approach would be to define a 'store' the same
way you define Django models. The result is Django-ModelStore -- a
reusable app to turn Django models into dojo.data stores.

The code is here:
https://launchpad.net/django-modelstore/

and the docs are here:
http://sites.google.com/site/djangomodelstore/

I'd really love some feedback!

Thanks,

Ben

[klipstein]

Wow Ben,

now with a completely new approach :). I'll definitely need some time
digging/playing around with what you've created so far to give you a
decent answer? ATM it really looks like a good idea solving it that
way, but I also liked the Python serializer approach.

Downloading django-modelstore and testing it now.

Thanks for your work on that, Tobias