Colander: Is preparer supported in schema binding?

[Mariano Mara]

Hi there, I have a deferred preparer I want to apply in schema binding,
however it seems it's ignoring the kw parameter. Is this the expected behaviour
or I'm missing some detail?

TIA,
Mariano

[Eric Rasmussen]

Hi Mariano,

I haven't had any trouble with this but it would help to see your code to make sure I understand you correctly. The kw parameter should be the second argument to your deferred widget, and it will let you access keyword arguments passed to the bind function like this:

# deferred select widget
@colander.deferred
def deferred_widget(node, kw):
    options = kw.get('mykeyword', [])
    return widget.SelectWidget(values=options)

# binding
SomeSchema().bind(mykeyword=data)

There's a more complete example at http://docs.pylonsproject.org/projects/colander/dev/binding.html?highlight=deferred, but let me know if I misunderstood.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To post to this group, send email to pylons-...@googlegroups.com.
To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.

[Mariano Mara]

On 07.11.11 21:17, Eric Rasmussen wrote:
> Hi Mariano,
> I haven't had any trouble with this but it would help to see your code
> to make sure I understand you correctly. The kw parameter should be the
> second argument to your deferred widget, and it will let you access
> keyword arguments passed to the bind function like this:
> # deferred select widget
> @colander.deferred
> def deferred_widget(node, kw):
> options = kw.get('mykeyword', [])
> return widget.SelectWidget(values=options)
> # binding
> SomeSchema().bind(mykeyword=data)
> There's a more complete example at

> [1]http://docs.pylonsproject.org/projects/colander/dev/binding.html?hig

> hlight=deferred, but let me know if I misunderstood.
> Thanks,
> Eric
>

Hi Eric, thanks a lot for your quick answer.

In order to give a little of extra context regarding my problem: I'm using colander to
deserialize an excel file. One of the values is a date and excel stores it as
a float. The xlrd package provides a suitable method to turn that float into a
proper date value, however it requires a 'datemode' parameter that can only be
obtained once you opened and parsed the excel file. That's why I'm deferring
this preparer.

I managed to create the following auto contained example (this can be
reproduced installing the xlrd package). After some rounds of debugging my
understanding is that a deferred preparer method is unable to receive the kw
parameter.

import colander, datetime
from xlrd import xldate_as_tuple

@colander.deferred
def deferred_xls_date(value, kw):
"""Preparer method to turn an excel float into the corresponding date
value. We will know the datemode parameter once the excel book has been
opened and parsed"""
if value is None or value=='': return None
datemode = getattr(kw.get('book'), 'datemode')
return xldate_as_tuple(value, datemode)

class CUser(colander.MappingSchema):
id = colander.SchemaNode(colander.String())

cstruct = {'id':u'TEST', 'sdate':40808.0}
schema = CUser().bind(book=type('Book', (object,), {'datemode':0}))
schema.add(colander.SchemaNode(colander.Date(),
preparer=deferred_xls_date,
validator=colander.Range(max=datetime.datetime.now())
))
schema.deserialize(cstruct)

This happens when you execute it:

$ python colander_example.py
Traceback (most recent call last):
File "colander_example.py", line 22, in <module>
schema.deserialize(cstruct)
File "/home/mariano/Code/SmD/env/lib/python2.7/site-packages/colander/__init__.py", line 1576, in deserialize
appstruct = self.typ.deserialize(self, cstruct)
File "/home/mariano/Code/SmD/env/lib/python2.7/site-packages/colander/__init__.py", line 477, in deserialize
return self._impl(node, cstruct, callback)
File "/home/mariano/Code/SmD/env/lib/python2.7/site-packages/colander/__init__.py", line 439, in _impl
result[name] = callback(subnode, subval)
File "/home/mariano/Code/SmD/env/lib/python2.7/site-packages/colander/__init__.py", line 475, in callback
return subnode.deserialize(subcstruct)
File "/home/mariano/Code/SmD/env/lib/python2.7/site-packages/colander/__init__.py", line 1579, in deserialize
appstruct = self.preparer(appstruct)
TypeError: __call__() takes exactly 3 arguments (2 given)

[Eric Rasmussen]

Hi Mariano,

I understand the issue now and see what you're trying to do. My initial thought is this isn't possible with preparer because it takes a value as its only argument and returns the value as-is or modified.

However, I'll take a look at this again tonight if no one else knows a solution off-hand.

Best,
Eric

[Eric Rasmussen]

Hi Mariano,

I was hoping one of the colander experts might jump in, but I'll go
out on a limb and say there's probably not a really clean way to do
exactly what you want. However, you might consider passing in a
deferred missing function to the SchemaNode. It does bypass
validation, but otherwise appears to do what you want. Here's an
example (needs some cleanup):

import colander, datetime
from xlrd import xldate_as_tuple

# missing values not validated!
@colander.deferred
def xldate_missing(node, kw):
book = kw.get('book')
datemode = getattr(book, 'datemode')
datefloat = kw.get('datefloat')
tupledate = xldate_as_tuple(datefloat, datemode)
return datetime.datetime(*tupledate)

class CUser(colander.MappingSchema):
id = colander.SchemaNode(colander.String())

xldate = colander.SchemaNode(colander.Date(), missing=xldate_missing)

if __name__ == '__main__':
""" assign floating date to bound keyword datefloat
deferred missing method will fill in value"""

cstruct = {'id':u'TEST'}

schema = CUser().bind(
book=type('Book', (object,), {'datemode':0}),
datefloat=40808.0)
schema.deserialize(cstruct)
print 'final xldate value: %s' % schema.children[1].deserialize()

Or if you have a default datemode that's used most of the time, you
could use a preparer function that converts with the default, and then
modify the value if needed with an after_bind callback.

Best,
Eric

[Mariano Mara]

On 09.11.11 13:56, Eric Rasmussen wrote:
> Hi Mariano,
>

Eric, thanks a lot for your time and detailed suggestion. I will try to apply
it right away and see how it works and let you know.

Regards,
Mariano

[Mariano Mara]

Eric your suggestion works like charm (I'm preparing a good battery of tests
to be sure), thanks again!
I also took a look at colander's source code to see why preparers didn't support
deferreds and I added a little modification that seems to fix this:

$ git diff
diff --git a/colander/__init__.py b/colander/__init__.py
index a900aff..8510ee4 100644
--- a/colander/__init__.py
+++ b/colander/__init__.py
@@ -1576,7 +1576,8 @@ class SchemaNode(object):
appstruct = self.typ.deserialize(self, cstruct)

if self.preparer is not None:
- appstruct = self.preparer(appstruct)
+ if not isinstance(self.preparer, deferred): # unbound
+ appstruct = self.preparer(appstruct)

if appstruct is null:
appstruct = self.missing

I even ran the tests and it worked ok (and my problem seems to be fixed too).
It would be great if somebody with great colander jutsu can say why such
behavior was not implemented.

[Chris Withers]

On 10/11/2011 17:57, Mariano Mara wrote:
>
> I even ran the tests and it worked ok (and my problem seems to be fixed too).
> It would be great if somebody with great colander jutsu can say why such
> behavior was not implemented.

I implemented the feature, but I don't used deferreds.
(I didn't like the idea myself, but that doesn't mean they're bad...)

I also have my finger in the xlrd pie, and personally I wish xlrd would
just create date objects on parsing the .xls rather than the annoying
number-you-need-to-bend-over-backwards-to-turn-into-a-date ;-)

Why not just store the book *and* the preparer as attributes of the
Schema instance, then they can both get access to the state?

cheers,

Chris

--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk

[Mariano Mara]

On 11.11.11 18:48, Chris Withers wrote:
> On 10/11/2011 17:57, Mariano Mara wrote:
> >
> >I even ran the tests and it worked ok (and my problem seems to be fixed too).
> >It would be great if somebody with great colander jutsu can say why such
> >behavior was not implemented.
>
> I implemented the feature, but I don't used deferreds.
> (I didn't like the idea myself, but that doesn't mean they're bad...)
>
> I also have my finger in the xlrd pie, and personally I wish xlrd
> would just create date objects on parsing the .xls rather than the
> annoying number-you-need-to-bend-over-backwards-to-turn-into-a-date
> ;-)
>
> Why not just store the book *and* the preparer as attributes of the
> Schema instance, then they can both get access to the state?
>
> cheers,
>
> Chris

Hi Chris, thanks for your input, I'm already following Eric's advice and I'm
storing datemode in a previous step (users has the option to validate their
excel file and while at it, I retrieve the data and store it), precluding the
need for a deferred, which proved to be a really tough adversary.
I have even ended creating my own type to have a better error message for users and
so far it seems to be working (actually I don't need the serialize method so I
didn't test it):

class XLSDate(object):
"""
http://docs.pylonsproject.org/projects/colander/dev/extending.html
"""
def serialize(self, node, appstruct):
if appstruct is colander.null:
return colander.null
if not isinstance(appstruct, datetime.datetime):
raise colander.Invalid(node, _('{0} is not a value that can be '
'parse into the excel internal date representation'.\
format(appstruct)))
return appstruct and 'true' or 'false'

def deserialize(self, node, cstruct):
if cstruct is colander.null or cstruct==u'':
return colander.null
try:
return float(cstruct)
except ValueError, e:
raise(colander.Invalid(node, _('{0} is not value that can be '
'parse into a valid Excel date'.format(cstruct))))