backsliding on schemaish

[Chris McDonough]

Hi folks,

Recently I had occasion to want to use schemaish plus convertish as part of a
framework to validate data scraped out of a YAML document for a configuration
system named "repoze.configuration". This turned out to be slightly less
pleasant than I would have liked it to be.

It's mostly a style thing. I don't yet understand the purpose of the
simplegeneric adaptation indirection from schema element to converter. I'm
sure there's a good reason, but I just don't quite get it. I can definitely
understand an adaptation from schema element to *widget*, but I'm having
difficulty understanding why there's a similar indirection between schema
element and converter. It seems like a false affordance.

Since the thing I'm developing is meant to be extensible, I'd really rather not
have people create both a schema type and a converter and wire them up together
using a simplegeneric adapter. It seems to just require slightly too much
brainpower on the behalf of the exender for this particular purpose.

As a result, in the meantime, I've begun to develop a slightly simpler schema /
validation / conversion system that jams together data type validation and
value conversion into a set of cooperating classes. The package is nowhere
near finished, it's mostly experimental... I wonder if you guys might be able
to convince me to stop going down this path by shedding some light on the
purpose of the converter adapter indirection, and by maybe offering some sample
code that mimics what's in the below docs using schemaish+convertish.

Docs: http://docs.repoze.org/cereal/

Code: http://svn.repoze.org/cereal/trunk/

Hope you're good!

--
Chris McDonough
Agendaless Consulting, Fredericksburg VA
The repoze.bfg Web Application Framework Book: http://bfg.repoze.org/book

[Tim Parkin]

I'll have to have a think about this one. I'd like to support what you're doing here within the schema/convert system but can't work out how at the moment. a couple of questions..

 - passing struct's into the validators seems a bit strange. I like the idea of a validator that just validates
 - we convert from a string to a sequence in some cases (in fact from a string to a sequence of sequences in some cases). If the converter and schema type are combined, this doesn't seem possible?

One of the issues with just having a string converter (i.e. just serialise) is that you can't convert date part fields. For instance, in formish we have date tuple fields that return a tuple of string values. This doesn't fit directly into a schema that just serializes to strings. Take a look at the DateParts widget.

It would be good if we could solve this without having multiple libraries though ..

More thinking soon..

Tim

--
You received this message because you are subscribed to the Google Groups "ish.io" group.
To post to this group, send an email to is...@googlegroups.com.
To unsubscribe from this group, send email to ishio+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/ishio?hl=en-GB.

[Chris McDonough]

On 3/12/10 4:08 PM, Tim Parkin wrote:
> I'll have to have a think about this one. I'd like to support what
> you're doing here within the schema/convert system but can't work out
> how at the moment. a couple of questions..
>
> - passing struct's into the validators seems a bit strange. I like the
> idea of a validator that just validates

The validator does still just validate, really. But in order to form a valid
exception chain (a chain is required to show an "error dict"), it needs to have
a struct to pass in to the Invalid exception. An Invalid exception can't be
constructed without one.

This might be improved.

> - we convert from a string to a sequence in some cases (in fact from a
> string to a sequence of sequences in some cases). If the converter and
> schema type are combined, this doesn't seem possible?

Well, this is maybe another style issue. It's sort of unrelated to my original
reason for sending the mail, but I'll express it here anyway.

I think I might prefer it if "raw" type conversion (like converting a string
composed with "\n" delimiters into a sequence of lines) were the the job of a
widget rather than a converter. The widget would be responsible for converting
a string into data structure composed of very raw values (either a string, a
sequence, or a mapping) for later consumption by a converter.

This seems like it might be a more sensible division of labor, at the expense
of potentially passing off some responsibility to widget code because it keeps
a lot of responsibility out of the conversion code that is not useful when
you're dealing with things like YAML or JSON serializations which are
relatively rich compared to a flat set of keys and string values like you might
get from something like an HTML form post.

> One of the issues with just having a string converter (i.e. just
> serialise) is that you can't convert date part fields. For instance, in
> formish we have date tuple fields that return a tuple of string values.
> This doesn't fit directly into a schema that just serializes to strings.
> Take a look at the DateParts widget.

Good point. That's something to chew on.

> It would be good if we could solve this without having multiple
> libraries though ..
>
> More thinking soon..

Thanks for the response! It'd be really useful to brainstorm this with you.

- C

> Groups "ish.io <http://ish.io>" group.

> To post to this group, send an email to is...@googlegroups.com

> <mailto:is...@googlegroups.com>.

> To unsubscribe from this group, send email to
> ishio+un...@googlegroups.com

> <mailto:ishio%2Bunsu...@googlegroups.com>.

> For more options, visit this group at
> http://groups.google.com/group/ishio?hl=en-GB.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "ish.io" group.
> To post to this group, send an email to is...@googlegroups.com.
> To unsubscribe from this group, send email to
> ishio+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/ishio?hl=en-GB.

[Tim Parkin]

On Fri, Mar 12, 2010 at 9:40 PM, Chris McDonough <chr...@plope.com> wrote:

On 3/12/10 4:08 PM, Tim Parkin wrote:

 - passing struct's into the validators seems a bit strange. I like the
idea of a validator that just validates

The validator does still just validate, really.  But in order to form a valid exception chain (a chain is required to show an "error dict"), it needs to have a struct to pass in to the Invalid exception.  An Invalid exception can't be constructed without one.

This might be improved.

The schema knows it's own keys so it should be able to call and catch exceptions in the convert/validate steps and store the related key in a structure (at least this is what formish is doing in Form()._validate)

 

 - we convert from a string to a sequence in some cases (in fact from a
string to a sequence of sequences in some cases). If the converter and
schema type are combined, this doesn't seem possible?

Well, this is maybe another style issue.  It's sort of unrelated to my original reason for sending the mail, but I'll express it here anyway.

I think I might prefer it if "raw" type conversion (like converting a string composed with "\n" delimiters into a sequence of lines) were the the job of a widget rather than a converter.  The widget would be responsible for converting a string into data structure composed of very raw values (either a string, a sequence, or a mapping) for later consumption by a converter.

This seems like it might be a more sensible division of labor, at the expense of potentially passing off some responsibility to widget code because it keeps a lot of responsibility out of the conversion code that is not useful when you're dealing with things like YAML or JSON serializations which are relatively rich compared to a flat set of keys and string values like you might get from something like an HTML form post.

Fair point and if this can be simplified it would be good.. 

It would be good if we could solve this without having multiple
libraries though ..

More thinking soon..

Thanks for the response!  It'd be really useful to brainstorm this with you.

Indeed :-)

[Barry Warsaw]

On Mar 12, 2010, at 12:23 AM, Chris McDonough wrote:

>Recently I had occasion to want to use schemaish plus convertish as part of a
>framework to validate data scraped out of a YAML document for a configuration
>system named "repoze.configuration". This turned out to be slightly less
>pleasant than I would have liked it to be.

I'm not sure this is really related, or even fits the same use cases that you
have, but I was looking for a really lightweight converter for integrating
restish with Mailman 3 [*]. I didn't find anything that i was totally happy
with so I wrote a simple class that does both validation and conversion. I
mean *really* simple. :)

The code is the Validator class in this file:

http://bazaar.launchpad.net/~mailman-coders/mailman/3.0/annotate/head%3A/src/mailman/rest/helpers.py

and it's used like this:

@resource.POST()
def create(self, request):
"""Create a new member."""
...
validator = Validator(fqdn_listname=unicode,
address=unicode,
real_name=unicode,
delivery_mode=unicode,
_optional=('real_name', 'delivery_mode'))
try:
member = service.join(**validator(request))
except ValueError as error:
return http.bad_request([], str(error))

where service.join() is defined as:

def join(self, fqdn_listname, address,
real_name= None, delivery_mode=None):

I don't much like having to repeat myself for _optional, and I could probably
introspect the method to figure that out, but as I said, I wanted something
really simple. I do like the fact that all parameters have to validate (well,
convert actually) in order to pass, and that conversions are just done through
callables. So far, I haven't minded that validation and conversion are
conflated.

-Barry

[*] While at Pycon I converted Mailman 3 from lazr.restful to restish. There
are a lot of nice things about restish, not the least of which that it's much
more agile to add new APIs. I also think it leads to more readable, more
discoverable code because there are no ZCML files :). restish also allows you
to more cleanly separate API from internal model interfaces. The things it
doesn't have that lazr.restful gives you are support for collections
(including batching), etags, and a few other niceties. I have some thoughts
on adding a few things to restish, but haven't had time to put together some
code. Feel free to hijack this thread if you want to discuss in more
detail. :)