Validation inconsistencies and custom routes

[Raphael Slinckx]

I was playing the other day with validation and custom routes, here
are some comments

1) Validation
1.1) Positional parameters validation
Using @validate on an exposed method, it seems I can't validate
positional parameters, and errors are raised when the call has more
than one value defined per parameter. Example:

First without validator

@expose()
def foo(self, a, b, c=None, d=None):
return "a=%r b=%r c=%r d=%r" % (a, b, c, d)

/foo/x/y/?a=z&b=w -> foo() got multiple values for keyword argument
'a'
/foo/x/y/?c=z&d=w -> a=u'x' b=u'y' c=u'z' d=u'w'
/foo/x/y/z/w -> a=u'x' b=u'y' c=u'z' d=u'w'
/foo/x?b=y -> a=u'x' b=u'y' c=None d=None
/foo?a=x&b=y&c=z&d=w -> a=u'x' b=u'y' c=u'z' d=u'w'

These results are what would be expected since a/b are mandatory, and
c/d are optional and default to None.
If I call it with two values for a (one in the positional part,
another in the query string) then it errors out
because we can't possibly disambiguate

Now, let's try to add a validator to the foo() method

@validate(validators=dict(
a=validators.String(), b=validators.String(),
c=validators.String(), d=validators.String(),
))
@expose()
def foo(self, a, b, c=None, d=None):
return "a=%r b=%r c=%r d=%r" % (a, b, c, d)

/foo/x/y/?a=z&b=w -> foo() got multiple values for keyword argument
'a'
/foo/x/y/?c=z&d=w -> foo() got multiple values for keyword argument
'a'
/foo/x/y/z/w -> foo() got multiple values for keyword argument
'a'
/foo/x?b=y -> foo() got multiple values for keyword argument
'a'
/foo?a=x&b=y&c=z&d=w -> a=u'x' b=u'y' c=u'z' d=u'w'

Now that's surprising. What happens is the following code (in tg/
controllers.py:_perform_validate):

for field, validator in validation.validators.iteritems():
try:
validator.to_python(params.get(field))
new_params[field] = validator.to_python(params.get(field))
# catch individual validation errors into the errors dictionary
except formencode.api.Invalid, inv:
errors[field] = inv

Meaning that for each defined validators we are going to insert a new
keyword argument with the result of the validation.
(By the way is it normal that the validator is run twice ?)

The result is that when the controller method is called with /foo/x?
b=y what really happens is
foo(*remainder, **kwargs) -> foo(*['x'], **{'a': '', 'b': u'y', 'c':
'', 'd': ''})
resulting in 'a' being defined twice in the method call.

This means in the current situation that you can only use validators
for keyword arguments.
It's also worth mentioning that positional arguments are simply
ignored in the validation process, so there's no way to validate them
I can't for example define a pagination using /foo/1, /foo/2, /foo/
3... and at the same time validate the page number as an Int.

I believe the way to fix this is to inspect the exposed method's
signature and match validators with arguments name, pylons seems to do
this,
but i don't know if there are any implications...

1.2) Default values with validation
First:

@validate(validators=dict(
a=validators.String(), b=validators.Int()
))
@expose()
def foo(self, a=None, b=None):
return "a=%r b=%r" % (a, b)

/foo -> a='' b=None
/foo?a=x -> a='x' b=None
/foo?a=x&b=0 -> a='x' b=0

Then:
@validate(validators=dict(
a=validators.String(), b=validators.Int()
))
@expose()
def foo(self, a="bar", b=1):
return "a=%r b=%r" % (a, b)

/foo -> a='' b=None
/foo?a=x -> a='x' b=None
/foo?a=x&b=0 -> a='x' b=0

Basically default values are not taken into account, and will be
replaced by the validator's to_python result
which for String() is '' and for Int() is None

While this is 'normal' given the code i showed above for
_perform_validate, it might be unexpected and deserves at least
documentation
The 'correct' way to handle default values is to use:

@validate(validators=dict(
a=validators.String(if_empty='bar'), b=validators.Int(if_empty=1)
))
@expose()
def foo(self, a=None, b=None):
return "a=%r b=%r" % (a, b)

/foo -> a='bar' b=1
/foo?a=x -> a='x' b=1
/foo?a=x&b=0 -> a='x' b=0

1.3) @validate default error_handling when using validators=dict()
when using @validate(validators=dict(...)) when there is a validation
error and no error_handler explicitely defined, the
default behavior is to call the method regardless with unvalidated
arguments, which is surprising. The only way to detect that errors
happened
is to look at 'pylons.c.form_errors' which is then a non-empty
dictionary. Maybe the default should be to raise an Exception
triggering a 500 internal error ?
or some other form of error preventing the controller code to be
executed with bogus arguments

@validate(validators=dict(
a=validators.Int()
))
@expose()
def foo(self, a=None):
return "a=%r" % a

/foo -> a=None
/foo?a=x -> a='x' ... unexpected, but can be detected by inspecting
pylons.c.form_errors
/foo?a=1 -> a=1

2) Routes
It is advertised that TG2 supports custom routes (using the routes
package) as well as a default object-dispatch mechanism.
The object dispatch is actually implemented as a stub, catch-all
route, that redirects to a special controller.

This controller (TGController) inherits from pylon's WSGIController
and overrides _perform_call to do the dispatch magic
instead of letting pylons do the 'regular' route dispatch.

def _perform_call(self, func, args):
controller, remainder, params = self._get_routing_info(args.get
('url'))
func_name = func.__name__
if func_name == '__before__' or func_name == '__after__':
if hasattr(controller.im_class, '__before__'):
return controller.im_self.__before__(*args)
if hasattr(controller.im_class, '__after__'):
return controller.im_self.__before__(*args)
return
return DecoratedController._perform_call(
self, controller, params, remainder=remainder)

As we can see tg2 ignores the 'args' coming from pylons and assumes
there's going to be only the 'url' key then goes on with object
dispatch.
This means that I can't use a custom route leading to a method inside
a subclass of TGController (or BaseController as defined in a
quickstarted project)
and if I do it, all routes arguments will be ignored, for example in
the route mapping in app_cfg:

map.connect('foo/:page/:filter', controller="root", action="foo"
page=1, filter="all")
map.connect('*url', controller='root', action="routes_placeholder")

calling /foo/1/2 will result in foo(*['1', '2'], **{}) being called

Next I tried to bypass the object dispatch using a controller that
didn't inherit from 'BaseController(TGController)' but from
'Controller' (both defined in $project/lib/base.py).
It turns out that 'Controller' inherits from 'object' meaning that I
simply can't use it since Pylons expect a __call__ method on a
controller.

in $project/controllers/foo.py:

from tg import expose
from $project.lib.base import Controller
class FooController(Controller):
@expose()
def foo(self):
return ""

Calling /foo leads to:
File '/home/kikidonk/Whatever/tg2/lib/python2.5/site-packages/
Routes-1.10.2-py2.5.egg/routes/middleware.py', line 118 in __call__
response = self.app(environ, start_response)
File '/home/kikidonk/Whatever/tg2/lib/python2.5/site-packages/
Pylons-0.9.7rc4-py2.5.egg/pylons/wsgiapp.py', line 117 in __call__
response = self.dispatch(controller, environ, start_response)
File '/home/kikidonk/Whatever/tg2/lib/python2.5/site-packages/
Pylons-0.9.7rc4-py2.5.egg/pylons/wsgiapp.py', line 316 in dispatch
return controller(environ, start_response)
TypeError: 'FooController' object is not callable

In tg2's tg/controllers.py there is a better parent class i could use
(from which TGController inherits): DecoratedController which also
seems
to perform the decoration magic for expose/validate, etc
I'm guessing that the base.py's 'Controller' should inherit from that
DecoratedController to work properly.

The problem if I do that is that now, pylons will try to call
__before__ and __after__ (since they are both defined in that class)
and those methods have no decoration, leading to an error:

from tg.controllers import DecoratedController
from tg import expose
class FooController(DecoratedController):
@expose()
def foo(self):
return ""

File '/home/kikidonk/Whatever/tg2/lib/python2.5/site-packages/
Pylons-0.9.7rc4-py2.5.egg/pylons/wsgiapp.py', line 117 in __call__
response = self.dispatch(controller, environ, start_response)
File '/home/kikidonk/Whatever/tg2/lib/python2.5/site-packages/
Pylons-0.9.7rc4-py2.5.egg/pylons/wsgiapp.py', line 316 in dispatch
return controller(environ, start_response)
File '/home/kikidonk/Whatever/tg2/lib/python2.5/site-packages/
Pylons-0.9.7rc4-py2.5.egg/pylons/controllers/core.py', line 200 in
__call__
response = self._inspect_call(self.__before__)
File '/home/kikidonk/Whatever/tg2/lib/python2.5/site-packages/
Pylons-0.9.7rc4-py2.5.egg/pylons/controllers/core.py', line 95 in
_inspect_call
result = self._perform_call(func, args)
File '/home/kikidonk/Whatever/tg2/lib/python2.5/site-packages/
TurboGears2-2.0b5.1-py2.5.egg/tg/controllers.py', line 109 in
_perform_call
controller.decoration.run_hooks('before_validate', remainder,
AttributeError: 'function' object has no attribute 'decoration'

where function is __before__ (<bound method FooController.__before__
of <$project.controllers.foo.FooController object at ...
0xa6e074c>>)

I must admit i'm a bit puzzled here on what to do, probably removing
__before__ and __after__ would work, i'm not sure...
I'm tempted to conclude that I can't currently have a controller that
uses TG2's decorators, and at the same time uses completely custom
routes dispatch?

I hope I didn't lose any readers at this point :)


Now for the record I worked around these issues with the following
hack:
I defined a decorator:
def validate_routes():
def routes_args(remainder, params):
# Empty remainder arguments, that are passed for routes
for i in range(len(remainder)):
remainder.pop()

# Create a copy of the routes dict
routes_params = request.environ['pylons.routes_dict'].copy()
# Remove cruft
routes_params.pop('controller', None)
routes_params.pop('action', None)
routes_params.pop('url', None)
# Update params with routes params, and let the validation
proceed as usual
params.update(routes_params)

def decorate_method(decorated_method):
return decorators.before_validate(routes_args)
(decorated_method)
return decorate_method

Which basically removes everything from the positional arguments list,
then adds to the keyword args the parameters from the routes matching.
The validator runs before validation, so that validation works on all
parameters from route, and for the route setup i use for example:

m.connect('public/:top/:page', controller='root',
action='routes_placeholder',
page=1, top=None,
requirements=dict(top='day|week|month', page=r'\d+'),
)

Then the controller looks like this, in a subclass of TGController:

@validate_routes()
@validate(dict(
page=validators.Int(if_empty=1),
top=validators.String(),
))
@expose("public.mak")
def public(self, page, top):
return "foo"

So that calling /public/day/2 will result in
public(self, **{top: 'day', page: 2}) being called and everything
works as expected
calling it with /public will result in
public(self, **{top: '', page: 1}) being called and everything works
as expected

It's ugly but works for now...

[Mark Ramm]

Wow, this is quite a thorough post!

Yea, validators and the @validate decorator are designed to validate user input from get params, or post values, and those are always keyword arguments.   We're not validating the positional params because those come from the routing process, and the standard TG idiom has been to handle conversion/validation of those params internally to the controller method, though I do think that perhaps we could do this better in TG.

I don't know if a @validate_routes decorator is exactly the right API, but I like the direction you're going.   Perhaps we could open a ticket for 2.1 to make something like this standard.  Unfortunately it's too late for changes to 2.0 since we're in feature freeze, and this is actually a new feature, not a bug or backwards incompatibilty issue.

As for the issue of validation failure calling the method with unvalidated arguments, that's only the case if you don't also define an error handler, which you mention.   There was some discussion of this in the past, and it was decided to keep this behavior, but if we can resolve those issues, this is something that we may also want to clean up for 2.1.

And I'm not exactly sure what's happening to you with DecoratedController not working, but that's likely a bug which should be filed against 2.0rc1 so that we can resolve it before releasing the release candidate.   It may already be fixed in trunk, I remember seeing something like it, but can't find the discussion at the moment.

--
Mark Ramm-Christensen
email: mark at compoundthinking dot com
blog: www.compoundthinking.com/blog