Form handling in Pyramid

[Mike Orr]

As part of the Pylons-to-Pyramid howto I'm writing, I need to make
some recommendations for form handling. This covers the sprawling
topic of form generation, validation, error display, usage pattern in
the view, and side topics such as database records,
internationalization, and REST resources.

Pylons 1 includes a minimum form pattern based FormEncode, WebHelpers,
and @validate. Pyramid does not have any form dependencies; it's up to
you to choose a library. A baseline scenario using the closest
equivalent gives us:

Form creation: WebHelpers HTML tags
Validation: FormEncode
Error display: FormEncode htmlfill
Usage: pyramid_simpleform (new package by zeemonkey)
Usage: two-view (one for form, another for submission)
Database records: WebHelpers ModelTags
Internationalization: FormEncode messages property
REST: pyramid_routehelper (not written yet; equivalent to Routes
map.resource)

This does everything except form generation; instead it provides
helpers to create your own form in your template. The question is, is
this still the best usage pattern to promote in 2011, and if so, are
these the best packages to do it? I also need to survey the various
alternative form packages and rank them into beginner/advanced, and
note how widely used and supported they are. So this email is to
collect feedback for that: what are people using, and what do they
think makes sense for Pyramid.

My own feeling is to stick with this pattern as the first
recommendation. It's pyramidesque and flexible. Mako has a
call-with-content feature and a new unobtrusive syntax that allows
template functions to look almost like built-in template commands; we
have never sufficiently explored this as a basis for a form library. I
also like how FormEncode's validators can be used generally for
validation and conversion: I use them to check and transform the INI
settings at startup. I also use formencode.declarative as a general
base class because it handles constructor args for you.

However, FormEncode is 6+ years old, its documentation is somewhat
confusing for using it in a Pylons/Pyramid context, some things are
undocumented, and the source has some code that looks like dead ends
(backward compatibility only). I've been meaning to write a new
FormEncode manual for years but have never had time to. So maybe it's
time to streamline it into a new package.

Which brings us to Formish and Deform. At first glance, Formish seems
to be a streamlining of FormEncode into discrete classes for schemas,
validation, type conversion (which it separates from validation), with
specific support for nested form data and file uploads. It also adds a
layer for form generation using Mako. Deform, by our own ChrisM, is a
reimplementation of the Formish concept using different libraries and
Chameleon templates.

So, for those who've used them, is this true? Are they equivalent to
FormEncode, or are they missing any of FormEncode's functionality? How
well does it work separating validation from type conversion? To me it
seems like you have to type convert in order to validate, and a failed
conversion is a failed validation, so I don't really see the point in
separating them, and I'm concerned the schema definitions may be more
verbose than FormEncode's is. Has there been any work on Mako
templates for Deform? What has users' experience with these libraries
been?

There's also WTForms, which we considered recently for Pylons 2. It
has a simple interface like Django newforms which we liked, but it
wasn't as flexible as FormEncode and the validators seemed like they
weren't designed for non-web use, so we were afraid of losing some
existing functionality. Is anyone using WTForms with Pylons or Pyramid
who would like to comment?

ToscaWidgets has always been popular but has been reputed as harder to
learn and bigger, so it seems to belong to the "advanced forms"
category.

Then there are several form/model generators tied to SQLAlchemy, such
as FormAlchemy, GeoFormAlchemy, Sprox (which seems to have replaced
DBSprockets?), etc. I'm not sure of the differences between these.
Those tied to SQLAlchemy are subset of form generation, so they would
go in a subsection.

Another issue is one-view vs two-view form processing. The tradtional
way (before I came to Pylons) was to use one view with an 'if' that
would display the form if there are no POST variables, or validate it
if there are, and an inner 'if' to do the action if the validation
succeeds, or to redisplay the form with errors if it fails. Pylons
@validate went with a two-form approach, which means you have to call
the form view from the submission view if the validation fails. And
REST essentially requires a two-vlew approach because the form and
submission are on different URLs, and the same URL is doubled up for
submitting one form (PUT or POST) and displaying a different page
(GET). So different URLs for form and submission require two views.
Most people have kept the two-view approach with @validate even for
non-REST forms. I think it would be best to support both approaches,
two-view and one-view.

One thing I was concerned about in pyramid_simpleform
(http://packages.python.org/pyramid_simpleform/) is that the view
handler has the form view and submission view that look almost the
same. It's like one could go to either URL? What's the purpose of
this, and how does it handle the case of failed validation?

Anyway, if you have any ideas on what should be in the Pyramid-Pylons
form howto (which will probably be the Pyramid form howto too, since
there isn't one of those I could find), now's the time to speak up.

--
Mike Orr <slugg...@gmail.com>

[Gael Pasgrimaud]

Hi,

I don't really have something to say because I'm using FormAlchemy for
years now and it work fine with pyramid (and all webob based
application) too so I will continue to use it.
So I'm really not objective. I just want to end up the misunderstanding.

On Mon, Jan 17, 2011 at 8:23 PM, Mike Orr <slugg...@gmail.com> wrote:
(...)

>
> Then there are several form/model generators tied to SQLAlchemy, such
> as FormAlchemy, GeoFormAlchemy, Sprox (which seems to have replaced
> DBSprockets?), etc. I'm not sure of the differences between these.
> Those tied to SQLAlchemy are subset of form generation, so they would
> go in a subsection.
>

That's not true. FormAlchemy does not rely on SQLAlchemy although it
is a required dependency.
I've wrote some subclass of FormAlchemy's Field and FieldSet to
implement other backends.

As an example there is an implementation for zope.schema:

from formalchemy.ext.zope import FieldSet
from zope.interface import Interface, implements
from zope import schema

class IObj(Interface):
id = schema.TextLine()

class Obj(object):
implements(IObj)

fs = FieldSet(IObj)
obj = Obj()
fs.rebind(obj, data={'Obj--id': 'my id'})
if fs.validate():
fs.sync()
assert obj.id == 'my id'
print fs.render()

This will output:

<div>
<label class="field_req" for="Obj--id">id</label>
<input id="Obj--id" name="Obj--id" type="text" value="my id" />
</div>

<script type="text/javascript">
//<![CDATA[
document.getElementById("Obj--id").focus();
//]]>
</script>

You can read more about this in the documentation:
http://docs.formalchemy.org/formalchemy/ext/zope.html

There is also a ldap implementation not listed in the documentation:
http://pypi.python.org/pypi/afpy.ldap

But... this is totally out of topic. Sorry for the noise.

Regards,

--
Gael

[Wichert Akkerman]

On 2011-1-17 20:23, Mike Orr wrote:
> So, for those who've used them, is this true? Are they equivalent to
> FormEncode, or are they missing any of FormEncode's functionality?

I have used formencode a lot in a Pylons 0.x context, and I've used a
lot of formish in repoze.bfg/Pyramid contexts. I have not used Deform so
far, so I can't comment on that.

From my perspective formish felt like an improvement over FormEncode:
it has sane abstractions of schema definition, validation and form
rendering, which is an area where FormEncode always felt unclean to me.
Personally I never used html generation for forms, and formish was a lot
friendlier to support that than FormEncode was to me. I also had the
feeling FormEncode was no longer maintained: bugreports did not get
answers and patches were never applied.

Having said that I am not quite happy with formish either, especially
its handling of error messages which breaks i18n very badly. The formish
developers also have been lacking time to respond to problems and
continue its development recently, which makes me worry.

Deform seems like a nicer and better maintained version of formish, so
I'm tempted to try it for a next project. The only thing that irks me
about it is the way it does marshalling of fields as implemented by
colander. I've once promised Chris I would try a simpler and possibly
less capable alternative for that, but so far have not had the time to
do that.

Wichert.

--
Wichert Akkerman <wic...@wiggy.net> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.

[Ian Wilson]

Hello Mike,

On Mon, Jan 17, 2011 at 11:23 AM, Mike Orr <slugg...@gmail.com> wrote:

So this email is to
collect feedback for that: what are people using, and what do they
think makes sense for Pyramid.

I'm still using formencode for validation/conversion.  I use a custom class that encapsulates the minimum functionality necessary for validation use cases(http://packages.python.org/formprocess/).  I use htmlfill to fill the forms. I write some html and some mako helper functions to create the form html.  In web forms I use one method to display the form, one method to process the form and I have a shared method which renders the form for both cases(initial display and processing errors display).  I have not had many problems with this pattern except the obvious warts in formencode which are slim documentation, odd conventions and seemingly unnecessary features.  I have no experience with internationalization.

I haven't switched to pyramid yet but I would expect that I would just keep using the same tools and patterns.  I am interested in new solutions though and believe one day a tiered solution could be developed to finally solve this problem for almost everyone.

Which brings us to Formish and Deform. At first glance, Formish seems
to be a streamlining of FormEncode into discrete classes for schemas,
validation, type conversion (which it separates from validation), with
specific support for nested form data and file uploads. It also adds a
layer for form generation using Mako. Deform, by our own ChrisM, is a
reimplementation of the Formish concept using different libraries and
Chameleon templates.

Maybe a feature/library matrix could be compiled somewhere to compare all these libraries and test their support of certain features?  I could help with that if you think that would be helpful.
 

Another issue is one-view vs two-view form processing. The tradtional
way (before I came to Pylons) was to use one view with an 'if' that
would display the form if there are no POST variables, or validate it
if there are, and an inner 'if' to do the action if the validation
succeeds, or to redisplay the form with errors if it fails.

 

It might be important to differentiate between one/two urls and one/two views since any combination of these could occur:
* There could be one url with two views(views matching on the method -- one for GET and one for POST). 
* There could be one url with one view(handles both GET and POST with conditional)
* There could be two urls with two views(one for GET and one for POST).
* There could be two urls with one view (view handles both GET and POST with conditional)


-Ian

[Holger Frey]

Hello,

I am very interested in this topic since in my experience form
handling is always a pain.
Did you consider the "flatland" form processing package (http://
discorporate.us/projects/flatland/). I know this is a young project
but I think its a balanced compromise of schema creation, validation
and helpers for rendering. I stumbled across this package on a quick
talk on the new moinmoin 2 project (http://moinmo.in/MoinMoin2.0)

regards

Holger

[Mike Orr]

How does Pyramid handle HTTP method tunneling? Where the formal method
is POST and the actual method is in a '_method' param? it's not really
a form library issue because you'd want it uniform across all
libraries. How does Pylons do it? Is it in WebOb? Is it something we
need to add?

--
Mike Orr <slugg...@gmail.com>

[Chris McDonough]

It doesn't handle it. I need to see credible examples use cases for it,
I think.

Various folks have wanted to add e.g. <input type="hidden"
name="_method" value="PUT"/> to a Pyramid form post in my tenure in
#pylons, but it's beyond me why someone would try to be emulating a
"REST client" when using a browser, as the access patterns and expected
responses for REST APIs are dramatically different than those of a user
using a browser (authentication is usually different, usually REST
methods don't return HTML, etc).

- C

[Daniel Holth]

I always setup two view functions, one for GET and one for POST.

I don't do PUT, but if you want Pyramid lets you condition a view on both the request method and parameters. The second function should only be called for a PUT, or a POST with _method=PUT in the parameters:

def some_method(request):
   # handle everything else

def some_method_PUT(request):
    # handle the PUT

...

config.add_view(some_method)
config.add_view(some_method_PUT, request_method='PUT')
config.add_view(some_method_PUT, request_method='POST', request_param="_method=PUT")

[Chris McDonough]

True, I guess we handle it already by default, so unless there's some
usage pattern I don't understand (likely), we don't really need to argue
about supporting it or not.

- C

[Mike Orr]

It's a tradition in Pylons to use the Atom REST protocol for model
objects with add/modify/delete forms. I've been half questioning that
myself because there's little advantage if all the clients are
interactive web browsers. But regardless of whether it's been
oversold, it's the way most Pylons users do their forms, so it needs a
straightforward migration path. WebHelpers form() helper automatically
converts the method if it's not POST or GET, and a lot of people will
be using that and expecting it to be converted back for their routes.

> --
> You received this message because you are subscribed to the Google Groups "pylons-devel" group.
> To post to this group, send email to pylons...@googlegroups.com.
> To unsubscribe from this group, send email to pylons-devel...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/pylons-devel?hl=en.
>
>

--
Mike Orr <slugg...@gmail.com>

[Mike Orr]

We at least need to document it as a FAQ, and mention it in the
routing section and view section.

This syntax does make the routes less straightforward (they show the
low-level details rather than the high-level intention). If the
application is expecting both tunneled PUT and direct PUT, it will
have to accomodate both possibilities. And views will also have to
handle tunneled PUT themselves because it won't be converted.
(Although that could be done in the view handler .__init__, but that's
kind of a band-aid approach because it covers only the views and not
the routes.)

--
Mike Orr <slugg...@gmail.com>

[Chris McDonough]

On Fri, 2011-01-21 at 12:20 -0800, Mike Orr wrote:
> It's a tradition in Pylons to use the Atom REST protocol for model
> objects with add/modify/delete forms. I've been half questioning that
> myself because there's little advantage if all the clients are
> interactive web browsers. But regardless of whether it's been
> oversold, it's the way most Pylons users do their forms, so it needs a
> straightforward migration path. WebHelpers form() helper automatically
> converts the method if it's not POST or GET, and a lot of people will
> be using that and expecting it to be converted back for their routes.

I think that's a different issue than just handling "_method", because
it implies a predefined set of view actions associated with a route.

Various people have been threatening to write a pyramid_restcontroller
thing that emulates Pylons' restcontroller for a while now.

I'm not going to do it myself, because I think it's a fairly poor
pattern (especially for web browser access), and I don't want to
encourage that.

On the other hand, if folks want to do it, it's fine by me (just upload
a pyramid_restcontrollers package to PyPI and document it) but we're
really going to have to move on without things unless people are willing
to get them done in a timely way.

- C

[Mike Orr]

On Fri, Jan 21, 2011 at 11:55 AM, Mike Orr <slugg...@gmail.com> wrote:
> How does Pyramid handle HTTP method tunneling? Where the formal method
> is POST and the actual method is in a '_method' param? it's not really
> a form library issue because you'd want it uniform across all
> libraries. How does Pylons do it?

Pylons does it in the Routes middleware
(routes.middleware.RoutesMiddleware). There's a constructor arg
'use_method_override', default true.

--
Mike Orr <slugg...@gmail.com>

[Chris McDonough]

On Fri, 2011-01-21 at 12:30 -0800, Mike Orr wrote:
> On Fri, Jan 21, 2011 at 12:15 PM, Chris McDonough <chr...@plope.com> wrote:
> > On Fri, 2011-01-21 at 12:13 -0800, Daniel Holth wrote:
> >> I always setup two view functions, one for GET and one for POST.
> >>
> >> I don't do PUT, but if you want Pyramid lets you condition a view on
> >> both the request method and parameters. The second function should
> >> only be called for a PUT, or a POST with _method=PUT in the
> >> parameters:
> >>
> >> def some_method(request):
> >> # handle everything else
> >>
> >> def some_method_PUT(request):
> >> # handle the PUT
> >>
> >> ...
> >>
> >> config.add_view(some_method)
> >> config.add_view(some_method_PUT, request_method='PUT')
> >> config.add_view(some_method_PUT, request_method='POST',
> >> request_param="_method=PUT")
> >
> > True, I guess we handle it already by default, so unless there's some
> > usage pattern I don't understand (likely), we don't really need to argue
> > about supporting it or not.
>
> We at least need to document it as a FAQ, and mention it in the
> routing section and view section.

It could and probably should be a cookbook entry.

> This syntax does make the routes less straightforward (they show the
> low-level details rather than the high-level intention). If the
> application is expecting both tunneled PUT and direct PUT, it will
> have to accomodate both possibilities. And views will also have to
> handle tunneled PUT themselves because it won't be converted.
> (Although that could be done in the view handler .__init__, but that's
> kind of a band-aid approach because it covers only the views and not
> the routes.)

What is the distinction you're trying to make between "PUT" and
"tunneled PUT"? Are you saying that a view should return something
different to a browser client than to a non-browser client based on
whether the _method param exists?

- C

[Mike Orr]

I'm saying that a PUT-capable client will send a real PUT, while a
browser will send a tunneled PUT. The difference may come down to
whether Javascript is enabled in the browser thus allowing an Ajax
request, with a fallback using an HTML form for non-Javascript
browsers. If the view and routing are expecting only one kind of PUT,
they may mishandle the other kind. It's easier to just convert the
request than to explain the problem to application developers.

--
Mike Orr <slugg...@gmail.com>

[Daniel Holth]

I think it is very relaxing that my view handles unedited requests. I'm confused about the problem you are having. Since the framework keeps GET requests out, why does the PUT-handler have to check request.method at all?

[Mike Orr]

There are two issues. Going back to Chris's routing example, although
I'll change add_view to add_route because I'm mainly thining of URL
Dispatch.

> config.add_route(NAME, PATTERN, VIEW, request_method='PUT')
> config.add_route(NAME, PATTERN, VIEW, request_method='POST',
> request_param="_method=PUT")

If the appdev omits one of these, either because they don't know about
tunneled PUT or because they underestimate the variety of clients and
uses, the URL won't match at all and either the user will get a 404 or
a later (wrong) route will match.

In the view, if it's a single-purpose view that handles only record
modification (i.e., a submission from a form or the non-form
equivalent), it won't care what the method is. As a corollary, it will
have to call another view (or return 400 status) if the data fails
validation, because it doesn't have the form-display code.

If it's a multi-purpose view, the user better check for both kinds of
PUTs or DELETEs, because if he naively does ``if self.request.method
== 'POST'`:` or ``if self.request.method == 'PUT':``, the result will
be wrong if the other PUT style was used. So maybe they should do ``if
self.request.method != 'GET':`` instead, although that raises the
possibility that the method may be entirely unexpected ('DELETE'
instead of 'PUT'), in which case again the result is wrong. (It
*should* return 405 status if the method is inappropriate for the
view., and not just act like it were a different method.)

--
Mike Orr <slugg...@gmail.com>

[Chris McDonough]

On Fri, 2011-01-21 at 13:58 -0800, Mike Orr wrote:
> On Fri, Jan 21, 2011 at 1:13 PM, Daniel Holth <dho...@gmail.com> wrote:
> > I think it is very relaxing that my view handles unedited requests. I'm
> > confused about the problem you are having. Since the framework keeps GET
> > requests out, why does the PUT-handler have to check request.method at all?
>
> There are two issues. Going back to Chris's routing example, although
> I'll change add_view to add_route because I'm mainly thining of URL
> Dispatch.
>
> > config.add_route(NAME, PATTERN, VIEW, request_method='PUT')
> > config.add_route(NAME, PATTERN, VIEW, request_method='POST',
> > request_param="_method=PUT")
>
> If the appdev omits one of these, either because they don't know about
> tunneled PUT or because they underestimate the variety of clients and
> uses, the URL won't match at all and either the user will get a 404 or
> a later (wrong) route will match.

If the problem is distinguishing here, two routes are not necessary
here. Just a single route and two views (or handler actions).

config.add_route('thename', '/foo/put', 'theview', request_method='PUT')

@view_config(route_name='thename', request_param='_method',
request_method='PUT')
def tunneled_put(request):
...

@view_config(route_name='thename', request_method='PUT')
def untunneled_put(request):
....

> In the view, if it's a single-purpose view that handles only record
> modification (i.e., a submission from a form or the non-form
> equivalent), it won't care what the method is. As a corollary, it will
> have to call another view (or return 400 status) if the data fails
> validation, because it doesn't have the form-display code.
>
> If it's a multi-purpose view, the user better check for both kinds of
> PUTs or DELETEs, because if he naively does ``if self.request.method
> == 'POST'`:` or ``if self.request.method == 'PUT':``, the result will
> be wrong if the other PUT style was used. So maybe they should do ``if
> self.request.method != 'GET':`` instead, although that raises the
> possibility that the method may be entirely unexpected ('DELETE'
> instead of 'PUT'), in which case again the result is wrong. (It
> *should* return 405 status if the method is inappropriate for the
> view., and not just act like it were a different method.)

I'd just let the framework decide which view to call if they need to do
very distinct things. If this needs to be codified into some pattern, a
restcontroller-esque package can make the necessary view registrations
against named methods of a handler.

- C

[Chris McDonough]

On Fri, 2011-01-21 at 17:10 -0500, Chris McDonough wrote:
> On Fri, 2011-01-21 at 13:58 -0800, Mike Orr wrote:

> If the problem is distinguishing here, two routes are not necessary
> here. Just a single route and two views (or handler actions).
>
> config.add_route('thename', '/foo/put', 'theview', request_method='PUT')

Whoops, that should have read:

config.add_route('thename', '/foo/put', 'theview')

[Chris McDonough]

On Fri, 2011-01-21 at 17:18 -0500, Chris McDonough wrote:
> On Fri, 2011-01-21 at 17:10 -0500, Chris McDonough wrote:
> > On Fri, 2011-01-21 at 13:58 -0800, Mike Orr wrote:
>
> > If the problem is distinguishing here, two routes are not necessary
> > here. Just a single route and two views (or handler actions).
> >
> > config.add_route('thename', '/foo/put', 'theview', request_method='PUT')
>
> Whoops, that should have read:
>
> config.add_route('thename', '/foo/put', 'theview')
>
>
> > @view_config(route_name='thename', request_param='_method',
> > request_method='PUT')
> > def tunneled_put(request):

And that should have read:

@view_config(route_name='thename', request_param='_method=PUT')
def tunneled_put(request):
....

Apologies,

- C

[ZHUO Qiang]

Hi,

Since I find the old pylons's way of RESTful HTTP "_method" tunnel very handy, I have the following customize predicts to emulate the same effects in Pyramid:

# this predicts will using the both real HTTP method and "_method" for prediction. And it will also convert the request.method with the "_method" if necessary
def allowed_methods(*allowed):
    def predicate(info, request):
        if request.method == 'POST':
            request.method = request.params.get('_method', request.method).upper()
        if request.method in allowed:
            return True
    return predicate

# Here we use the predicts to define a route only for DELETE (including the real DELETE and POST tunneled DELETE)
config.add_route('picture_delete', r'picture/{id}', custom_predicates=(allowed_methods('DELETE'),)) 

I kindly hope some similar help function could go into Pyramid package, or at least have a FAQ | Cookbook entry for this particular requirement

--
http://zhuoqiang.me

[Reed L O'Brien]

I don't know what the old tunneling method did but you can define a view only for DELETE using the request_method arg.

config.add_route('picture_delete', r'picture/{id}', request_mehthod='DELETE')

see http://is.gd/nq0vlm

If the above is unrelated there is https://github.com/Pylons/pyramid_cookbook

Cheers,
~ro

[karantan]

Hi,

So is there any news if pyramid will have anything like "from
pylons.decorators import validate"? i tried to use formencode but it's
not so easy to implement in pyramid. i will try formish next and i
hope i'll have more success with it...

I would also like to say that it is great that you implemented auth in
pyramid (really love it) but i think that you should implement form
handling/validating (just the one pylons has it? -
pylons.decorators.validate)

best karantan

[Dan Jacob]

The @validate decorator was an appallingly bad idea and one of the
most detested features of Pylons. I don't think anyone wants to bring
it back.

[Mike Orr]

On Tue, Mar 29, 2011 at 4:06 AM, karantan <kara...@gmail.com> wrote:
> Hi,
>
> So is there any news if pyramid will have anything like "from
> pylons.decorators import validate"? i tried to use formencode but it's
> not so easy to implement in pyramid. i will try formish next and i
> hope i'll have more success with it...

Have you tried pyramid_simpeform? It's a different interface than
@validate but it seems to be becoming the most common on on Pyramid.

--
Mike Orr <slugg...@gmail.com>