yesod-form validation

[Luite Stegeman]

hi,

I've been trying to use yesod-form (0.2.0.1) in my web application,
but unfortunately I have run into some limitations. I'd like the form
library to handle more complex validation than it seems to be designed
for. While it's possible to handle all example situations (below)
outside the form library, I think it's much nicer if yesod-form
supported them directly, and this makes it much easier to display
validation error messages the same way as the field parser errors.

Please note that I'm still fairly new to yesod-form and to yesod in
general, so please correct me if things I say don't make sense or are
plainly wrong.

I'd like to see support for the following situations in applicative
forms:

1. Simple (pure) validation, for example a ranged integer field

2. Advanced validation: A date field that allows only dates in the
future, a username field that only allows usernames that aren't
already taken.

3. Form consistency: Are the contents of the two password fields the
same? Is the selected payment option compatible with the shipping
option?

I think none of these is properly supported by yesod-form:

1. While it's possible to do this with fieldParse, I think it's not
the right place to do so. The fieldParse method is to convert the user
input to the appropriate Haskell value, additional restrictions should
be placed somewhere else, in a more composable way (usually these
restriction don't have much to do with the parser)

2. It doesn't seem possible to to this with the current version of
yesod-form, however if support for explicit validators (item 1.) is
added, this can be supported if they have the correct type.

3. More difficult, I don't have code for this at the moment. Could
probably be done by adding validators for the result type of the whole
form to fields. The field to which the validator is added could
determine the placement of the error message.

I have some preliminary code to support 1. and 2. at: http://hpaste.org/49708

The key is the new Validator msg m a = a -> m (Maybe msg) type. A
Validator returns Nothing if the field is ok, an error message
otherwise (the Maybe type is a bit counterintuitive for this in my
opinion). The VFieldSettings type gets a Maybe Validator field

I had hoped to do this by reimplementing just a few functions, but in
the end almost all of the yesod-form code for this is duplicated.

Here's an example of how I use the form:

profileForm p uid = case p of
Nothing -> profileForm' p uid (vreq (jqueryValidateRemoteTextField
UserExistsR [] [])
"Username" { vfsValidate = Just validateUserExists }
Nothing)
Just pr -> profileForm' p uid (vshow outputField
"Username" (profileUserName pr))

profileForm' p uid field = renderVDivs $ Profile
<$> pure uid
<*> field
<*> vopt textField "Full name" (profileRealName <$> p)
<*> vopt urlField "Website" (profileWebsite <$> p)
<*> vreq boolField "Hide email" (profileHideEmail <$> p)
<*> vopt textareaField "Description" (profileDesc <$> p)
<*> pure False

validateUserExists :: T.Text -> Handler (Maybe T.Text)
validateUserExists t = do
muser <- runDB . getBy . UniqueProfileUsername $ t
case muser of
Nothing -> return Nothing
Just _ -> return . Just . T.concat $ ["User '", t, "' already
exists"]

Actually, this form shows another limitation of yesod-form: If the
user already exists (p /= Nothing), then the formlet is different: It
just displays the username, the user cannot change it anymore after
the first time. This means that when creating a profile, the new form
should be displayed, while the current POST data is for the old one.
Therefore, the POST data must be ignored. I believe the same problem
occurs when running a multi-step (wizard style) form.

To solve this, I run the form with the following function after
creating the profile in the database:

{-
Run a form with POST method, but ignore all submitted values
(useful if we have to show a different form after submitting)
-}
runFormPostDefault :: RenderMessage master FormMessage =>
(Html -> Form master (GHandler sub master)
(FormResult a, xml)) ->
GHandler sub master ((FormResult a, xml),
Enctype)
runFormPostDefault form = do
req <- getRequest
let nonceKey = "_nonce" :: T.Text
let nonce =
case reqNonce req of
Nothing -> mempty
Just n -> [hamlet|<input type=hidden name=#{nonceKey}
value=#{n}>|]
m <- getYesod
langs <- languages
((res, xml), enctype) <- runFormGeneric (form nonce) m langs
Nothing
return ((res, xml), enctype)

Now I'd like to know, is this the right approach to do form validation
with yesod? Any suggestions for improvements, maybe alternatives,
things I have overlooked? Any comment is welcome!

cheers,

Luite

[Michael Snoyman]

On Sun, Jul 31, 2011 at 1:42 PM, Luite Stegeman <steg...@gmail.com> wrote:
> hi,
>
> I've been trying to use yesod-form (0.2.0.1) in my web application,
> but unfortunately I have run into some limitations. I'd like the form
> library to handle more complex validation than it seems to be designed
> for. While it's possible to handle all example situations (below)
> outside the form library, I think it's much nicer if yesod-form
> supported them directly, and this makes it much easier to display
> validation error messages the same way as the field parser errors.
>
> Please note that I'm still fairly new to yesod-form and to yesod in
> general, so please correct me if things I say don't make sense or are
> plainly wrong.

You're absolutely correct that the current design needs to be fixed,
and I'm very glad you've taken the time to address it.

> I'd like to see support for the following situations in applicative
> forms:
>
> 1. Simple (pure) validation, for example a ranged integer field
>
> 2. Advanced validation: A date field that allows only dates in the
> future, a username field that only allows usernames that aren't
> already taken.

It seems the only difference between these two is that (2) involves
monadic actions, right?

> 3. Form consistency: Are the contents of the two password fields the
> same? Is the selected payment option compatible with the shipping
> option?
>
> I think none of these is properly supported by yesod-form:
>
> 1. While it's possible to do this with fieldParse, I think it's not
> the right place to do so. The fieldParse method is to convert the user
> input to the appropriate Haskell value, additional restrictions should
> be placed somewhere else, in a more composable way (usually these
> restriction don't have much to do with the parser)

I disagree here, I think fieldParse is exactly the place for this. You
can say it's not part of parsing, but then I'd just say we need to
rename fieldParse to something else. As for composable, it seems
pretty straight-forward to be able to string together a bunch of
"Either ParseError a" functions (it's just the Monad instance for
Either).

> 2. It doesn't seem possible to to this with the current version of
> yesod-form, however if support for explicit validators (item 1.) is
> added, this can be supported if they have the correct type.

If we go in the direction of using fieldParse for this, we could just
change fieldParse to return a value inside the monad.

> 3. More difficult, I don't have code for this at the moment. Could
> probably be done by adding validators for the result type of the whole
> form to fields. The field to which the validator is added could
> determine the placement of the error message.

I've never been able to think of a good way of placing the error
messages for these. And I don't just mean from a programming
perspective; there's no clear UI rule for where these kinds of things
should go. I tend towards putting these messages outside of the form
entirely. Though specifically in the case of passwords, I think it's
well accepted to put the error message on the first password field.

> I have some preliminary code to support 1. and 2. at:  http://hpaste.org/49708
>
> The key is the new   Validator msg m a = a -> m (Maybe msg)   type. A
> Validator returns Nothing if the field is ok, an error message
> otherwise (the Maybe type is a bit counterintuitive for this in my
> opinion). The VFieldSettings type gets a Maybe Validator field
>
> I had hoped to do this by reimplementing just a few functions, but in
> the end almost all of the yesod-form code for this is duplicated.

I think if we (1) change the signature for fieldParse and (2) change
Validator to a -> m (Either msg a), it will work. This also allows us
to double-up validators and modifiers.

Can you clarify the point here? I don't think I follow.

I think you definitely have the right idea here, though if possible
I'd like to avoid introducing yet another type of form. Let's see if
we can work what you've done back into what's already there.

Michael

[Greg Weber]

I haven't dug into the forms implementation yet. My main comment to
Luite was that we should also consider this issue in the future
context of having validations on models- some of the capabilities he
wants may make more sense to implement there.

On Jul 31, 11:39 am, Michael Snoyman <mich...@snoyman.com> wrote:

[Michael Snoyman]

I have to call it quits for the evening, but I just pushed a commit to
add some of the features Luite was describing. However, the commit is
ugly, in the sense that it forces validation to run in the IO monad,
instead of allowing for the Handler monad. That's next on my agenda to
solve. I think at the same time I'm going to make Fields slightly less
generic, and tie them down to using Widgets. My guess is that this
will make error messages just a tiny bit clearer. One other idea is to
write a check function to deal with entire forms instead of just
fields.

If anyone wants to take a crack at this, feel free.

Michael

[Luite Stegeman]

hi,

sorry for the late reply. I typed one yesterday but lost it.

> > I'd like to see support for the following situations in applicative
> > forms:
>
> > 1. Simple (pure) validation, for example a ranged integer field
>
> > 2. Advanced validation: A date field that allows only dates in the
> > future, a username field that only allows usernames that aren't
> > already taken.
>
> It seems the only difference between these two is that (2) involves
> monadic actions, right?

Yes

> I disagree here, I think fieldParse is exactly the place for this. You
> can say it's not part of parsing, but then I'd just say we need to
> rename fieldParse to something else. As for composable, it seems
> pretty straight-forward to be able to string together a bunch of
> "Either ParseError a" functions (it's just the Monad instance for
> Either).

I personally think that there are two problems with this, though the
second might be seen as a personal preference:

1. Usually, form validation is much more application specific than
field parsing. Ideally, yesod-form, or another library, would offer
field parsers for most common input field types. The validation will
often need to be supplied by the application itself. Usually the
validation error messages need to be different from the existing
messages (FormMessage in case of yesod-form fields). If you were to
simply move the fieldParser into IO (or the Handler monad), with a
single msg type, the user would have to rewrite all fields for which
he wanted to get new error messages or somehow convert the messages
from the original field.

2. As for composition, I like it better when the composition of
validators is still a validator. If you move the validator into the
parser, then you could have for example a rangedIntField and an
oddIntField, but you couldn't combine them to make a
rangedOddIntField, unless you had access to the parser and validator
separately. Always having the validator separately gives you more
options for combining it with other validators, for example if you
would add validation rules to the models, then this gives you the
option of running the model validator before or after the extra form
validator fields added by the user. With a Monoid instance for the
error messages, you could also combine two validators and collect both
error messages. (Of course this doesn't really require a separate
place for storing the validator, you could try to combine them with
the parser as late as possible, but this again raises the issue with
the error message types)

That said, it might still be useful to have the parser in the IO or
other monad, some field types might require it to properly parse a
field (for example to read xml schema's)

> > Actually, this form shows another limitation of yesod-form: If the
> > user already exists (p /= Nothing), then the formlet is different: It
> > just displays the username, the user cannot change it anymore after
> > the first time. This means that when creating a profile, the new form
> > should be displayed, while the current POST data is for the old one.
> > Therefore, the POST data must be ignored. I believe the same problem
> > occurs when running a multi-step (wizard style) form.
>
> Can you clarify the point here? I don't think I follow.

I'll try to clarify the example: I have a form for a user profile,
where the form that needs to be displayed after the profile has been
created is different (the username can only be chosen when the user
creates his account, it cannot be changed afterwards). A simplified
version of this idea can be found here:
http://hpaste.org/49821

This is a two-step form: Once the form for Page1 has been submitted
and validated, the application saves the value (in a session in this
example, in the database in a real application) and displays the form
for Page2. With the above code, this (line 49 and 50) goes wrong,
because it runs the Page2 formlet while there is already POST form
data for Page1. Therefore the default Page2 values are never
displayed, and the submitted Page1 form data is incorrectly inserted
into the Page2 form fields.

regards,

Luite

[Michael Snoyman]

Look at the most recent commit. I've added a "SomeMessage" existential
type that allows for multiple message types to be used. I'll cover it
in more depth in a blog post (soon hopefully).

> 2. As for composition, I like it better when the composition of
> validators is still a validator. If you move the validator into the
> parser, then you could have for example a rangedIntField and an
> oddIntField, but you couldn't combine them to make a
> rangedOddIntField, unless you had access to the parser and validator
> separately. Always having the validator separately gives you more
> options for combining it with other validators, for example if you
> would add validation rules to the models, then this gives you the
> option of running the model validator before or after the extra form
> validator fields added by the user. With a Monoid instance for the
> error messages, you could also combine two validators and collect both
> error messages. (Of course this doesn't really require a separate
> place for storing the validator, you could try to combine them with
> the parser as late as possible, but this again raises the issue with
> the error message types)
>
> That said, it might still be useful to have the parser in the IO or
> other monad, some field types might require it to properly parse a
> field (for example to read xml schema's)

I don't think we really disagree here. I'm not suggesting we stick
these two concepts together in the library, just that we don't need to
create a whole new mechanism to support it. If you look at the code I
pushed recently, the idea is that we have "check" and "checkM" to
apply validators to fields. We can define a Validator as "type
Validator a = a -> Either SomeMessage a".

>> > Actually, this form shows another limitation of yesod-form: If the
>> > user already exists (p /= Nothing), then the formlet is different: It
>> > just displays the username, the user cannot change it anymore after
>> > the first time. This means that when creating a profile, the new form
>> > should be displayed, while the current POST data is for the old one.
>> > Therefore, the POST data must be ignored. I believe the same problem
>> > occurs when running a multi-step (wizard style) form.
>>
>> Can you clarify the point here? I don't think I follow.
>
> I'll try to clarify the example: I have a form for a user profile,
> where the form that needs to be displayed after the profile has been
> created is different (the username can only be chosen when the user
> creates his account, it cannot be changed afterwards). A simplified
> version of this idea can be found here:
> http://hpaste.org/49821
>
> This is a two-step form: Once the form for Page1 has been submitted
> and validated, the application saves the value (in a session in this
> example, in the database in a real application) and displays the form
> for Page2. With the above code, this (line 49 and 50) goes wrong,
> because it runs the Page2 formlet while there is already POST form
> data for Page1. Therefore the default Page2 values are never
> displayed, and the submitted Page1 form data is incorrectly inserted
> into the Page2 form fields.
>
> regards,
>
> Luite

OK, I think I get it now. Basically, you're looking for a
"generateFormPost" as had been discussed a few days ago, right?

Michael

[Michael Snoyman]

For those following this thread, I put up a blog post detailing the
changes I made yesterday:
http://www.yesodweb.com/blog/2011/08/yesod-form-overhaul . The
decisions are still open for discussion, but I think that we have a
good approach for the problem now.

[Luite Stegeman]

On Aug 3, 7:40 pm, Michael Snoyman <mich...@snoyman.com> wrote:

> Look at the most recent commit. I've added a "SomeMessage" existential
> type that allows for multiple message types to be used. I'll cover it
> in more depth in a blog post (soon hopefully).

Yes, the main issue was the mixing of message types, this commit seems
to fix that, thanks! I'm going to try to convert my application to the
new api and see if I run into more issues.

> OK, I think I get it now. Basically, you're looking for a
> "generateFormPost" as had been discussed a few days ago, right?

Yes looks like it, although I had already written the code from my
first post before that discussion. Is there a better way to do this
with the latest yesod-form?

Luite

[Michael Snoyman]

I just push a commit to Git containing generateForm(Get|Post), should
be ready for testing.

Michael