Re: [tg-trunk] ToscaWidgets: options validatation

[Alberto Valverde]

On Apr 5, 2007, at 9:45 AM, Max Ischenko wrote:

> Hello,
>
> I wonder why this test fails:
>
> def test_options():
> w = MultipleSelectField(options="Group1 Group2 Group3".split())
> eq_(['Group3', 'Group1'], w.validate(['Group3', 'Group1']))
> print w.validate(['GroupX'])
> assert 0, "formencode.Invalid not raised"
>
> I was assuming that the select widget will only accept options that
> are in the options list. No?

This is a tricky case. For that test to pass a OneOf valiadator is
needed, however, that would break the callable support for the
"options" parameter since the validator is fixed at construction
time. I can think of 2 ways to solve it:

1) avoid using a callable for "options" and pass a OneOf validator
explicitly
2) Use a hidden field to store view state and make validators aware
of it to validate accordingly.

2 is probably the best option in the long-run but the hardest to
design/implement. It would also allow disabling fields dynamically
which is now not possible for required fields since validation would
fail.

Ideas?

Alberto

P.S: Cross-posting to toscawidgets-discuss

[Alberto Valverde]

On Apr 5, 2007, at 1:50 PM, Max Ischenko wrote:

Alberto,

On 4/5/07, Alberto Valverde <alb...@toscat.net> wrote:

> def test_options():
>     w = MultipleSelectField(options="Group1 Group2 Group3".split())
>     eq_(['Group3', 'Group1'], w.validate(['Group3', 'Group1']))
>     print w.validate(['GroupX'])
>     assert 0, "formencode.Invalid not raised"
>
> I was assuming that the select widget will only accept options that
> are in the options list. No?

This is a tricky case. For that test to pass a OneOf valiadator is
needed, however, that would break the callable support for the
"options" parameter since the validator is fixed at construction
time. I can think of 2 ways to solve it:

1) avoid using a callable for "options" and pass a OneOf validator
explicitly

That's hardly an option (sorry for pun) -- callable are invaluable in certain situations.

I know. I was refering to avoiding callabes only when you absolutely need a OneOf validator (hence know needed options before-hand when instantiating the widget).

Another solution would be to create the form instance dynamically in every request, however, this would "break" TG's way of validating through the validate decorator since it needs the form instance at class or module scope (or do some dirty hacks with the callable you can pass it to provide the form instance). This could be solved by validating inside the controller with an equivalent form (twForms doesn't check on form identity like TG widgets do so different but equivalent form instances can work)

2) Use a hidden field to store view state and make validators aware
of it to validate accordingly.

What's a "view state"?

By design, widgets are static: you can create one as a module global and keep using it for the program's lifetime.

OTOH, to validate with OneOf you need to know which options were actually used to render the form last time and this information has to be kept somewhere. You can pickle and sign this into a bundle and put as a hidden field on a form. Is it what you means?

Exactly

 
It may be not very elegant but I don't see a better way. You can also save options temporarily in the the widget itself and passing some bucket-id as a hidden. This can also work as long as widgets does it in a thread safe way.

I've heard ASP does something similar. I don't thiink that saving options temporarily in the widget is a viable solution because they have to survive across requests. How could we handle that?

Alberto

P.S: lets move this thread to toscawidgets-discuss please :)

[Max Ischenko]

On 4/5/07, Alberto Valverde <alb...@toscat.net> wrote:

 
It may be not very elegant but I don't see a better way. You can also save options temporarily in the the widget itself and passing some bucket-id as a hidden. This can also work as long as widgets does it in a thread safe way.

I've heard ASP does something similar. I don't thiink that saving options temporarily in the widget is a viable solution because they have to survive across requests. How could we handle that?

It is certainly doable and not too complicated.

To render a widget you need to convert all options to strings, right?

So you just concat all the strings together, take md5 hexdigest out of it and use it as a key to thread-safe hash to store the actual values.

Then you need to pass this digest down to the form so that it's get submitted and the options can be retrieved from the hashmap.

Max.

[Alberto Valverde]

On Apr 5, 2007, at 2:29 PM, Max Ischenko wrote:

On 4/5/07, Alberto Valverde <alb...@toscat.net> wrote:

 
It may be not very elegant but I don't see a better way. You can also save options temporarily in the the widget itself and passing some bucket-id as a hidden. This can also work as long as widgets does it in a thread safe way.

I've heard ASP does something similar. I don't thiink that saving options temporarily in the widget is a viable solution because they have to survive across requests. How could we handle that?

It is certainly doable and not too complicated.

Saving this "view-state" is certainly not hard, what I'm not sure how hard it would be is to generalize this enough and integrate with formencode...

There's a RequestLocalDescriptor in toscawidgets.util which I use in InputWidget.error_at_request and InputWidget.value_at_request to store the value and error for the current request. Currently these descriptors affect all widgets (that is, if widget A stores something in one of them, widget B will see the same value when accessing the attribute) but a subclass is possible that provided widget-request-local storage.

This can be used to add another similar attribute, maybe, InputWidget.view_state which could be a dict where each widget that needs saving this state can place a key there which the form will finally picke and sign (in a similar manner SecureFormMixin does to prevent CSRF attacks) before rendering. The form would also take care of unsigning and de-pickling and passing the saved state to the Schema.

However, I haven't thought of a way yet for the validators to make good use of this info in a general enough way...

 

To render a widget you need to convert all options to strings, right?

yep

So you just concat all the strings together, take md5 hexdigest out of it and use it as a key to thread-safe hash to store the actual values.

a simple md5 digest wouldn't be very secure I guess. Keep in mind this info will affect how validation will occur so it's esssential for it to be secure  to prevent attacks. SecureFormMixin uses hmac plus a secret which I think it's safer.

Then you need to pass this digest down to the form so that it's get submitted and the options can be retrieved from the hashmap.

RequestLocalDescriptor (or a variant of it) can provide the mechanism for inter-widget communication in a thread-safe, request-local way.

Alberto

[Jorge Godoy]

"Max Ischenko" <isch...@gmail.com> writes:

> It is certainly doable and not too complicated.
>
> To render a widget you need to convert all options to strings, right?
>
> So you just concat all the strings together, take md5 hexdigest out of it
> and use it as a key to thread-safe hash to store the actual values.
>
> Then you need to pass this digest down to the form so that it's get
> submitted and the options can be retrieved from the hashmap.

Where would you store this hashmap? What about applications without database
servers? Or heavy loaded database servers and very complex forms?

--
Jorge Godoy <jgo...@gmail.com>

[Jorge Godoy]

Alberto Valverde <alb...@toscat.net> writes:

> Saving this "view-state" is certainly not hard, what I'm not sure how

Do you think so? I'm thinking on a distributed server scenario, where one
request can go to server1 and the submit to server5... I'm also thinking on
how much memory would one need in a one server scenario with 300 or 400 users
using the system and having some complex forms to fill (usually a digital
version of a one or two paper pages forms...).

> hard it would be is to generalize this enough and integrate with
> formencode...

This is always a problem on some designs to me as well :-) Custom widgets
aren't always easy...

> This can be used to add another similar attribute, maybe,
> InputWidget.view_state which could be a dict where each widget that
> needs saving this state can place a key there which the form will
> finally picke and sign (in a similar manner SecureFormMixin does to
> prevent CSRF attacks) before rendering. The form would also take care
> of unsigning and de-pickling and passing the saved state to the Schema.
>
> However, I haven't thought of a way yet for the validators to make
> good use of this info in a general enough way...

Maybe you'll need some glue code to decode that and pass it to validators in a
more usual way...

> a simple md5 digest wouldn't be very secure I guess. Keep in mind
> this info will affect how validation will occur so it's esssential
> for it to be secure to prevent attacks. SecureFormMixin uses hmac
> plus a secret which I think it's safer.

And is also easy to compute.

--
Jorge Godoy <jgo...@gmail.com>

[Max Ischenko]

On 4/5/07, Jorge Godoy <jgo...@gmail.com> wrote:

> To render a widget you need to convert all options to strings, right?
>
> So you just concat all the strings together, take md5 hexdigest out of it
> and use it as a key to thread-safe hash to store the actual values.
>
> Then you need to pass this digest down to the form so that it's get
> submitted and the options can be retrieved from the hashmap.

Where would you store this hashmap?  What about applications without database
servers?  Or heavy loaded database servers and very complex forms?

In a widget itself (as a class-level variable). Where else could it be? ;)

Max.

[Max Ischenko]

On 4/5/07, Alberto Valverde <alb...@toscat.net> wrote:

So you just concat all the strings together, take md5 hexdigest out of it and use it as a key to thread-safe hash to store the actual values.

a simple md5 digest wouldn't be very secure I guess. Keep in mind this info will affect how validation will occur so it's esssential for it to be secure  to prevent attacks. SecureFormMixin uses hmac plus a secret which I think it's safer.

Whatever, the concept is the same.

Then you need to pass this digest down to the form so that it's get submitted and the options can be retrieved from the hashmap.

RequestLocalDescriptor (or a variant of it) can provide the mechanism for inter-widget communication in a thread-safe, request-local way.

I don't know anything about request-local stuff but if there is a way to 'attach' this to request than great - use it and forget about the need for  thread-safe global storage.

But there are two requests involved here: first to generate HTML form and another to actually send it (and validate). Does RequestLocalDescriptor handle this?

Max.

[Alberto Valverde]

On Apr 5, 2007, at 3:15 PM, Max Ischenko wrote:

On 4/5/07, Alberto Valverde <alb...@toscat.net> wrote:

So you just concat all the strings together, take md5 hexdigest out of it and use it as a key to thread-safe hash to store the actual values.

a simple md5 digest wouldn't be very secure I guess. Keep in mind this info will affect how validation will occur so it's esssential for it to be secure  to prevent attacks. SecureFormMixin uses hmac plus a secret which I think it's safer.

Whatever, the concept is the same.

Yes

Then you need to pass this digest down to the form so that it's get submitted and the options can be retrieved from the hashmap.

RequestLocalDescriptor (or a variant of it) can provide the mechanism for inter-widget communication in a thread-safe, request-local way.

I don't know anything about request-local stuff but if there is a way to 'attach' this to request than great - use it and forget about the need for  thread-safe global storage.

RequestLocalDecsriptor actually proxies to an attribute at framework.request_storage so yes,  there's no need to re-implement this :)

But there are two requests involved here: first to generate HTML form and another to actually send it (and validate). Does RequestLocalDescriptor handle this?

I'll try to explain it better (this should also answer Jorge's questions):

request A - form renders itself

1) widgets which need to save state for request B do it somwhere at widget.view_state (request local storage for all widgets)

2) form picks up this dict, serializes it, signs it and places it at a hidden field

notice that this state is *not* saved server-server side except a shared secret which could be saved in the session or in a config file. This is framework dependent so a callable should be passed to provide this secret.

request B - form validates

1) form (more precisely, the schema) picks hidden field, decrypts it, un-pickles it and passes it to all child validators. Should also pop the key from the input dict in order to be transparent to client code.

2) child validators act accordingly: if a field is disabled, ingore key missing, if options A,B,C were passed, only allow A,B,C, etc...

(B2 is actually the part I said I don't have very clear how-to design/implement, the rest is quite easy...)

Key-points:

1) RequestLocalDescriptor is only used to save state for inter-widget communication in *one* request. Anyhing stored here will not be saved among requests.

2) state between requests (render and validate) is passed in the form itself inside a hidden field, no need for server-side storage of any sort unless a different secret is required for each session.

SecureFormMixin implements a *very* similar pattern which can be used for inspiration. 

Alberto