Validate a GET request using SQLFORM.factory and request vars

[Alfonso Serra]

Hi everybody.

Im trying to validate a simple GET request using SQLFORM.factory. but cant get it to pass. I would like just to validate those variables but i dont know what im missing. On the docs theres not much info about forms and the get method but form.accepts(request.vars, ...).

The url would be:

http://myapp/get_rates?rooms=5&agency=1

The controller:

def get_rates():
    # http://myapp/get_rates?rooms=5&agency=1
   
    print "request:",  request.vars
    # <Storage {'agency': '1', 'rooms': '5'}>
   
    form = SQLFORM.factory(
        Field("rooms", "integer", required=True)
        , Field("agency", "integer", required=True)
        , _method="GET"
    )
    #remove the auto id field
    form.fields.pop(0)
   
   
    print "accepts:", form.accepts(request.vars, session=None, dbio=False)
    # False
    print "validate:", form.validate(request_vars=request.vars, session=None)
    # False
    print "fvars:", form.vars
    # {}

    if form.accepted:
        print "All good"
    elif form.errors:
        print form.errors
   
    print "Done"
    return 0.0

And the console output:

request: <Storage {'agency': '1', 'rooms': '5'}>
accepts: False
validate: False
fvars: <Storage {}>
Done

Version 2.12.3-stable+timestamp.2015.08.19.00.18.03

Why is the form not validating?


Many thanks in advance.

[Anthony]

SQLFORM adds a hidden "_formname" field, which it expects to be present for validation. For SQLFORM.factory, the default value of that field is "no_table/create", though .process(), .validate(), and .accepts() take a "formname" argument that allows you to specify any name. So, your URL should include _formname in the query string:

http://myapp/get_rates?rooms=5&agency=1&_formname=no_table/create

Alternatively, you could add _formname to request.get_vars before processing the form:

request.get_vars._formname = 'no_table/create'

    form = SQLFORM.factory(
        Field("rooms", "integer", required=True)

Note, setting required=True will be irrelevant here, as it is used by the DAL, not by the form validation system. If you want validation to enforce a required value, use requires=IS_NOT_EMPTY().

Anthony

[Alfonso Serra]

Thanks Anthony. It worked but now it always validates no matter what.

I have passed

# request.vars
<Storage {'rooms': 'asd'}>
request.vars._formname = "get_rates"

print "accepts:", form.accepts(request.vars, session=None, dbio=False, formname="get_rates")
print "validate:", form.validate(request_vars=request.vars, session=None, formname="get_rates")

And the form is accepted on both cases.

Is not, at least, the variable type compared?

Would it make sense to modify the Field class to add initial validators when it is instantiated, or have in mind the field type and requirement?
This way you wont have to repeat the declarations as for "integer", IS_INT, "date", IS_DATE and so on. required=True could also serve as a validator.

I may be wrong but havent seen this behaviour on a post submit. Does this only happen on get submits?

Thanks for the help.

[Anthony]

On Wednesday, April 25, 2018 at 5:56:08 PM UTC-4, Alfonso Serra wrote:

Thanks Anthony. It worked but now it always validates no matter what.

I have passed

# request.vars
<Storage {'rooms': 'asd'}>
request.vars._formname = "get_rates"

print "accepts:", form.accepts(request.vars, session=None, dbio=False, formname="get_rates")
print "validate:", form.validate(request_vars=request.vars, session=None, formname="get_rates")

And the form is accepted on both cases.

What validators did you define?

 

Is not, at least, the variable type compared?

Not without validators.

 

Would it make sense to modify the Field class to add initial validators when it is instantiated, or have in mind the field type and requirement?

When using db.define_table() with a regular DAL instance, there are some default validators added: http://web2py.com/books/default/chapter/29/06/the-database-abstraction-layer#Field-types. However, this does not happen when simply using SQLFORM.factory.

 

I may be wrong but havent seen this behaviour on a post submit. Does this only happen on get submits?

Same with POST or GET if using SQLFORM.factory.

Note, internally, SQLFORM.factory uses DAL(None) to generate a dummy DAL instance and table definition. For some reason, with DAL(None), the default validators (noted above) are not applied. Feel free to submit a Github issue about that, or maybe even make a pull request to change that behavior (I haven't looked at the code to see why it works that way now).

Anthony

[Alfonso Serra]

Thanks Anthony.

I  think ill submit a patch, Im doing some debugging and i have noted a few things:
Version 2.16.1-stable+timestamp.2017.11.14.05.54.25

gluon/html.py line 2217

        kwargs['request_vars'] = kwargs.get(
            'request_vars', current.request.post_vars)

the validate method assumes post_vars are used. Using request.vars instead would save us having to declare the request_vars parameter. Its actually good we are able to override but the form should be aware what variables to test against by default.

gluon/sqlhtml.py line 1904

            elif field.type == 'integer':
                if value is not None:
                    fields[fieldname] = safe_int(value)

sqlform.accepts actually does some kind of type checking but it looks that does not affect the acceptation. The lines were executed using .factory so maybe they could be used to check the variable types.

Im gonna try to fix these issues and submit a patch or create an issue request.
Thanks again.

[Anthony]

On Wednesday, April 25, 2018 at 11:01:23 PM UTC-4, Alfonso Serra wrote:

Thanks Anthony.

I  think ill submit a patch, Im doing some debugging and i have noted a few things:
Version 2.16.1-stable+timestamp.2017.11.14.05.54.25

gluon/html.py line 2217

        kwargs['request_vars'] = kwargs.get(
            'request_vars', current.request.post_vars)

the validate method assumes post_vars are used. Using request.vars instead would save us having to declare the request_vars parameter.

No, we should leave it as is. First, request.vars is a mix of post and get vars, which may be used simultaneously for different purposes (e.g., a page with a POST form on it may be at a URL that includes a query string). A single form, on the other hand, is either POST or GET, so the form processing code should by default use only the POST or the GET vars. Second, even if we wanted to change it, that would break backward compatibility, which we cannot do.

 

Its actually good we are able to override but the form should be aware what variables to test against by default.

gluon/sqlhtml.py line 1904

            elif field.type == 'integer':
                if value is not None:
                    fields[fieldname] = safe_int(value)

sqlform.accepts actually does some kind of type checking but it looks that does not affect the acceptation. The lines were executed using .factory so maybe they could be used to check the variable types.

I think validation should be left to the validators. In any case, if you propose any changes, be sure they do not break backward compatibility.

Anthony

[Anthony]

Note, SQLFORM was not intended for validating query strings. For that, you might instead want to use a library geared more toward that purpose, such as Cerberus, schema, or Voluptuous.

Anthony

[Alfonso Serra]

Hi Anthony.

Didnt think of the mixed case but it makes sense. getvars on a post could be used to indicate an app state at a given time for example.

Please allow me some considerations:

- The only case request.vars could be a problem would be when there are variables with the same name on both methods. On this case i dont know which one will be overwritten, the get var most likely.
- The form discards extra variable names not declared for processing, so including get vars should not affect post processing, besides when post and querystring vars collides.
- If the form was not meant to process get vars, implementing the functionality would not break compatibility. This would be easy since request.get_vars and post_vars should be processed/validated the same way when the form uses either methods.

I have made and test the changes and everything is working fine so far.
- The form would do type checking. It does not make sense to pass rooms = "asd" when it expects an integer. or declare field type "integer" and IS_INT_IN_RANGE and so on for every Field on our app.
- The form will detect the variables to test against when _method="get" is passed as parameter. It would be easy to use request.get_vars/post_vars instead request.vars to prevent the issues you mention.
- The form wont auto generate an id field when the adapter is NullAdapter (the one used when Fields does not belong to any table, DAL(None)).

Didnt know about those libraries (will have a look) but i think web2py has all the tools needed to perform these operations in an elegant and easy way. With a little bit of refinement it could be even easier or avoid repetition which is what im trying. This would also simplify our models and form declarations.

Thank you very much for your unvaluable support and the links.
King Regards.

[Anthony]

Please allow me some considerations:

- The only case request.vars could be a problem would be when there are variables with the same name on both methods. On this case i dont know which one will be overwritten, the get var most likely.

If post and get vars each contain variables with the same name, request.vars ends up with a list of the two values.

 

- The form discards extra variable names not declared for processing, so including get vars should not affect post processing, besides when post and querystring vars collides.

Or maybe if the query string happens to include a variable with the same name as a field left blank in the form (in particular, check boxes, which are not included in the post data when left unchecked).

 

- If the form was not meant to process get vars, implementing the functionality would not break compatibility. This would be easy since request.get_vars and post_vars should be processed/validated the same way when the form uses either methods.

Actually, it would probably make sense to check the "_method" attribute of the form and use request.post_vars if it is "POST" and request.get_vars otherwise. I would not use request.vars -- too many potential edge cases to worry about, and no benefit over simply using post_vars vs. get_vars conditional on _method.

 

I have made and test the changes and everything is working fine so far.
- The form would do type checking. It does not make sense to pass rooms = "asd" when it expects an integer. or declare field type "integer" and IS_INT_IN_RANGE and so on for every Field on our app.

I'm not sure about this. We already have default validators in place when using SQLFORM with a DAL table. This breaks down when using SQLFORM.factory because DAL(None) does not generate the default validators -- but it would probably be a better approach to change that behavior so that DAL(None) does get the default validators. That approach would have more general utility beyond helping with SQLFORM.factory, and it would avoid having to add complexity/duplicate validation code to the FORM class. In other words, let's use a mechanism we already have rather than add more code to the framework.

 

- The form will detect the variables to test against when _method="get" is passed as parameter. It would be easy to use request.get_vars/post_vars instead request.vars to prevent the issues you mention.

OK, yes, same conclusion I reached above.

 

- The form wont auto generate an id field when the adapter is NullAdapter (the one used when Fields does not belong to any table, DAL(None)).

What's the problem with that? Have you considered any possible backward compatibility issues with that change?

 

Anthony

[Alfonso Serra]

I'm not sure about this. We already have default validators in place when using SQLFORM with a DAL table. This breaks down when using SQLFORM.factory because DAL(None) does not generate the default validators -- but it would probably be a better approach to change that behavior so that DAL(None) does get the default validators. That approach would have more general utility beyond helping with SQLFORM.factory, and it would avoid having to add complexity/duplicate validation code to the FORM class. In other words, let's use a mechanism we already have rather than add more code to the framework.

 

I havent seen where this happens yet, ill have a look. Agree that it will better to reuse the code but, please correct me if im wrong, validators will only trigger if the form is rendered into a view as they lie into the form's components? Will it work if the form is html hardcoded? Im just using web2py server side and creating the html using the form data (fields, attrs, form.errors).

About the id field, i thought it didnt make sense to add it as it does not serve any purpose when the form has a NullAdapter. Would be easier to create custom html without having to pop the field. Removing it would break compatibility only if anyone has found any use for it on this particular case, which is posible but unlikely.

Best Regards.

 

[Anthony]

On Thursday, April 26, 2018 at 4:23:02 PM UTC-4, Alfonso Serra wrote:

I'm not sure about this. We already have default validators in place when using SQLFORM with a DAL table. This breaks down when using SQLFORM.factory because DAL(None) does not generate the default validators -- but it would probably be a better approach to change that behavior so that DAL(None) does get the default validators. That approach would have more general utility beyond helping with SQLFORM.factory, and it would avoid having to add complexity/duplicate validation code to the FORM class. In other words, let's use a mechanism we already have rather than add more code to the framework.

 

I havent seen where this happens yet, ill have a look. Agree that it will better to reuse the code but, please correct me if im wrong, validators will only trigger if the form is rendered into a view as they lie into the form's components?

No, validation happens within the controller -- it only depends on the form object in the controller and the submitted request vars.

We really don't want to try to replicate validation logic directly in the form methods -- you would not only need to validate the data but also provide a way to specify error messages as with the validators. It would be a big unnecessary mess.

 

About the id field, i thought it didnt make sense to add it as it does not serve any purpose when the form has a NullAdapter. Would be easier to create custom html without having to pop the field.

Why do you have to pop 'id' from form.fields? Your code runs fine without doing that.

Anthony

[Alfonso Serra]

The auto id field, may not be a problem, i removed it for convenience for a couple of reason:
- iterate form.fields in the view to create the html. in this case was not needed but also the special treatment as it should lie on hidden inputs.
- the type checking on this case was validating the id which was not part of the factory declaration, not accepting the form. Ofc this could be solved with some code refinement and leave everything how it is.
 

We really don't want to try to replicate validation logic directly in the form methods -- you would not only need to validate the data but also provide a way to specify error messages as with the validators. It would be a big unnecessary mess.

Hmm i dont know if you would like this but i have also thought about it. Validator's error messages could be done by web2py translator, kinda like IS_DATE already does for date formats. If these messages are automated they could be easily changed at the translation file instead of having to declare them everytime on all the apps. Would also shorten the Fields declarations this way.

I have been recently involved in a project to translate a custom complex app and been thinking on ways to improve web2py translator.

Improvable things im planning to change on my project would be.

- Load the translations within a module to stay in memory when the server starts. Not sure if this is already happening or are being read from disk on demnand.
- Plurals should not be translated by themselves without their contexts as they may change the grammatical form of sentences depending on the language.

There is a mechanism that i like which is to store pluralizable sentences as dictionaries, for example:

"15 items found."

the translation declaration could be:

{"%(count) items found": {
      "zero": "No items found"
     , "one": "one item found"
     , "many": "some items found"
     , "other": "%(count)s items found"
   }
}

then T("%(count) items found", count=1)

will detect this is a pluralizable string since the translation is a dictionary expecting the count parameter.
Then will choose the right pluralized translation.
The "many" keyword could be also configurable.
There would be no need to have 2 separate translation files (not a problem if they are), neither isolate plural words from phrases to be translated. Also take care of plurals genetive saxon "The children's ballons" and other syntaxis or grammars on other languages.
Translators would have context to translate the sentences.

It goes without saying this would break compatibility but it would make my app much more easy to handle.

Please let me know any thoughts.

Best Regards.

[Anthony]

On Friday, April 27, 2018 at 3:36:41 AM UTC-4, Alfonso Serra wrote:

The auto id field, may not be a problem, i removed it for convenience for a couple of reason:
- iterate form.fields in the view to create the html. in this case was not needed but also the special treatment as it should lie on hidden inputs.

You can instead iterate over form.custom.widget. It includes 'id' as a key as well, but its value is an empty string, so easy to conditionally exclude by checking that.

 

- the type checking on this case was validating the id which was not part of the factory declaration, not accepting the form. Ofc this could be solved with some code refinement and leave everything how it is.

Not sure what workflow led to that, but I haven't seen any such issues with either SQLFORM or SQLFORM.factory.

Note, even with a standard SQLFORM based on a DAL table, "id" is included in form.fields even though it is not an input in the actual form nor processed upon form submission. I'm not sure if "id" serves any purpose in either case, but if we make a change, it should apply in all cases, not just SQLFORM.factory.

 

 

We really don't want to try to replicate validation logic directly in the form methods -- you would not only need to validate the data but also provide a way to specify error messages as with the validators. It would be a big unnecessary mess.

Hmm i dont know if you would like this but i have also thought about it. Validator's error messages could be done by web2py translator, kinda like IS_DATE already does for date formats. If these messages are automated they could be easily changed at the translation file instead of having to declare them everytime on all the apps. Would also shorten the Fields declarations this way.

That can already be done based on the default error messages associated with each validator (all validator error messages are passed to T() before being returned from the validator), but that is not a sufficient replacement for the current system, which allows custom messages to be associated with particular instances of particular validators dynamically.

 

I have been recently involved in a project to translate a custom complex app and been thinking on ways to improve web2py translator.

Probably worth a separate post in the web2py-developers Google group.

Anthony