Property-based testing of Web2py - an experiment

[Kiran Subbaraman]

Hello,
I had come across Hypothesis for Property based testing (https://hypothesis.readthedocs.org/en/master/index.html), and felt it is a good candidate to write much improved tests for web2py. And also, because am lazy, more exhaustive tests, with minimal effort.
My initial attempt: https://gist.github.com/kirsn/abda630d8752fd8734ce

• The initial attempt was to just match all the input arguments for method, and describe the property values for those arguments.
  
• Also, the `assert` could definitely improve - check for a definite 'pass' and 'fail'

Your thoughts?

Kiran

[Leonel Câmara]

Your tests have bugs :P Still, it's an interesting idea to complement the regular tests so we verify some properties, for instance validators should probably never throw exceptions.

[Kiran Subbaraman]

yikes... they do? please tell me what needs to be corrected.
Also, this sort of test brought out aspects like - if min / max is specified, then should they be zero / positive, and should min <= max, and if our validators need to check these first before proceeding with the rest of the validation logic, etc...
Will explore Hypothesis a bit more to see its fit into web2py. There are some examples with Django, and with state-machines ....

[Leonel Câmara]

An example would be this line:

assert (result == (Decimal(value), None)) or (result == (Decimal(value), 'Enter from %s to %s characters' % (minsize, maxsize)))

It should be

assert (result == (Decimal(value), None)) or (result == (value, 'Enter from %s to %s characters' % (minsize, maxsize)))

[Kiran Subbaraman]

web2py-devs,
First cut of Property based testing of validators: https://github.com/kirsn/web2py/blob/prop_test_validators/gluon/tests/test_validators_property.py
The results are attached as an XML file. Please take a look, and let me know your thoughts.

Observations:

• The tests basically look for points where the code breaks - therefore the arguments' values passed to a validator may not have any semantic correctness. For example, if the validator expects an ip address in the form of a string, the property-test passes in any unicode string or None. If the expectation is a min/max int value, then the values passed are integers (-ve, 0, +ve). Question: Should the validators handle such cases, where the inputs are spurious? And if so, should they catch all these exceptions, and not further raise exceptions?
• Validators need to handle unicode strings, consistently.
  
• IS_LOWER(), and IS_UPPER() are formatters, and not validators.
• CLEANUP() is a utility method, not a validator
• How to property-test IS_EMPTY_OR(), IS_STRONG() ... I need to take a look at this again.
• Should the validators change the `value` being validated? For example, the IS_NOT_EMPTY()(u' ') returns a ('', None). Basically it modifies the value. I understand it has utility, but was a little confused with the do's/don'ts of validators.

[Leonel Câmara]

> Should the validators change the `value` being validated? For example, the IS_NOT_EMPTY()(u' ') returns a ('', None). Basically it modifies the value. I understand it has utility, but was a little confused with the do's/don'ts of validators.

Yes, sometimes they should. The reason for this is that the only thing the user inputs is text, I think it would be rare to have an instance where you want a value validated and you don't want it converted into the correct type, so it's just pratical.

[Kiran Subbaraman]

'user text' -> [validator] --> 'system type/value'  ... so basically the validator does a 'check', and 'transform'.
So, if a new validator is being created, then it needs to have a mandatory 'check' step, followed by an optional 'transform' step. This will ensure that we don't have validators like IS_UPPER() - in its current form, it is a TO_UPPER / transformation only functionality.

In any case, am just gently pushing for some definitions of what these core components will do, and won't, so we know how to identify one when we see it in the wild.

Having said that, I have liberally used the validators without any second-thoughts about design purity. So there :p

________________________________________
Kiran Subbaraman
http://subbaraman.wordpress.com/about/

On Tue, 29-03-2016 2:11 PM, Leonel Câmara wrote:

> Should the validators change the `value` being validated? For example, the IS_NOT_EMPTY()(u' ') returns a ('', None). Basically it modifies the value. I understand it has utility, but was a little confused with the do's/don'ts of validators.

Yes, sometimes they should. The reason for this is that the only thing the user inputs is text, I think it would be rare to have an instance where you want a value validated and you don't want it converted into the correct type, so it's just pratical.

--
-- mail from:GoogleGroups "web2py-developers" mailing list
make speech: web2py-d...@googlegroups.com
unsubscribe: web2py-develop...@googlegroups.com
details : http://groups.google.com/group/web2py-developers
the project: http://code.google.com/p/web2py/
official : http://www.web2py.com/
---
You received this message because you are subscribed to the Google Groups "web2py-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Richard]

Some really good points Kiran...

I would say that proper documentation is the key and proper requirements... That why Gherkin Language and Behave are so interresting because you write your requirements...

Richard