Testing in web2py

[morningovermidnight]

I've been reading earlier posts on unittests and using doctests in
web2py. I have tried and run some successful doctests in web2py.
However, it seems that the doctest is testing the value that my
function returns, as it should, but is there a way to write a test to
check an intermediate value in my function, say the value of a session
variable that is stored but not returned to the view?

Also, just curious, what are some of the solutions people here in the
group are using for testing and for debugging web2py applications?

[Timothy Farrell]

An excellent question. Testing in web2py is somewhat of a misnomer
since you use web2py to create web applications but you can only truly
test the models or controllers, not the product of all three MVC
components. However, there isn't really a better way unless Massimo
wants to include another external libary (and therefore dependency).

What I use for webpage testing is Selenium. Selenium is a
javascript-based scripting/testing engine. However, it has Python
modules that you can use to write and invoke tests (which it translates
to Javascript and runs.) It even has a cool IDE plugin for Firefox
which makes test-making pretty easy. Take a look.
http://www.openqa.org/selenium/

-tim

[Timothy Farrell]

Massimo,

I implied in my previous post, that I don't use web2py's built-in testing.  This brings me to a question...what is the "test" directory for under the application root?  Since the admin interface only runs doctests, what would be a good way of running a unittest that I place in /applications/init/tests/ ?

This sort of setup, combined with selenium-rc, would be the ultimate in web app testing (well almost, but it's really close).

-tim

[morningovermidnight]

Thanks for pointing me to selenium (http://selenium.openqa.org/). It
looks great. I've also seen some other automated web testing soultions
such as twill (http://twill.idyll.org/) and TestGen4Web (http://
developer.spikesource.com/wiki/index.php?title=Projects:TestGen4Web).
If you don't mind, could you give me a scenario of how you test your
web application with selenium ( selenium rc in particular) and how
testing with selenium has helped your web dev?

I ask because honestly i am still on the fence about whether or not to
implement something such as selenium, or twill, or TestGen4Web. They
all seem to run tests well, but -- and correct me if i'm wrong -- when
thinking about the MVC architecture, it seems all of these only test
the V, the view. You know, testing to see that a page displayed, that
a link is present, or that a page contains some text. And while
testing the view is of course important, testing the MC part of
things, the application logic is just as important -- or at least a
testing solution is not complete if not all components, the M, the V,
and the C can be tested. Such a solution may be a combination of
several different products...but it is something definitely worth
looking into. Again, maybe I am only seeing Selenium on the
surface...is there a way for Selenium to test application logic (e.g.
setting the status of the website to "down for maintenance" then
verifying that no users can access the site via login or otherwise) ?

>  tfarrell.vcf
> < 1KViewDownload

[yarko]

there are some other projects that use "nose", and from what little
I looked at selenium, it looks like you can generate a test script (at
least a sample) by recording actions. Selenium is open, written in
Java, and looks to have a Python interface (to drive tests).

Anyone look at this more?

On Oct 18, 11:15 am, morningovermidnight <thotzdatbreat...@gmail.com>
wrote:

[mdipierro]

You can write two types of tests: doctests and tests/*.py tests. You
can runs the latters from the shell (try -h). I will look into
selenium.

>  tfarrell.vcf
> < 1KViewDownload

[Timbo]

Since the start of this thread, I've only just looked into using
Selenium RC and how to integrate that with web2py. I haven't come up
with a "good" solution yet, but I'll keep you posted.

How does selenium compare with other products? I haven't used
anything else, but a cursory look at the two mentioned, shows Selenium
as being more flexible/powerful than twill and more cross-browser than
TestGen4Web.

How I use selenium now is I've downloaded selenium core and put it in
my static files (it's just a few webpages and lots of javascript). I
have a test script that runs through all possible combinations of how
logging into my website should behave.

How has selenium helped my web dev? Easy question, it speeds up
testing by magnitudes and makes testing less boring and more
thorough. Also it gives you a CYA factor for when someone says
something doesn't work right. All you have to do is show them the
test. If you haven't tested a particular aspect of something, it will
become apparent rather quickly. The solution is to add that
particular regression to your test script and in no time, you'll have
an extensive battery of tests for your webapp.

Web app testers such as Selenium do not only test the View...however,
it is through the view that they do their testing. In a true MVC
model, the Controller is the only thing that can be tested since it is
the only place where logic should exist in the process. However, most
developers have views that show things differently based on different
input from the controller. So we have logic in our views as well. It
goes without saying that certain behaviors in the controller produce
certain manifestations in the corresponding view. For example, in my
login application, if the user enters an incorrect password, it
returns with a flash-message based on how it is incorrect (illegal
characters, no password supplied, etc.). This flash is testable (via
Selenium or others) and it is testing both the view logic and the
controller logic. Hence, we can use Selenium to test both the
controller and the view. Models don't possess logic in themselves
(unless you want to test constraints but that's silly).

Selenium core can only run javascript and manipulate a browser. For
that reason, it cannot set application state without you providing a
web-facing front-end to facilitate that (if you do, be wary of
security concerns). Selenium RC, can run as a normal Python unittest
and could therefore manipulate application state when run from the
web2py shell as Massimo pointed out. I'm going to look more into this
on Monday.

@yarko:
The Selenium RC server is the only portion written in Java and that is
only necessary for tests written in languages other than HTML
Selenese. You could make tests via the IDE and run them purely with
Selenium core without having to mess with Java at all. Actually, I'd
like to find a way to cleanly start/stop Selenium RC Server from
web2py for good integrated testing. I'll let you know. But if you
want to get started with it, download Selenium Core and the IDE and
get going.

=)

On Oct 18, 11:15 am, morningovermidnight <thotzdatbreat...@gmail.com>
wrote:

[yarko]

Thanks for all this, Timbo!

Looking forward to hearing more on your experience, thoughts with
this.

Yarko

[Timothy Farrell]

Here's some more for ya.

I got SeleniumRC working in a reasonable fashion.  Here's how you do it:
1) Build your test with Selenium IDE, Save it as a Python Unit test (the IDE will write the source code for you)
2) Download selenium rc server.  Put the 'selenium-server.jar' and the attached seleniumtest.py in site-packages (for now).
2.5) Don't forget to install a recent version of Java and the selenium Python module ('easy-install selenium').
3) In your test from step one, import the seleniumtest module and change:
    unittest.main()
to:
    seleniumtest.runInSeleniumRC(unittest.main)()

Caveats for now:
- I can't get Firefox or Chrome to work, the tests will only work in IE and Opera for me.  I haven't tried Safari.
- After running a test, my little module doesn't appropriately kill (at least on windows) the seleniumRC server, so you'll have to kill that manually. (I would appreciate any tips on this.)

I put my test in /web2py/applications/init/tests/Test_Login_Process.py and I can run it just fine from there.  This test only accesses the webpage side of things, so it does not need to be run from a web2py shell.  However, you could very easily add controller-level testing to it and run it from the web2py shell.

-tim

[mdipierro]

Yes, you can using doctests

def index():
"""
This is a sample test:
>>> tmp=index()
>>> session.x
3
"""
session.x=3
return dict(message=T('Hello World'))


On Oct 17, 3:33 am, morningovermidnight <thotzdatbreat...@gmail.com>
wrote:

[mdipierro]

I only gave it a quick look but I like selenium, yet I do not believe
it belongs to web2py. I think it is mainly client side so it belongs
to an app.

Massimo

> tfarrell.vcf
> < 1KViewDownload

[Timothy Farrell]

No, I wasn't implying that selenium should be part of web2py.  I agree with you that it is more of an app.  But it would be nice to be able to invoke normal python unit tests from the admin interface.

-tim