Designing tests for a complex scientific site

[Mike Orr]

I'm porting a Pylons 1 application to Pyramid and I want to write some automated tests. It has only a few screens but it has a large number of input variables that are shared between screens, and it makes numerous calls to a C library (using ctypes) that may return error messages, which are propagated to the user in a delayed manner similar to flash messages.

Currently I have a pair of rudimentrary Twill scripts to test the new site and compare it to the old site. But the Twill shell has a lot of limitaitons so I'm thinking of switching to Twill's Python API or unittest. So I'm wondering if anybody has any suggestions between these or ideas for how to design the tests in Pyramid.

I'm leaning more toward functional tests first because the Pylons code was already written by someone else, and the client is most interested in whether the site behaves the same as the old site and returns the same results, as opposed to what each individual function does. I should fill in those low-level tests later but I think I need some more "practical" tests first.

So, a basic use case looks like this:
(1) go to the home page, which loads a canned "weather scenario" into the session (a bunch of numeric and string variables).
(2) go to the chemical selection page and choose a chemical. This loads a bunch of chemical properties into the session.
(3) choose one of four dispersion models.
(4) each model has 1-3 screens to fill out. Some fields are shown/required only if other fields are filled in.
(5) submitting each screen does standard form validation (I think it's all Javascript rather than FormEncode?) and loads its fields into the session. But it also calls C functions, and the data may be rejected due to some obscure detail in the C model that can't be caught by ordinary Javascript or Python validation. E.g., you say the chemical is a liquid with such a volume, but that chemical can't be a liquid at the weather's temperature. The C function spits back one or more "stop messages" (errors) and/or "show messages" warnings, which go into the session and then it redirects back to the form page. The form page redisplays the form, but the stop and show messages are extracted from the session to the template and appear in a Javascript "lightbox" (a kind of modal dialog, with the dimmed form fields showing around it).
(6) If there were no errors, it continues to a result page which calls some more C functions and displays a bunch of results.

The client likes the current user interface and lightboxes and doesn't want any changes there. And with a hundred variables to keep track of, I'm not inclined to change how they flow in and out of the session and templates.

 So my three choices are Twill scripts, Twill Python code, or unittests. Twill scripts are becoming annoying limiting, especially for recognizing server-site exceptions and stop messages. So I'm thinking about Twiil Python, which mimics a browser programmatically. But I also like Pyramid's innovations in unit testing. But I'm not sure if unittest can accommodate the kind of high-level use cases I need with multiple POSTs in a test, session persistence, and recognizing stop messages. E.g., I may be able to recognize stop messages by looking in the session, maybe? And I'd also kind of like to have tests I can also run against the old version, and I'm not even considering writing Pylons unittests for it, which would be a different API and is on its way out anyway.

So, anyone have any ideas?

--
Mike Orr <slugg...@gmail.com>

[Marius Gedminas]

On Thu, Dec 06, 2012 at 04:21:45PM -0800, Mike Orr wrote:
> I'm porting a Pylons 1 application to Pyramid and I want to write some
> automated tests. It has only a few screens but it has a large number of
> input variables that are shared between screens, and it makes numerous
> calls to a C library (using ctypes) that may return error messages, which
> are propagated to the user in a delayed manner similar to flash messages.
>
> Currently I have a pair of rudimentrary Twill scripts to test the new site
> and compare it to the old site. But the Twill shell has a lot of
> limitaitons so I'm thinking of switching to Twill's Python API or unittest.
> So I'm wondering if anybody has any suggestions between these or ideas for
> how to design the tests in Pyramid.
>
> I'm leaning more toward functional tests first because the Pylons code was
> already written by someone else, and the client is most interested in
> whether the site behaves the same as the old site and returns the same
> results, as opposed to what each individual function does. I should fill in
> those low-level tests later but I think I need some more "practical" tests
> first.

...

> So, anyone have any ideas?

I like zope.testbrowser for functional tests. AFAIU it's Twill-like
(I've never used Twill), but you have the full power of Python instead
of a restricted domain language.

Marius Gedminas
--
The BeOS takes the best features from the major operating systems. It's got
the power and flexibility of Unix, the interface and ease of use of the MacOS,
and Minesweeper from Windows.
-- Tyler Riti

[Jesaja Everling]

I think you should have a look at Selenium.
http://seleniumhq.org/
It remote-controls real web-browsers, thus you can use it to test
things like the JavaScript "lightboxes" that are displayed in your
application.
There is a Firefox extension called Selenium IDE, which allows you to
record what you do in the browser, and which is a great starting point
for your tests. You can export the recorded tests as Python code. With
just a little work you will have a test that opens your browser,
interacts with it just like you would, and is able to verify that CSS
classes or strings exist in the HTML. I think Selenium is a great tool
for functional tests of websites, and it should allow you to make sure
that the ported application works like the Pylons application did.

Best Regards,

Jesaja Everling

[Jasper]

+1 for Jesaja's answer

[Mike Orr]

On Thu, Dec 6, 2012 at 4:21 PM, Mike Orr <slugg...@gmail.com> wrote:
> (5) submitting each screen does standard form validation (I think it's all
> Javascript rather than FormEncode?) and loads its fields into the session.
> But it also calls C functions, and the data may be rejected due to some
> obscure detail in the C model that can't be caught by ordinary Javascript or
> Python validation. E.g., you say the chemical is a liquid with such a
> volume, but that chemical can't be a liquid at the weather's temperature.
> The C function spits back one or more "stop messages" (errors) and/or "show
> messages" warnings, which go into the session and then it redirects back to
> the form page. The form page redisplays the form, but the stop and show
> messages are extracted from the session to the template and appear in a
> Javascript "lightbox" (a kind of modal dialog, with the dimmed form fields
> showing around it).
> (6) If there were no errors, it continues to a result page which calls some
> more C functions and displays a bunch of results.

So I think what I want is for the redisplay result to have a
non-Javascript way of indicating that it's displaying a message. I
could do that by adding "X-Stop-Message" headers to the response. Or
in Pylons there was this thing called ``request.environ[
"paste.testing_variables"] = {}``, which would be set only in testing
mode and where the controller could put extra data for the test. Does
Pyramid have anything equivalent to that?

--
Mike Orr <slugg...@gmail.com>

[Robert Forkel]

hi mike,
i went the custom http header way as you already mentioned. like you i
started doing this for testability, but now consider it as making the
json api of my app more usable outside of the ajax context it was
invented for, i.e. non-browser clients do also profit from the added
headers.
regards
robert

> --
> You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
> To post to this group, send email to pylons-...@googlegroups.com.
> To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.
>