Testing GAE application best practices

[Kegan]

I have been searching this group for testing (unit testing mostly)
Python GAE webapp, and the threads about testing (Pyhon) is mostly
more than a year ago. They don't contains much discussion or material
as well.

As a comparison to Django (not in GAE), the framework provides a nice
testing framework for things such as, clearing database for each test,
installing fixtures, checking for response context variables, a "test
client", etc.

Is there such "tools" for Python GAE webapp? Are you using bare
unittest with webtest? Are you using windmill?

How are you testing your Python GAE webapp now? What is the best
practices?

Thanks.

[NealWalters]

I just learned about the Gaebar - Backup and Restore utility this
morning:
http://aralbalkan.com/1784.
No time to use it yet, but they have a nice 16:42 minute video.
It talks about moving data between a staging and production
environment, so they might answer part of your question.

Neal Walters

[Kegan]

Thank you for the reply.

But this is not what I am looking for. I am looking for a unit (and
functional) testing framework that work nicely with GAE. Much like how
Django has a built-in unit and functional (test client) testing, a
good set of related assertions, etc.

~KEGan

[Federico Builes]

Kegan writes:
>
> Thank you for the reply.
>
> But this is not what I am looking for. I am looking for a unit (and
> functional) testing framework that work nicely with GAE. Much like how
> Django has a built-in unit and functional (test client) testing, a
> good set of related assertions, etc.

Take a look at nose-gae, it's kind of simple but it has worked for me so far: http://code.google.com/p/nose-gae/

--
Federico

[Michelschr]

Hello,

Thanks to all of you for your contributions!

On my side, following your suggestion, I tried nose-gae. It seems to
work with webapp...

Then I tried it with: http://code.google.com/p/rietveld/ which is a
sample application using django maintained by Guido van Rossum.
Unfortunately, the unit tests are still on the TODO list, and alone, I
was not able to make it work! (problems with 'application' and imports
like 'use_library' to choose the new django version...) Even with an
older version of Rietveld (to by pass the django version problem) I
add problems.

I can easily make the hypothesis that this way of using django
proposed by Guido van Rossum is a good direction...

But after all these problems with nose-gae, I am not sure that this is
the right direction for testing?

Reading again the discussions, I founded another way to explore:
http://code.google.com/p/gaeunit/

I don't know you... but I am beginning to cruelly miss some direction
from more advanced GAE developers.

I was thinking that professional developers start early with important
tests. On your wish-list I will add the importance for me to be able
to measure test coverage. (Personally I will never embark in a project
without a complete testing architecture...) I could not imagine how to
attract professional senior developers without some clear and
effective ways to test?

I read in the GAE documentation: "Writing unit tests that make use of
the local service implementations that are bundled with the SDK is a
natural and healthy thing to do. This chapter describes how to
accomplish this task with..."
But wait: with JUnit 3... This is extracted from:
http://code.google.com/intl/fr/appengine/docs/java/howto/unittesting.html

Yes, in Java, they have a nice How To from Google giving clear, sure,
official directions!

Thus I think that this subject of good testing is important enough for
the success of everyone's projects to ask one more time, please, could
advanced users give us directions?

Thanks in advance,

Michel

[Waldemar Kornewald]

On Jun 25, 11:19 pm, Michelschr <michels...@gmail.com> wrote:
> On my side, following your suggestion, I tried nose-gae. It seems to
> work with webapp...
>

> Then I tried it with:http://code.google.com/p/rietveld/which is a

> sample application using django maintained by Guido van Rossum.
> Unfortunately, the unit tests are still on the TODO list, and alone, I
> was not able to make it work! (problems with 'application' and imports
> like 'use_library' to choose the new django version...) Even with an
> older version of Rietveld (to by pass the django version problem) I
> add problems.

If you already mention Django, you should also mention app-engine-
patch which provides full unit test support:
http://code.google.com/p/app-engine-patch/

AFAIK, someone also got nose working on it, though I haven't had time
to test it, yet.

Bye,
Waldemar Kornewald

[Michelschr]

Still testing the test tools...
[My tests with the 'older' version of Rietveld was not significant,
this version was unfortunately based on django 1.1...]

GaeUnit seems to work for simple test on Rietveld... More later...

To make justice to App Engine Patch, I should say that I was able to
have access to the Django test framework with AEP, including the
Django test client and test coverage reports (you can see other post
about this subject...)

In the GaeUnit doc, I read: "GAEUnit for Django is an Django app. So
it can be installed like other app into any Django framework based on
Google App Engine, such as app-engine-patch and Google App Engine
Helper."

Well, if I use AEP, I don't need so much GaeUnit... So I will test now
the integration in Rietveld...

[Michelschr]

I had GaeUnit working on Rietveld (à la Guido...), the test:

import unittest
from django.test.client import Client

class Test(unittest.TestCase):

def testDjango(self):
c = Client()
response = c.get('/code')
self.failUnlessEqual(response.status_code, 200)
self.failIf(not ('SERVER' in response.content))

passed, not possible (for me) to 'from django.test import TestCase'
only from unittest.

But it is not out of the box! You should search a lot by yourself...

I will continue to investigate...

[Michelschr]

It seems that they are some conceptual incompatibilities between GAE,
GaeUnit and what I would have prefered:

1) It seems me at bit crazy to upload the tests directories to GAE if
I only use them in local...
2) Inside GAE, you do not have access to file outside what is uploaded
3) GAEUnit execute the tests inside GAE... (by way of a django app)

Well, it's more easy if you accept to upload all your test stuff to
GAE...
But I will still try a bit to find a way to test with tests outside of
GAE??

[Akume]

ok,
what about this scenario. some of my pages require a login prior to
seeing the content and i can't see to get this to work using the
django
client. any suggestions? this is what my test looks like so far.

import unittest
from django.test.client import Client

class Test(unittest.TestCase):

def setUp(self):
self.client = Client()


def testStart(self):
response = self.client.get('/app/layout')
self.failUnlessEqual(response.status_code, 301)

#attempting to log in here.
response = self.client.get('/_ah/login?continue=http%3A//localhost
%3A8081/app/layout')
self.failUnlessEqual(response.status_code, 302)

# another attempt at logging in.
#response = self.client.get('/_ah/login?email=Aliquam.nec
%40Vestibulumante.edu&action=Login&continue=http%3A%2F%2Flocalhost
%3A8081%2Fapp%2Flayout')
response = self.client.login(path='/app/layout',
username='Aliquam.nec%40Vestibulumante.edu', password='')
self.failUnlessEqual(response, True)

response = self.client.get('/app/layout')
self.failUnlessEqual(response.status_code, 200)

[Michelschr]

Hello,

Let's make a bit of walk together, here are some ideas:

1) I am testing Netbeans 6.7 python early access 2, and it looks
pretty fine for testing purpose (nice presentation of test results,
easy to go to the code, able to run test in the Netbeans project, but
above what I will transmit to GAE... + I still don't say that
everything is perfect, for example, it mix a bit the counting of
tests...)
http://download.netbeans.org/netbeans/6.7/python/ea2/

2) I founded yesterday these interesting links in the GAE discussion,
but I don't reden them yet:
http://domderrien.blogspot.com/2009/01/automatic-testing-of-gae-applications.html
http://blog.appenginefan.com/2008/06/unit-tests-for-google-app-engine-apps.html

3) I don't have investigated your problem in detail yet but:
a) I suppose you have read the django doc about this topic, normally,
they have material about that...
b) I am begining to investigate a bit more... It's not easy to make
that import of the django Client, I don't know in which context you
try this import... I will try to solve my one... With Netbeans 6.7,
when I try this import, it simply don't take the test any more in
consideration, silently... Thus, I tried it in a python console:
NbPython console
Python 2.5.4 (r254:67916, Apr 4 2009, 17:55:16)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.test.client import Client
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/usr/lib/python2.5/site-packages/django/test/__init__.py",
line 6, in <module>
from django.test.testcases import TestCase
File "/usr/lib/python2.5/site-packages/django/test/testcases.py",
line 10, in <module>
from django.db import transaction
File "/usr/lib/python2.5/site-packages/django/db/__init__.py", line
9, in <module>
if not settings.DATABASE_ENGINE:
File "/usr/lib/python2.5/site-packages/django/conf/__init__.py",
line 28, in __getattr__
self._import_settings()
File "/usr/lib/python2.5/site-packages/django/conf/__init__.py",
line 57, in _import_settings
raise ImportError("Settings cannot be imported, because
environment variable %s is undefined." % ENVIRONMENT_VARIABLE)
ImportError: Settings cannot be imported, because environment variable
DJANGO_SETTINGS_MODULE is undefined.
>>>
Like that I see better some problems, but I don't know if this is also
relevant for you?
If you run in a django environment like App Engine Patch, you don't
have this kind of problems because these settings are done fou you...

Hope it helps,

Michel

[Michelschr]

Hello,

You can simulate being logged or not by using os.environ
['USER_EMAIL']. The following tests pass on Rietveld/codereview:

from django.test.client import Client
from django.test import TestCase
import os

class Test(TestCase):

def setUp(self):
self.client = Client()

def testStart(self):

response = self.client.get('/')
self.failUnlessEqual(response.status_code, 200)
self.assertContains(response,'Code Review')

os.environ['USER_EMAIL'] = '' # simulate not logged in
response = self.client.get('/mine')
self.failUnlessEqual(response.status_code, 302)

os.environ['USER_EMAIL'] = 'te...@example.com' # simulate logged in
response = self.client.get('/mine')
self.failUnlessEqual(response.status_code, 200)
self.assertContains(response,'Issues Created by me')

os.environ['USER_EMAIL'] = '' # simulate not logged in
response = self.client.get('/mine')
self.failUnlessEqual(response.status_code, 302)

Hope it helps,

Michel

On Jun 30, 1:33 am, Akume <akume...@gmail.com> wrote:

[Akume]

i noticed that you're importing TestCase from django.test.
what are you using to run your unit test? nosegae? i tried to
implement your solution by changing the environment variable but
still no success. i still get a status_code of 301. i'm using
gaeunit as my testrunner. that may be the reason why.

On Jul 2, 8:10 am, Michelschr <michels...@gmail.com> wrote:
> Hello,
>
> You can simulate being logged or not by using os.environ
> ['USER_EMAIL']. The following tests pass on Rietveld/codereview:
>
> from django.test.client import Client
> from django.test import TestCase
> import os
>
> class Test(TestCase):
>   def setUp(self):
>     self.client = Client()
>
>   def testStart(self):
>     response = self.client.get('/')
>     self.failUnlessEqual(response.status_code, 200)
>     self.assertContains(response,'Code Review')
>
>     os.environ['USER_EMAIL'] = '' # simulate not logged in
>     response = self.client.get('/mine')
>     self.failUnlessEqual(response.status_code, 302)
>

>     os.environ['USER_EMAIL'] = 't...@example.com' # simulate logged in

[Michelschr]

Hi,

First of all, I think that Google should do the job of establishing
good practices and environments for testing GAE applications. That's
the only pragmatic solution! I don't means big integrated things but
small pieces that we can easily integrate in our environment of
choice. A lot of people like you or a newbie like me do their best to
build just basic things and I think that a large amount of
professionals will simply go for Django or Rails if they cannot find
the elementary tools.

That said, I should also mention this interesting discussion:
http://groups.google.com/group/google-appengine-python/browse_thread/thread/435b20de9b1e5cc4?hl=en

On my side, I have specific objectives and I still cannot said that my
solution is relevant, because it's specific and not yet sufficiently
tested. I will try to explain you everything, hoping I will forget
nothing...

For me, it's a poor choice to send the files with the tests to GAE.
That's why I try to exclude Gaeunit or classic django directory
structures.

On the other hand, my business is to develop applications and not
testing environments... and on the side of GAE/django integration, I
think that App Engine Patch do a pretty good job... It would be nice
if AEP can do for me (because Google don't do it) all the job of
integrating the different parts when I run tests...

Well, to conclude on my specific motivations, I like the Netbeans6.7
IDE which offer a perfect integration of tests, with coverage
information etc... Thus, let's describe each step at a time...

1) The test runner should start GAE/django, is it possible to benefit
from AEP? Here is the patch that I did to the nb_test_runner.py
(Netbeans):
(...)# Driver for running tests from NetBeans
##############################################################################

def init_gae_django(): # ++++++++++++++++++++++++++++++++++++++++++++++
+++++++++
os.environ['RUNNING_CONTEXT'] = 'test'
os.environ['SERVER_NAME'] = 'testserver'
os.environ['SERVER_PORT'] = '80'
os.environ['SERVER_SOFTWARE'] = 'Devel'
os.environ['USER_EMAIL'] = '' # set to '' for not logged in user

from common.appenginepatch import aecmd
aecmd.PROJECT_DIR = os.getcwd() + "/togae"
aecmd.setup_env() # +++++++++++++++++++++++++++++++++++++++++++++++
+++++++++

if __name__ == '__main__':
import unittest
import doctest
init_gae_django() # Here my patch and funct above
######################
(...)
You should adapt this to your environment... I try to reuse the core
business of AEP to the max, because we need a lot of patches in django
to have the things working together... I am not sure that this is a
good solution, comments are welcome! (so AEP shoud simply be in your
path, Netbeans make the cwd as the root of the project and I put all
the code in the ./togae directory, what will be send to GAE... The
test are in a ./mytest directory, just beside... AEP will try to start
its sample app and make some warnings... but after I changed the
PROJECT_DIR it find the right things, like app.yaml and
settings.py...)

2) Settings.py should be adapted a bit to run in both environment,
here are my modifications for the settings.py of Rietveld/codereview:
# Copyright 2008 Google Inc.
(...)
"""Minimal Django settings."""

import os

try: # Here my patch ###################### 5 lines
if os.environ['RUNNING_CONTEXT'] == 'test':
DATABASE_ENGINE = 'appengine'
DATABASE_SUPPORTS_TRANSACTIONS = False
DJANGO_STYLE_MODEL_KIND = False

finally: APPEND_SLASH = False

DEBUG = os.environ['SERVER_SOFTWARE'].startswith('Dev')
INSTALLED_APPS = (
'codereview',
)
(...)
The try: is the only modification that I had to make to the Rietveld/
codereview code! The little problem is that the 2 environments require
different settings, that's why I use "os.environ['RUNNING_CONTEXT'] =
'test'" in the test runner! In that case, I add the things that are
required by django/AEP like DATABASE_ENGINE = 'appengine' or
DJANGO_STYLE_MODEL_KIND = False... Don't ask me where I founded
that... I never founded any documentation about them...

3) I think, that's all... Netbeans find the tests, AEP find the
project... It seems to work... with tests as the one I published in
the previous mail...

4) Nice to have: coverage information!
With the previous settings, Netbeans6.7 will give you nice coverage
information, including on the patched/zipped django code an on your
tests... (What!? You run the patched/zipped files for tests!?!?)
I prefer to receive coverage information only about the real code that
form the application, in the /togae directory. Thus I made another
little patch to the Netbeans coverage_wrapper.py:
(...)
@atexit.register
def convert_to_nb_format():
import marshal

output_file = open(output, 'wb')

c1_dict = marshal.load(open(input, 'rb'))

for k in c1_dict.keys():
if k.startswith(os.getcwd()+"/togae/"): # Here my patch
###################### 1 line
output_file.write(k.__str__())
output_file.write('\n')
x = c1_dict.get(k)
output_file.write(x.keys().__str__())
output_file.write('\n')

output_file.close()


# Run coverage.py
execfile(coverage_py_file)
(...)

That's what I have up to now... It is still not really tested and thus
every comments will be welcome!
Hope it helps,

Michel

[Michelschr]

Hope it helps,

Michel

On Jul 2, 8:50 pm, Akume <akume...@gmail.com> wrote:

[Akume]

i couldn't agree more. google needs to make it easier for us to do
proper testing on
our app engine apps. i found another solution to represent the client
for functional
testing. using selenium RC. however i'm running into an annoying
timeout since i'm
using gaeunit as the test runner. if it's not one thing it's
another. still working
on it. i'll let you know how it goes.

-akume

On Jul 2, 2:15 pm, Michelschr <michels...@gmail.com> wrote:
> Hi,
>
> First of all, I think that Google should do the job of establishing
> good practices and environments for testing GAE applications. That's
> the only pragmatic solution! I don't means big integrated things but
> small pieces that we can easily integrate in our environment of
> choice. A lot of people like you or a newbie like me do their best to
> build just basic things and I think that a large amount of
> professionals will simply go for Django or Rails if they cannot find
> the elementary tools.
>

> That said, I should also mention this interesting discussion:http://groups.google.com/group/google-appengine-python/browse_thread/...

[Akume]

ok so i got selenium RC to work and it's probably a better solution
for functional testing of the browser than using the internal django
client. why? well because it runs in the browser via it's remote
control server so you can test content on all browsers. and it can
be integrated for use with the plugin for firefox (though they need to
fix it for firefox 3.5).

> ...
>
> read more »

[Andi Albrecht]

Hi Michael,

thanks for the explanations on how you do the testing. I assume you've
grabbed the Rietveld test cases you've mentioned from the testing
branch?

A few days ago I've tried something similar to get the Rietveld tests
running again. I've stripped the relevant parts of the App Engine
Patch (not helper) to simulate the parts of an Django environment that
were needed to run the tests (using Django's test framework as much as
possible). At least it's only the database backend and a few lines of
code to make manage.py's "test" command work.

Unfortunately I lost those sources during a workspace cleanup :(
But it should be easy to get it running again.

Regarding testing frameworks in general, it would be a nice to have if
the tests run in the App Engine environment (SDK, not production) and
not with the standard Python installation. I think nose-gae tries to
simulate that restricted environment, but it would be a much cleaner
approach if the App Engine SDK would support unittests (*and* the
Django test framework).

I'll let you know if I find some time to re-implement it.

Regards,

Andi

[Michelschr]

Hi,

Thanks for your comments!

The in-browser tests are a very important complement to tests with the
django client.

I will suggest first Windmill instead of Selenium because at the end I
suspect that we will arrive at a better integration with the Python
world.

I begin/prefer tests with the django client for several reasons:
- I have access to everything in python, like for example the models,
thus I can make a get, verify what append in the DB, clean some
things, try again, change os.environ... test again... You see what I
means?
- I can have an idea of the coverage of my tests and thus I see where
it could be important to make more efforts on tests first...

The big question is if it's possible to have coverage information when
running for example Windmill tests?? (I suspect this is not
straightforward, because you can get coverage information only in
child threats...)
Could anyone help-us to answer this question???

Michel

> ...
>
> read more »

[Andi Albrecht]

On Fri, Jul 3, 2009 at 2:18 PM, Michelschr<miche...@gmail.com> wrote:
>
> Hi,
>
> Thanks for your comments!
>
> The in-browser tests are a very important complement to tests with the
> django client.
>
> I will suggest first Windmill instead of Selenium because at the end I
> suspect that we will arrive at a better integration with the Python
> world.
>
> I begin/prefer tests with the django client for several reasons:
> - I have access to everything in python, like for example the models,
> thus I can make a get, verify what append in the DB, clean some
> things, try again, change os.environ... test again... You see what I
> means?
> - I can have an idea of the coverage of my tests and thus I see where
> it could be important to make more efforts on tests first...

I'd prefer tests using the Django client too. At least as long I won't
have to test any kind of browser behavior, in that case windmill or
webdriver (has Python bindings too) seem to be suitable tools. Besides
the advantages you've mentioned, UnitTests/Django tests are easy to
run on the command line :)

>
> The big question is if it's possible to have coverage information when
> running for example Windmill tests?? (I suspect this is not
> straightforward, because you can get coverage information only in
> child threats...)
> Could anyone help-us to answer this question???

I don't know. I think it should be possible to run the SDK server in
some kind of "coverage mode", but I'm not sure about that.

[Michelschr]

Hi,

First I should also thanks you for your work on http://code.google.com/p/django-gae2django/
That is where I discovered Rietveld!

> thanks for the explanations on how you do the testing. I assume you've
> grabbed the Rietveld test cases you've mentioned from the testing
> branch?

I had missed this branch before you mentioned it...

For the rest, I mostly agree with you...

That said, for me, the more important point is maintenance. Who will
maintain the code we use? That is why I have tried to NOT touch the
code of App Engine Patch. If they come with a new version I just need
to extract it in my path, provided that they continue to support the
aecmd.PROJECT_DIR and aecmd.setup_env() API.

Thus, it will be very interesting if you can from outside limit

> the relevant parts of the App Engine
> Patch (not helper) to simulate the parts of an Django environment that
> were needed to run the tests (using Django's test framework as much as
> possible). At least it's only the database backend and a few lines of
> code to make manage.py's "test" command work.

You comments are welcome?

Michel

PS: Here I correct a bug in the correction to the setting, we need an
except:
try:

if os.environ['RUNNING_CONTEXT'] == 'test':

DATABASE_ENGINE = 'appengine' # 'appengine' pour les tests

DATABASE_SUPPORTS_TRANSACTIONS = False
DJANGO_STYLE_MODEL_KIND = False

except:
pass

APPEND_SLASH = False

[Michelschr]

Hi,

> thanks for the explanations on how you do the testing. I assume you've
> grabbed the Rietveld test cases you've mentioned from the testing
> branch?

I got 1 file: tests.py from this branch and the tests passed easily!

Nice to have, I removed the os.environ['... as you asked!

import codereview.models as models
import codereview.engine as engine

# Set some essential variables.
# TODO(guido): Shouldn't InstallAppengineHelperForDjango() do this?
#os.environ['SERVER_NAME'] = 'localhost'
#os.environ['SERVER_PORT'] = '80'
#os.environ['USER_EMAIL'] = ''

These basic tests cover 31% of 5751 lines. Particularly:
9,9% of engine
16% of intra_region_diff
22% of views
(...)

For views, I can point quickly that these portions of code are not
tested:
def render(self, name, value, attrs=None):
def set_branch_choices(self, base=None):
def get_base(self):
def clean_base(self):
def clean(self):
def clean_nickname(self):
(...)

I suggest you make a quick try of Netbeans6.7 with my different
patches to see that in colour... :-)

Have a hot weekend!

Michel

[Andi Albrecht]

Sorry for the delayed reply, I'm not sure which patches you're talking
about... Does this only work with Netbeans?

Andi

[Michelschr]

Hello,

Yes, my recommended solution only works with Netbeans6.7. (I never
seen something comparable with that level of integration and test
coverage information...) But with a little effort you can probably
adapt it to another environment...

I detailed my motivations and the patches on my mail of July 2.
Essentially it was:
1) A patch in the Netbeans test runner to invoke App Engine Patch with
that:

os.environ['RUNNING_CONTEXT'] = 'test'
os.environ['SERVER_NAME'] = 'testserver'
os.environ['SERVER_PORT'] = '80'
os.environ['SERVER_SOFTWARE'] = 'Devel'
os.environ['USER_EMAIL'] = '' # set to '' for not logged in user

from common.appenginepatch import aecmd

aecmd.PROJECT_DIR = os.getcwd() + "/togae" # This should be your
settings.py dir
aecmd.setup_env()

2) You should add this in your settings.py:
import os
if 'RUNNING_CONTEXT' in os.environ.keys():

DATABASE_ENGINE = 'appengine' # 'appengine' pour les tests

DATABASE_SUPPORTS_TRANSACTIONS = False
DJANGO_STYLE_MODEL_KIND = False

To make it quicker, if you hesitate to install the Netbeans IDE, I can
send you a picture of the (in colour :-) coverage information... But I
need your e-mail to do that...

Bye,

Michel

On Jul 9, 10:09 am, Andi Albrecht <albrecht.a...@googlemail.com>
wrote: