Test Isolation

[Serguei]

Hallo,

If using the testbed library to run tests locally, I initialize the stubs and get the same empty datastore and memcache for each test. Are there any recommendations on how to setUp/tearDown with aeta to avoid side effects caused by parallel or previous tests?

The Java tool cloudcover seems to switch the namespace before executing test or track all datastore writes to remove test data later.

Thanks.

Serguei

[Robert Schuppenies]

Hi Serguei.

Could you explain your test setup and the problems you are having a bit more? I'm not quite sure I understand your question.

What I can say for now though is that if you use testbed for testing datastore-related operations, and you are using Testbed.activate() and Testbed.deactivate() in your setUp() and tearDown() methods, your tests should be isolated and no test data from one test should be seen by another. In particular, deactivate() will restore the state that the stubs where in when you called activate().

aeta does not touch stubs or any other App Engine related parts, it is merely a test runner that will locate tests and then run them.

cheers,

robert

Serguei

--
 
 

[Serguei]

Hallo Robert,

yes, the question is how to prevent one test from seeing data of another test. E.g. the following test can be executed on appengine but the test_user_put_2 will fail because it finds two user entries including one created in test_user_put_1:

class UserModelTest(unittest.TestCase):

    def test_user_put_1(self):

        models.User(email="1...@example.com").put()

        users = models.User.query(models.User.email=="1...@example.com").fetch(2)

        self.assertEqual(1, len(users))

    def test_user_put_2(self):

        models.User(email="1...@example.com").put()

        users = models.User.query(models.User.email=="1...@example.com").fetch(2)

        self.assertEqual(1, len(users))

Your answer really helps, adding these functions resolves the issue:

    def setUp(self):

        self.testbed = testbed.Testbed()

        self.testbed.activate()

        self.testbed.init_datastore_v3_stub()

        self.testbed.init_memcache_stub()

    def tearDown(self):

        self.testbed.deactivate()

But the usage of testbed within aeta testrunner is still not quite clear to me.I thought with aete we can and should call *real* services and not using stubs. I also seen that using some other stubs e.g. mail_stub does not work within appengine/aeta.

Serguei.

[Robert Schuppenies]

On Wed, Nov 28, 2012 at 12:07 AM, Serguei <serguei...@gmail.com> wrote:

Hallo Robert,

yes, the question is how to prevent one test from seeing data of another test. E.g. the following test can be executed on appengine but the test_user_put_2 will fail because it finds two user entries including one created in test_user_put_1:

class UserModelTest(unittest.TestCase):

    def test_user_put_1(self):

        models.User(email="1...@example.com").put()

        users = models.User.query(models.User.email=="1...@example.com").fetch(2)

        self.assertEqual(1, len(users))

    def test_user_put_2(self):

        models.User(email="1...@example.com").put()

        users = models.User.query(models.User.email=="1...@example.com").fetch(2)

        self.assertEqual(1, len(users))

Your answer really helps, adding these functions resolves the issue:

    def setUp(self):

        self.testbed = testbed.Testbed()

        self.testbed.activate()

        self.testbed.init_datastore_v3_stub()

        self.testbed.init_memcache_stub()

    def tearDown(self):

        self.testbed.deactivate()

But the usage of testbed within aeta testrunner is still not quite clear to me.I thought with aete we can and should call *real* services and not using stubs. I also seen that using some other stubs e.g. mail_stub does not work within appengine/aeta.

Serguei.

Ah, now I get it, thanks for your explanation. 

Here is my take on it: When you create your tests you have to decide what you want to test. If you want to test units of code in isolation aka unit tests, you should rely on libraries such as ext.testbed that help you set up and tear down data before every test. When you want to test the integration of different units, the isolation cannot be assumed when running tests in parallel as the tests rely on the same infrastructure. 

In the example you have given above your two tests will each put an entity into the datastore and then each assume that that there is only one entity in that store. This assumption is only valid if you have an isolation in place. But for integration tests you don't have this isolation. Instead, each test could query whether the exact entity created exists in the datastore (checking the count of the returned entities is really just a cheap and inaccurate way of checking whether the entity was stored).

You also mentioned that cloudcover does provide some isolation by setting up namespaces prior to a test run. aeta doesn't do anything like that and I am not sure whether I think it should. The reason for me is simply that it gives you some sense of isolation, but in reality your tests aren't isolated. What's your take?

So back to your original question "Are there any recommendations on how to setUp/tearDown with aeta to avoid side effects caused by parallel or previous tests?": 

My recommendation is to write your tests with the awareness that there may be side effects from other tests. When you clean up after your test, only clean up what your test has changed, when you test the effect of your test, be specific.

cheers,

robert

 

Am Dienstag, 27. November 2012 07:15:13 UTC+1 schrieb Robert Schuppenies:

Hi Serguei.

Could you explain your test setup and the problems you are having a bit more? I'm not quite sure I understand your question.

What I can say for now though is that if you use testbed for testing datastore-related operations, and you are using Testbed.activate() and Testbed.deactivate() in your setUp() and tearDown() methods, your tests should be isolated and no test data from one test should be seen by another. In particular, deactivate() will restore the state that the stubs where in when you called activate().

aeta does not touch stubs or any other App Engine related parts, it is merely a test runner that will locate tests and then run them.

cheers,

robert

On Mon, Nov 26, 2012 at 11:44 AM, Serguei <serguei...@gmail.com> wrote:

Hallo,

If using the testbed library to run tests locally, I initialize the stubs and get the same empty datastore and memcache for each test. Are there any recommendations on how to setUp/tearDown with aeta to avoid side effects caused by parallel or previous tests?

The Java tool cloudcover seems to switch the namespace before executing test or track all datastore writes to remove test data later.

Thanks.

Serguei

--
 
 

--
 
 

[Serguei Strigo]

Also integration tests can benefit from isolation. It can give an easy and consistent way to create test data, cleanup test data and enable parallel execution of tests.

Let say there is an application with user self-service. Integration tests for self-service and other parts of the application need a registered user. So each integration test starts with register_user(email="te...@exapmle.com", password=...). Email must be unique. With no isolation email address is randomly generated and the tests become not so readable. The test logic should care about other tests sharing datastore, memcache and queues. Cleanup of test data is difficult to maintain: register_user() can leave a task in a queue and several memcache/datastore entries.

I agree that tests cannot be completely isolated if running on appengine. But setting up a namespace would probably be enough. And the test data will not mess up with the application data. One more advantage: I have already tests using testbed and thus written for isolated environment.

Serguei.

--