duplicated selenium test fails

[Ulrich Laube]

Hi!

It boils down to this:

I have a selenium test in a django project against the admin page. All green.
If I duplicate the test class via copy and paste + renaming, then one is green and one is red, despite containing the same code. I expected
both to be green.

I have removed the tests from the project just to make sure that there is no inteference. The following steps should allow anyone to reproduce
the behavior in a new project/app.

For the record: Python 3.4.1 Django 1.8.5 Selenium 2.48.0


> django-admin startproject temp
> cd temp
> manage startapp blog

adjust settings.py to include 'blog' in INSTALLED_APPS

add a single class to models.py

class Post(models.Model):
text = models.TextField()

register the model with admin site in admin.py add:

from blog.models import Post
admin.site.register(Post)

> manage makemigrations
> manage migrate
> manage createsuperuser
admin
admin@localhost
admin
> manage runserver

Check that you can login via the admin page and add posts.
Now add the following test to tests.py:

from django.contrib.auth.models import User
from django.contrib.staticfiles.testing import StaticLiveServerTestCase

from blog.models import Post

from selenium import webdriver

class AdminTest(StaticLiveServerTestCase):
def login_to_django_admin(self):
self.credentials = {
'username': 'admin',
'email': 'admin@localhost',
'password': 'admin'
}
User.objects.create_superuser(**self.credentials)
# go to the admin site
self.browser.get(self.live_server_url + '/admin/')
# we are not logged in, it redirects us to the login page
self.assertEquals(self.browser.current_url, self.live_server_url + '/admin/login/?next=/admin/')

# login in
inputbox = self.browser.find_element_by_id('id_username')
inputbox.send_keys('admin')
inputbox = self.browser.find_element_by_id('id_password')
inputbox.send_keys('admin')
submit = 'input[type="submit"]'
submit = self.browser.find_element_by_css_selector(submit)
submit.click()
self.browser.find_element_by_class_name('app-blog')

def test_insert_Post_via_django_admin(self):
self.browser = webdriver.Firefox()
self.login_to_django_admin()

# now that we are logged in, look at the Post model add page to get a csrftoken
self.browser.get(self.live_server_url + '/admin/blog/post/add/')
inputbox = self.browser.find_element_by_id('id_text')
inputbox.send_keys('barfoo')
submit = 'input[type="submit"][name="_save"]'
submit = self.browser.find_element_by_css_selector(submit)
submit.click()

# we should see it on the other side via python db access
saved_beiträge = Post.objects.all()
self.assertEquals(saved_beiträge.count(), 1)
self.assertEquals(saved_beiträge[0].text, 'barfoo')

# try to look at it
self.browser.get(self.live_server_url + '/admin/blog/post/1/')

# modify it
inputbox = self.browser.find_element_by_id('id_text')
inputbox.send_keys('text')
submit = 'input[type="submit"][name="_save"]'
submit = self.browser.find_element_by_css_selector(submit)
submit.click()

# we should see the change at the other end
saved_posts = Post.objects.all()
self.assertEquals(saved_posts.count(), 1)
self.assertEquals(saved_posts[0].text, 'barfootext')

Now run it via:

> manage test

Selenium opens Firefox and walks through the admin page as it should. Test green.
Now duplicate the whole class and rename it and run the tests again.

> manage test

One is green the other is not. In fact Selenium opens a second Firefox instance as it should. But this time

self.browser.get(self.live_server_url + '/admin/blog/post/1/')

results in a 404 not found.

Now delete the whole class that is passing and run the test again.

> manage test

Now the failing one is green as well.

I can't wrap my head around this. What am I missing?
Is it a Django issue or a Selenium issue?
Is it a known problem?

Thanks

Uli

[Carl Meyer]

Hi Ulrich,

On 10/19/2015 03:03 AM, Ulrich Laube wrote:
> It boils down to this:
>
> I have a selenium test in a django project against the admin page. All
> green.
> If I duplicate the test class via copy and paste + renaming, then one is
> green and one is red, despite containing the same code. I expected both
> to be green.

[snip]

This line is the problem. It hardcodes the ID 1 in the URL. The
database-isolation method used by Django tests does not restore ID
sequences to a fixed value, so you cannot ever rely on fixed ID values.
In your case, when you only have the one test, it always creates the
first Post in the database, with ID 1; when you copy-paste the test, the
second test creates a Post with ID 2 instead, breaking this line. You
should use the actual ID value (you query the object just above, so you
have it here) instead of hardcoding `1`.

> # modify it
> inputbox = self.browser.find_element_by_id('id_text')
> inputbox.send_keys('text')
> submit = 'input[type="submit"][name="_save"]'
> submit = self.browser.find_element_by_css_selector(submit)
> submit.click()
>
> # we should see the change at the other end
> saved_posts = Post.objects.all()
> self.assertEquals(saved_posts.count(), 1)
> self.assertEquals(saved_posts[0].text, 'barfootext')
>
> Now run it via:
>
>> manage test
>
> Selenium opens Firefox and walks through the admin page as it should.
> Test green.
> Now duplicate the whole class and rename it and run the tests again.
>
>> manage test
>
> One is green the other is not. In fact Selenium opens a second Firefox
> instance as it should. But this time
>
> self.browser.get(self.live_server_url + '/admin/blog/post/1/')
>
> results in a 404 not found.

Right, because in this test there is no Post with ID 1, just one with ID 2.

> Now delete the whole class that is passing and run the test again.
>
>> manage test
>
> Now the failing one is green as well.

Yes, because now you're back down to just one test, so it is again
creating a Post with ID 1.

> I can't wrap my head around this. What am I missing?
> Is it a Django issue or a Selenium issue?
> Is it a known problem?

Not really a problem - you just need to be aware that you can't rely on
consistent database IDs in tests.

Carl

[Ulrich Laube]

Thanks Carl!

Am 21.10.2015 um 00:18 schrieb Carl Meyer:
> On 10/19/2015 03:03 AM, Ulrich Laube wrote:
>> It boils down to this:

>> [...]

>> Is it a known problem?
>
> Not really a problem - you just need to be aware that you can't rely on
> consistent database IDs in tests.

So there is no test isolation where I assumed it to be.
My assumption was, that each

class younameit(TestCase)

gets its own fresh database to work with, so that all

def test_foobar(self):

within, would work against the same DB.
Thanks for pointing that out to me.

Uli

[Carl Meyer]

On 10/21/2015 02:55 AM, Ulrich Laube wrote:
> Am 21.10.2015 um 00:18 schrieb Carl Meyer:
>> On 10/19/2015 03:03 AM, Ulrich Laube wrote:
>>> It boils down to this:
>>> [...]
>>> Is it a known problem?
>>
>> Not really a problem - you just need to be aware that you can't rely on
>> consistent database IDs in tests.
>
> So there is no test isolation where I assumed it to be.

There is test isolation, it just doesn't extend to sequences. Tables are
cleared of all data in between tests, so your second test still sees
only a single Post in the database, it's just a Post with ID 2 instead
of ID 1.

> My assumption was, that each
>
> class younameit(TestCase)
>
> gets its own fresh database to work with,

A completely fresh database would be the ideal situation from an
isolation standpoint, but it is prohibitively slow to actually create a
new test database for each test or test case.

The actual behavior is pretty close to "a fresh database for each test",
though -- ID numbering is really the only exception you're likely to
commonly see.

Really since auto ID numbers are chosen by the database, not your code,
it's better practice for your tests not to rely on them regardless.

> so that all
>
> def test_foobar(self):
>
> within, would work against the same DB.

No, the isolation is per test method, not per test class. Every single
test method effectively runs in a clean database (modulo the fact that
sequences aren't reset).

The details of how this isolation is implemented vary by the type of
test. For normal TestCases, each test method runs within a transaction,
and the test runner rolls back the transaction at the end of each test
so none of its changes survive. This is the fastest method, but it means
that the code under test cannot start or commit/rollback transactions.

For tests which may need to test transaction behavior, we have
TransactionTestCase (which LiveServerTestCase inherits from); in this
case the isolation is implemented by truncating all tables after each
test. This is slower, but doesn't interfere with transactions in the
code under test.

Carl