Django 1.6 and migrating to the new test runner: note on user experience

[Ivan Kharlamov]

Hi!

Maybe I'm a careless kind of guy, but when I test out new versions of
Django, reading all of the release notes is rarely the first thing that
I do... my bad. :) My first action is usually firing up the test suites
and looking at the number of failed tests and deprecation warnings.

As we all know, Django-1.6 is special in this regard, since it doesn't
run all the tests that the previous version would've discovered. Since
it doesn't raise any deprecation warnings, it becomes really easy to
assume that everything is fine and that the project is Django-1.6 ready,
when in fact it is completely broken.

Perhaps, Django-1.6 test runner could raise a warning and inform the
user that the number of tests that would've run on the previous
versions, was omitted.

My wild guess is that the lack of a deprecation warning is just asking
for some wildly entertaining user stories.

What do you think?

Best regards,
Ivan

[Carl Meyer]

Hi Ivan,

Thanks for reporting on your experience. I see (and was already aware
of) the problem, but all the solutions I see are ugly in one way or
another. Some options:

1) The status quo. If people run the new "manage.py check" command,
they'll get a warning directing them to the release notes. But given
that "manage.py check" is new, its unlikely anyone would run it unless
they've already read the release notes, so this isn't worth much.

2) The new test runner issues some kind of warning unconditionally on
1.6 (this seems to be what you're proposing). That would certainly catch
the people we need to catch, but it seems like an awfully big hammer.
Once you know about the new test runner and have adjusted your tests,
wouldn't it be irritating to keep seeing that warning every single time?
And people creating new 1.6 projects would be irritated too, since the
warning is useless to them.

3) A first-run warning that can be turned off (and presumably would be
off by default in new projects). The obvious mechanism for turning it
off would be something in settings. That either means introducing a new
setting specifically for this warning, or warning anybody who doesn't
have an explicit TEST_RUNNER override (meaning all 1.6 projects would be
required to needlessly specify the new test runner in their settings in
order to silence the warning.)

4) A variant of (3) where we introduce a more general setting that we
can use for similar situations in the future. Something like a
SILENCE_UPGRADE_WARNINGS setting that's a list or set of well-known
strings, or something: SILENCE_UPGRADE_WARNINGS = ['discovery-runner']
or some such.

5) The new test runner, every time its run, goes out and looks for tests
in the locations where the new runner won't find them (models.py files,
app tests/__init__.py files, tests.py in INSTALLED_APPS that are not
within the CWD, doctests) and warns you if it finds any. This would be a
fair chunk of code that would be easy to get wrong, and I think it's
really too late to add it to 1.6. It would also slow down test-running
unnecessarily on every run.

So, like I said, a bunch of bad options. Which is the least bad?
Personally I'd be open to considering (1), (3), or (4). Interested in
others' thoughts.

Carl

[Shai Berger]

I think in view of the GSoC work and the check command, (4) as phrased above
would be mostly duplication of effort. The 'check' command, IIRC, already
includes mechanisms to silence specific warnings.

I think my preferred solution is a mix of your (3) and (5) -- a search for
tests that can be turned off. I think (though I'm not too familiar with that
area of the code) that the coding here is not too hard -- essentially, run the
old code as well as the new code for building the test-suite, and diff the
results.

Another idea is a variation on your (4) -- add a setting to specify the Django
version expected by the project; so if a project says, for example,
DJANGO_USED="1.5", we emit suitable upgrade warnings.

This may have a problem -- it may create unwarranted expectations of backward-
compatibility support; people may think it tells newer Django versions to use
the behavior of older versions, rather than warn on breaking changes. But I do
find it has a certain appeal -- it's a lot like including the protocol version
in messages.

My 2 cents,

Shai.

[Ivan Kharlamov]

Hi, Carl,

Thank you for considering a fix!

Personally, I'd prefer a solution proposed by Shai, which is (5).

As another alternative to creating a new variable in settings.py, there
is an option, albeit also a little bit ugly, to add a command line
parameter, so that the tests could be run with './manage.py test
application --no-upgrade-warning'.

With best wishes,
Ivan

P.S.
I've recently found another minor bug in the new test runner:
https://code.djangoproject.com/ticket/21206#ticket

[Carl Meyer]

Hi Shai,

I think (5) -- searching for tests using the old and new runners and
comparing results -- is more attractive in theory than it is useful in
practice. Because the new test runner does not run tests in "external"
apps by default, it will always run a different set of tests than the
old runner (except in the rare case of a project that uses no contrib
apps or pip-installed apps with tests). So if we compare the set of
collected tests, we will basically always get different results, and
always issue the warning. So in practice it reduces to almost the same
thing as an always-issued warning, but with more code and moving parts
to achieve the same result. I don't think it's worth it.

Which brings us to the same problem we had before - the best way to turn
off a warning you would otherwise always get.

> Another idea is a variation on your (4) -- add a setting to specify the Django
> version expected by the project; so if a project says, for example,
> DJANGO_USED="1.5", we emit suitable upgrade warnings.
>
> This may have a problem -- it may create unwarranted expectations of backward-
> compatibility support; people may think it tells newer Django versions to use
> the behavior of older versions, rather than warn on breaking changes. But I do
> find it has a certain appeal -- it's a lot like including the protocol version
> in messages.

I agree that this idea would allow us to be a lot smarter about certain
upgrade warnings.

Carl

[Carl Meyer]

On 10/01/2013 06:36 AM, Ivan Kharlamov wrote:
> On 09/27/2013 09:01 PM, Carl Meyer wrote:
>> On 09/27/2013 03:07 AM, Ivan Kharlamov wrote:
> Thank you for considering a fix!
>
> Personally, I'd prefer a solution proposed by Shai, which is (5).

I commented in my reply to Shai why I don't think (5) is really any
better than a warning you always get unless you turn it off.

> As another alternative to creating a new variable in settings.py, there
> is an option, albeit also a little bit ugly, to add a command line
> parameter, so that the tests could be run with './manage.py test
> application --no-upgrade-warning'.

I think this is worse than a setting. A setting at least you can set
once and not have to think about it again, rather than have to arrange
to issue it every single time you run tests.

Carl

[Christopher Medrela]

I was working at system check framework [1] during this holiday. I've done

some changes that will solve all problems considered in this thread.

Maybe I'm a careless kind of guy, but when I test out new versions of
Django, reading all of the release notes is rarely the first thing that
I do... my bad. :) My first action is usually firing up the test suites
and looking at the number of failed tests and deprecation warnings.

 

As we all know, Django-1.6 is special in this regard, since it doesn't
run all the tests that the previous version would've discovered. Since
it doesn't raise any deprecation warnings, it becomes really easy to
assume that everything is fine and that the project is Django-1.6 ready,
when in fact it is completely broken.

The first change is that when you run tests, all checks are performed, so you

don't have to know that there is new check command in order to be notified

about the test runner issue.

The second problem is disabling the warning. The warning is issued when

TEST_RUNNER is not defined in settings.py file. If you set it, there is no

warning even if the new value is DiscoverRunner (the default one). So the

warning can be silenced by setting TEST_RUNNER. And there is new

SILENCED_SYSTEM_CHECKS setting so there is another way to disable the warning.

I guess that my branch will be merged with 1.7 version and we cannot do much

with 1.6 release.

[1] https://groups.google.com/forum/#!topic/django-developers/_sde4eoNRyQ