Re: How to test patch

[Russell Keith-Magee]

On Sun, Oct 28, 2012 at 7:33 AM, Jan Bednařík <jan.be...@gmail.com> wrote:

Hi all,

I'm using Django for more than four years and last week I started contributing.

In docs about contributing I didn't find how detailed should be my testing while I'm writing or reviewing patch? Is enough to run tests only for patched module? Or should I run full test suite for each patch? Which combinations of Python versions and database engines are mandatory?

So far I was running Python 2.7 & SQLite3 for development/review testing of module. Python 3.2 & SQLite3, Python 2.7 & PostgreSQL for patched module and in the end Python 2.7 & SQLite3 full test suite (with selenium, etc.). Is this workflow ok?

Hi Jan,

I've noticed your contributions on Trac this week. Thanks very much for taking the leap from user to contributor. Django wouldn't be where it was today without the efforts of people like yourself.

So - how much testing is required? Enough to ensure that whatever it is you're submitting won't break when we apply the patch.

Obviously, more is better, but you can apply a little discretion. If you're tweaking something in the template language, there's not much point running all the tests for the ORM. However, as a final check before you commit, it certainly wouldn't hurt to do one final run of the full test suite, just be sure. 

As for databases? Again, it depends a little bit on what you're testing. If you're modifying the template language, you can assume the ORM will do the right thing; just running the tests under SQLite will probably suffice. However, if your modifying the behaviour of queries, you need to run against every backend possible (that means SQLite, MySQL with MyISAM and InnoDB, PostgreSQL and Oracle). Don't worry *too* much about multiple versions of each database; there aren't *that* many inconsistencies between versions of databases, and if there is *that* will be the subject of the bug report (in which case, running the tests with different versions of the same database *will* be required).

As for Python versions? At a minimum, you should be running the tests on a Python 2 version and a Python 3 version. It's generally a good idea to use the lowest supported version you can - so, Python 2.6 and 3.2 - because that's the most likely to reveal problems with accidentally using new features in the language. 

Ultimately, the goal is to make the best use of the development community's time. The core team are the bottleneck in the development process, because we're the only people who can commit code. That means that we need the community to make sure we waste as little of our time as possible. If we receive a patch that fails because it breaks the test suite, then that slows down the rate at which we can commit patches. So - the more testing you do to make sure that the core team doesn't get a broken patch, the better. Of course, we're not going to blame anyone if a test breaks in a weird or unexpected way. Mistakes happen -- we'd just rather they didn't :-)

Yours,

Russ Magee %-)

[ptone]

Thanks Jan for the contributions.

I'll add a couple bits to Russ's excellent reply.

I generally will run just a specific test, or a subset of the tests while developing the patch initially, this is much faster and can let you iterate much more quickly.

Julien has put together a great tool for running the full test suite  - allowing you to test more python versions including the GIS stuff:

https://github.com/jphalip/djangocore-box

Finally - note that for some work, you will occasionally hit test interaction issues, where global state causes unintended failures to happen in apparently unrelated tests. Hopefully this won't happen, but if it does and you're stumped, reach out for help.

-Preston

On Saturday, October 27, 2012 4:33:35 PM UTC-7, Jan Bednařík wrote:

Hi all,

I'm using Django for more than four years and last week I started contributing.

In docs about contributing I didn't find how detailed should be my testing while I'm writing or reviewing patch? Is enough to run tests only for patched module? Or should I run full test suite for each patch? Which combinations of Python versions and database engines are mandatory?

So far I was running Python 2.7 & SQLite3 for development/review testing of module. Python 3.2 & SQLite3, Python 2.7 & PostgreSQL for patched module and in the end Python 2.7 & SQLite3 full test suite (with selenium, etc.). Is this workflow ok?

Jan Bednařík aka Architekt

[Russell Keith-Magee]

On Sun, Oct 28, 2012 at 3:55 PM, Dominic Rodger <domini...@gmail.com> wrote:

Another long time user here (at some point I'd love to make the jump to contributor, just need to find more time) - out of interest, is there a reason we don't use Travis? I wonder if that might help those with commit access, since pre-tested pull requests would mean you could have confidence that the tests at least pass. Apologies if this has been discussed before (there are a few contributors to this group called Travis, which made searching a bit tricky!).

We do have a CI server (a Jenkins install) - you can find it here:

http://ci.django-cms.org/job/Django/

We don't currently test every fork or pull request on GitHub, but that's certainly an interesting idea, especially if we could get validation that a pull request passes the full test suite.

I haven't spent any time messing around with Travis configuration to know how hard this would be to set up -- and whether the build failures can be communicated in an effective way. If you're looking for a way to contribute, investigating the options here might be an interesting place to start.

Yours,

Russ Magee %-)

[Jan Bednařík]

Thanks for your feedbacks, they are really helpful.

I noticed existence of Jenkins test runner and also I found old
discussion about Django's testing infrastructure:
https://groups.google.com/d/topic/django-developers/_HfxwT7sVu0/discussion

What is the actual status of automated testing infrastructure? Is it
just one Jenkins server which is continuously running tests on master
branch?

Jan

> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-develop...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.

[Donald Stufft]

Travis or Jenkins can be setup to test PR's and use Github's API to mark the PR as good to merge or not.

--

[Aymeric Augustin]

We tried moving Django's CI to Travis about 6 months ago. Unfortunately, Travis' time limit is too low for Django's test suite. We contacted them, but as of August 6th, we were still waiting for them to increase the limit. I haven't had any news since then.

Over the last months I've had mixed feedback about Travis and I'm less enthusiastic than I used to be. The tendency to report failures of their infrastructure as failed tests is annoying. The .travis.yml format is simple but it's often difficult to make the builds run. I've seen many people making series of commits, blindly attempting to make their code importable, and not always succeeding. If Django uses a service, it's a strong signal for the community; I can't say I'm ready to give this signal right now for Travis.

Jenkins isn't perfect and it's painful to maintain. But it does the job and makes it easy for us to run experiments (eg. pypy, Oracle) or diagnose failures. We don't have short- or medium-terms plans to stop using Jenkins.

Finally, I'd like to thank Divio AG <https://www.divio.ch/> for funding the server that hosts ci.djangoproject.com.

-- 
Aymeric.

2012/10/29 Marko Tibold <ma...@tibold.nl>

Hi all, 

I can really recommend travis. We use it for testing Django REST Framework. The config can be as simple as this https://github.com/tomchristie/django-rest-framework/blob/restframework2/.travis.yml

The availability of the travis service is not 100 %, this is mainly due to the very large amount of OSS projects (5000+) that rely on the service. But whenever downtime occurs, the problems tend be solved quickly and status reports are sent via twitter.

Marko

To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/ddqnV4W690IJ.

[Florian Apolloner]

Hi,

On Monday, October 29, 2012 1:29:14 PM UTC+1, Aymeric Augustin wrote:

We tried moving Django's CI to Travis about 6 months ago. Unfortunately, Travis' time limit is too low for Django's test suite. We contacted them, but as of August 6th, we were still waiting for them to increase the limit. I haven't had any news since then.

We got it raised enough (I think for all Travis projects now) that the test suite does run (at least most of the times).
 

Over the last months I've had mixed feedback about Travis and I'm less enthusiastic than I used to be. The tendency to report failures of their infrastructure as failed tests is annoying.

Hehe, also arbitrary lock ups etc are not helping (I assume those are fixed/found by now).

 

The .travis.yml format is simple but it's often difficult to make the builds run. I've seen many people making series of commits, blindly attempting to make their code importable, and not always succeeding. If Django uses a service, it's a strong signal for the community; I can't say I'm ready to give this signal right now for Travis.

Indeed, in most serious cases you require a few extra bash (or whatever) scripts around it, see my branch https://github.com/apollo13/django/blob/travisci/.travis.yml and the needed config stuff: http://apolloner.eu/~apollo13/django/ -- Currently Selenium is a bit flaky, and I can't figure out why. But overall Django could run on travis (without oracle maybe…)

Jenkins isn't perfect and it's painful to maintain. But it does the job and makes it easy for us to run experiments (eg. pypy, Oracle) or diagnose failures. We don't have short- or medium-terms plans to stop using Jenkins.

+1, debugging is just hard on travis (although I assume I could download their vm and test locally, but usually local stuff works :( ). We could choose a lightweight route and just enable travis for sqlite to get pull requests tested [Which would be pretty useless for eg postgres patches]. But using Jenkins __and__ Travis-CI to test all combinations on both systems will consume more time than we have.

Cheers,
Florian

[Anssi Kääriäinen]

On 30 loka, 14:54, Florian Apolloner <f.apollo...@gmail.com> wrote:
> +1, debugging is just hard on travis (although I assume I could download
> their vm and test locally, but usually local stuff works :( ). We could
> choose a lightweight route and just enable travis for sqlite to get pull
> requests tested [Which would be pretty useless for eg postgres patches].
> But using Jenkins __and__ Travis-CI to test all combinations on both
> systems will consume more time than we have.

If PR-testing is easy and reliable (as in doesn't take much time from
our committers to maintain) then SQLite testing of pull requests on
py2 + py3 would be a good addition. Even a smaller test set of
smoketests would be good as a filter for obviously broken stuff. One
less thing to do when reviewing a patch...

Another option for better CI is "django-next" branch in git. On commit
patches are first pushed to django-next. CI tests django-next as it
does master, 1.5 and 1.4. We could then merge django-next to master
when things are looking good there (or revert if things are looking
really bad).

In addition to having one more safety net for master, the django-next
branch could also work as testing ground for more experimental
patches.

I am not at all sure django-next is a good idea. I have zero
experience about "-next" branches. All I know is that some very
successful projects are using -next branches (Git and Linux for
example).

- Anssi