Debugging parallel nose tests?

[Roy Smith]

I've got a suite of about 150 tests written using unittest. It takes
5-10 minutes to run, so I'd really like to use nose to run things in
parallel. The problem is, when I do that, I get lots of test failures.
Obviously, we've got some kind of inter-test dependency that I haven't
been able to locate.

Is there some way to make nose print a report of how it partitioned the
tests across the various processes? If I could see that, it might help
us reproduce the failures.

We're using:

nosetests --process-timeout=60 --processes=40 test_api.py

and

_multiprocess_can_split_ = True

[Dave Angel]

On 05/23/2013 09:09 AM, Roy Smith wrote:
>
> <SNIP>
>
> nosetests --process-timeout=60 --processes=40 test_api.py
>

Do you have a 40-processor system? And do you have enough RAM to run
all of those processes?


--
DaveA

[Roy Smith]

In article <mailman.2027.1369333...@python.org>,

Dave Angel <da...@davea.name> wrote:

> On 05/23/2013 09:09 AM, Roy Smith wrote:
> >
> > <SNIP>
> >
> > nosetests --process-timeout=60 --processes=40 test_api.py
> >
>
> Do you have a 40-processor system?

No, but many of the tests are I/O bound.

> And do you have enough RAM to run all of those processes?

Yes. The box I'm on now has 8 gig. I'd doing a 40 process run right
now, and I've still got 1/2 gig free.

[Roy Smith]

On May 24, 2013, at 5:05 AM, Jean-Michel Pichavant wrote:

> ----- Original Message -----
>> In article <mailman.2027.1369333...@python.org>,
>> Dave Angel <da...@davea.name> wrote:
>>
>>> On 05/23/2013 09:09 AM, Roy Smith wrote:
>>>>
>>>> <SNIP>
>>>>
>>>> nosetests --process-timeout=60 --processes=40 test_api.py
>>>>
>>>
>>> Do you have a 40-processor system?
>>
>> No, but many of the tests are I/O bound.
>

> Sorry to hijack your thread but what do you mean by that ?

That means that a lot of time running the test is spent waiting for I/O (in this case, network reads), as opposed to actually running on a CPU.

> I have a lot of tests myself that spend most of their time writing and reading files. Should I try to multiprocess them ?

Maybe :-)

--
Roy Smith
r...@panix.com

[Roy Smith]

In article <roy-1CC2EA.0...@news.panix.com>,

Roy Smith <r...@panix.com> wrote:

> Is there some way to make nose print a report of how it partitioned the
> tests across the various processes?

I never found such a feature, but I did figure out a way to do what I
needed. We use a system of unique id's to track HTTP requests through
our system. All you need do is add a X-Unique-Request-ID header and any
of our server processes will log that. So, I just had my test suite add:

X-Unique-Request-ID: name-of-test.process-id

to each HTTP request. That got me what I was looking for (to know what
tests were grouped into the same process).

> We're using:
>
> nosetests --process-timeout=60 --processes=40 test_api.py

It turned out, the answer was purely a timing issue. We had some tests
that called out to facebook. Those were (intermittently) so slow they
caused the 60 second timeout to be exceeded. The solution there was to
pull the facebook tests out to their own suite (which we'll run in
single-process mode).