Running full test during development

[Guru Devanla]

Hello All,

When I run the complete tests using  bin/test,  my tests on the machine I run takes close to 5 hours. I am sure I am missing something.

Can someone point me to resource, where I configure my tests to run faster.

My current system:

Ubuntu 11.04 , 32-bit

Kernel Linux 2.6

Hardware

Memory : 4GB

Intel Dual Core 2.66 Ghz

Python version

Python 2.7.1  ( 32-bit)

Thanks

Guru

[Aaron Meurer]

It shouldn't take nearly that long. Can you make a note of which part of the suite takes the longest. And show us the info printed by the test runner at the beginning (Python version, ground types, etc.). I don't suppose you have the cache turned off?

Aaron Meurer

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To view this discussion on the web visit https://groups.google.com/d/msg/sympy/-/kiZ7e62bCB0J.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.

[Guru Devanla]

Aaron,

I did not pass any arguments while running the test. 

Here is what gets printed in the header:

executable:   /usr/bin/python  (2.7.1-final-0)

architecture: 32-bit

cache:        yes

ground types: python

random seed:  17908803

Also, I printed out the list of all the options that were set :

 {'sort': True, 'slow': False, 'verbose': False, 'cache': True, 'colors': True, 'seed': None, 'kw': '', 'timeout': False, 'force_colors': False, 'tb': 'short', 'pdb': False, 'types': None}

I will try to time the other tests and share that info when I run them again.

Regards

Guru

[Ronan Lamy]

Le samedi 26 mai 2012 à 13:03 -0500, Guru Devanla a écrit :

> I did not pass any arguments while running the test.
>
> Here is what gets printed in the header:

> executable: /usr/bin/python (2.7.1-final-0)
> architecture: 32-bit
> cache: yes
> ground types: python
> random seed: 17908803

> Also, I printed out the list of all the options that were set :

> {'sort': True, 'slow': False, 'verbose': False, 'cache': True,
> 'colors': True, 'seed': None, 'kw': '', 'timeout': False,
> 'force_colors': False, 'tb': 'short', 'pdb': False, 'types': None}

That seems normal.

>
> I will try to time the other tests and share that info when I run them
> again.

We have a few tests that are meant to take very long instead of failing.
Maybe you're triggering one of these pseudo-failures. Does it take as
long when you test "master"? How long does it take to run 'bin/test
sympy/core' (shouldn't be more than a minute)?

[Aaron Meurer]

That's a good point. You may be running the slow test for some
reason. When you run the tests, you should occasionally see a blue w
(instead of a . or an f). There is one in
sympy/assumptions/tests/test_query.py for example. Under normal
testing conditions, these should be skipped. So check if these are
the tests that are taking so long. You can run them manually with
./bin/test --slow.

Aaron Meurer

[Ronan Lamy]

That's not what I meant. According to the printed list of options, slow
tests are deactivated, as they should be. There's at least one test, in
the core, that behaves thus: when all is fine, it reports success in a
fraction of a second, but when there is a problem, it will still
(eventually) report success, but it will run for an eternity and a half.

[krastano...@gmail.com]

I doubt it is related, however I noticed that you are using 32bit
software on 64bit hardware. It has a slim chance to be related.

[Guru Devanla]

I ran all the tests individually ( by selecting each folder ) and most of the tests completed quickly and few of them took a few minutes. For example, here are some numbers:

( the durations are the values displayed by test results)

sympy/matrices :  195 sec

sympy/simplify : 268 sec

sympy/integrals/test/test_meijerint.py : 88 sec

But, after I run the test on sympy/integrals/test/test_integrals.py ( which always takes a long time ), my machine drastically slows down and then even the test that ran earlier took longer before restarting the machine. And, therefore, the tests takes hours to complete.  

(Note, how long simplify and test/test_meijer_int takes after running test/ingetrals/test_integrals.)

./bin/test sympy/integrals/tests/test_integrals.py 

================= tests finished: 77 passed, 3 expected to fail, in 3664.50 seconds =================

sympy/integrals/tests/test_meijerint.py[18] ..................                                   

=========================== tests finished: 18 passed, in 3656.80 seconds ===========================

./bin/test sympy/simplify/

========== tests finished: 113 passed, 13 skipped, 11 expected to fail, in 2443.58 seconds ==========

Regards

Guru

[Joachim Durchholz]

Am 27.05.2012 13:07, schrieb Guru Devanla:
> I ran all the tests individually ( by selecting each folder ) and most of
> the tests completed quickly and few of them took a few minutes. For
> example, here are some numbers:
>
> ( the durations are the values displayed by test results)
>
> sympy/matrices : 195 sec
> sympy/simplify : 268 sec
> sympy/integrals/test/test_meijerint.py : 88 sec

This is faster on my machine, but that may be hardware differences and
doesn't seem to be out of the ordinary.

> But, after I run the test on *sympy/integrals/test/test_integrals.py* (

> which always takes a long time ), my machine drastically slows down and
> then even the test that ran earlier took longer before restarting the
> machine. And, therefore, the tests takes hours to complete.

Sounds like the tests are filling up some resource (RAM, possibly disk)
and the machine is thrashing.
Check RAM and disk load.
RAM load: top
Disk load: df

SymPy is creating a cache, it may be that the cache is filling up your
available RAM and the machine starts to thrash.
(Does anybody know how much RAM a full SymPy test suite requires?)

[krastano...@gmail.com]

My bot runs well on a 512MB machine.

And I do not get how one run of the tests (integrals) affects
consecutive runs...

[Joachim Durchholz]

Am 27.05.2012 18:26, schrieb krastano...@gmail.com:
> And I do not get how one run of the tests (integrals) affects
> consecutive runs...

SymPy has a cache. Later tests reuse results from earlier computations.
This has caused problems with reproducing errors in the past, but it
does have the advantage that we have a rough estimate of how good SymPy
is at returning the same results for the same inputs.

BTW I agree that RAM pressure is an unlikely cause given the machine,
we're out of likely reasons already.
Feel free to propose checking a more likely cause if you know one.

[krastano...@gmail.com]

On 27 May 2012 19:09, Joachim Durchholz <j...@durchholz.org> wrote:
> Am 27.05.2012 18:26, schrieb krastano...@gmail.com:
>
>> And I do not get how one run of the tests (integrals) affects
>> consecutive runs...
>
>
> SymPy has a cache. Later tests reuse results from earlier computations.
> This has caused problems with reproducing errors in the past, but it does
> have the advantage that we have a rough estimate of how good SymPy is at
> returning the same results for the same inputs.

But the cache is not retained between different runs of the test
runner, is it? Or is it saved to disk?

[Vladimir Perić]

On Sun, May 27, 2012 at 6:26 PM, krastano...@gmail.com
<krastano...@gmail.com> wrote:
>> SymPy is creating a cache, it may be that the cache is filling up your
>> available RAM and the machine starts to thrash.
>> (Does anybody know how much RAM a full SymPy test suite requires?)
>>
> My bot runs well on a 512MB machine.

Yeah, the tests used to leak horribly, but we fixed that when we were
trying to get Jenkins to run, by clearing the cache between every two
files (see issue 2585 [1]).

[1] http://code.google.com/p/sympy/issues/detail?id=2585

>
> And I do not get how one run of the tests (integrals) affects
> consecutive runs...
>

Unfortunately, brings us back to swapping issues. I guess the root of
the cause is the 32-bit OS (and perhaps less importantly, the 32-bit
Python). Does anyone actually develop with a 32-bit Linux?

Guru, if you happen to have too much spare time :), perhaps you could
try running a LiveCD of the 64-bit version of Ubuntu, and see if that
helps perhaps. Other than that, I'm quite clueless.

--
Vladimir Perić

[Joachim Durchholz]

Am 27.05.2012 19:16, schrieb krastano...@gmail.com:
> On 27 May 2012 19:09, Joachim Durchholz<j...@durchholz.org> wrote:
>> Am 27.05.2012 18:26, schrieb krastano...@gmail.com:
>>
>>> And I do not get how one run of the tests (integrals) affects
>>> consecutive runs...
>>
>> SymPy has a cache. Later tests reuse results from earlier computations.
>> This has caused problems with reproducing errors in the past, but it does
>> have the advantage that we have a rough estimate of how good SymPy is at
>> returning the same results for the same inputs.
> But the cache is not retained between different runs of the test
> runner, is it? Or is it saved to disk?

It isn't AFAIK.
It is kept from test to test if you run a suite of tests though.
If it's exerting RAM pressure, that would be an explanation why a test
suite become slow after a few tests.

Well, that used to be my knowledge. Vlada's response in the other
subthread indicates that it's otherwise.

Which leaves me clueless about the actual problem in this case.
I guess we'll have to wait until Guru comes back with some numbers.

[Aaron Meurer]

This is indeed very odd. Can you run the integrals test with the -v
option and let us know which test(s) take so long?

Aaron Meurer

[Guru Devanla]

I did run the test using Live CD 12.04 64-bit. I see the same kind of behavior.

May be I should also mention that my machine slows down for things like Flash and I am forced to restart my machine during such instances as well ( just after watching 3/4 mi of video).  I am not sure if this is  related at all.

One more observation was, after a  complete restart, running  ./bin/test sympy/test/test_integrals.py does take just under 2 minutes. But, subsequent runs slow down drastically and takes up to 50 minutes..

Anyways, I don't plan on spending any more time on this now that I am out of the weekend, and plan to switch to 12.04 ubuntu, hoping it will fix my flash related problems as well. 

-Guru

[Guru Devanla]

Aaron,

I narrowed the test down to test/test_integrals.py ( if this is the information you are looking for).

Again, as I mentioned in the other post, first time around this test just takes few minutes, but after that it does slow down my machine.

I can run all other tests(skipping test_integrals.py) as many times as I want, and the tests complete in reasonable time ( few minutes)

Regards

Guru

> sympy+unsubscribe@googlegroups.com.

[krastano...@gmail.com]

On 29 May 2012 13:37, Guru Devanla <grd...@gmail.com> wrote:
> Aaron,
>
> I narrowed the test down to test/test_integrals.py ( if this is the
> information you are looking for).
>
> Again, as I mentioned in the other post, first time around this test just
> takes few minutes, but after that it does slow down my machine.
>
> I can run all other tests(skipping test_integrals.py) as many times as I
> want, and the tests complete in reasonable time ( few minutes)
>

Nevertheless, it would be interesting to know what exactly slows down
test_integrals. If you run `bin/test -v test_integrals` it will print
out each of the tests in this file and we would be able to pinpoint
the problem.

[Guru Devanla]

I did run it with -v option. Almost each of the tests seems to take longer than a minute and hence overall slowdown, and I did not notice a specific test siting there for long and contributing to the slowness.

Also, note that all other tests ( outside integrals) also slow down since the machine slows down.

I can still run some more tests, if anyone has more suggestions of narrowing down the problem.

-Guru

[Vladimir Perić]

On Tue, May 29, 2012 at 1:33 PM, Guru Devanla <grd...@gmail.com> wrote:
> I did run the test using Live CD 12.04 64-bit. I see the same kind of
> behavior.
>
> May be I should also mention that my machine slows down for things like
> Flash and I am forced to restart my machine during such instances as well (
> just after watching 3/4 mi of video).  I am not sure if this is  related at
> all.

Ah, could it be that your CPU is overheating (and hence downclocking
itself), which kills performance? The whole SymPy test suite is
CPU-bottlenecked, but some of the tests in test_integrals are
particularly CPU-intensive. The CPU fan in my laptop died a few weeks
ago, and these are definite symptoms of it. To check, try monitoring
your fan speeds or, failing that, CPU temperatures. If they are
consistently high (eg. my laptop goes up to 80 degrees), then that is
the definite issue.

Now, was the issue with Flash video present from the start? If so,
it's possible your fans just aren't supported by current Linux kernels
(or badly supported); in that case, perhaps try if Windows would work
better. If it's a recent issue, it might be that you had the same
situation as me, so you could look into getting a replacement fan. The
third, and best possible, option, is that you've just had a gradual
build-up of dust in your CPU: in that case, get a can of compressed
air and try to clean it as much as you can (there are guides for this
on the internet, but the most important thing to remember is to use
just short bursts). Even just blowing from the outside (without
dissasembling your whole laptop) can be a tremendous help.

Good luck! (also note, don't listen to random guys on the internet too
much -- ie. if your laptop stops working, it's not my fault :))

> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/sympy/-/r6JQ05B1DAMJ.

>
> To post to this group, send email to sy...@googlegroups.com.
> To unsubscribe from this group, send email to
> sympy+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/sympy?hl=en.



--

Vladimir Perić

[Guru Devanla]

You just described the state of my machine!!   I have noticed my machine slowing down lately when I watch Flash videos. I did notice that it was getting heated up.  

Similarly, the instant I start the tests under discussion, the temp shoots up to 80+ degrees

 

Good luck! (also note, don't listen to random guys on the internet too
much -- ie. if your laptop stops working, it's not my fault :))

Will look into your advice and blame the appropriate person :)

 

> sympy+unsubscribe@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/sympy?hl=en.



--
Vladimir Perić

> sympy+unsubscribe@googlegroups.com.