Strange difference in runtimes of functionally equivalent scripts
Ricey
# start
"""Project Euler problem 23 - slow version"""
from eulertools import factors
ab = []
for n in range(12, 28124):
if sum(factors(n)[:-1])>n:
ab.append(n)
nums = []
for n in range(len(ab)):
for m in range(n, len(ab)):
v = ab[n]+ab[m]
if v > 28123: break
nums.append(v)
print sum(range(28124))-sum(list(set(nums)))
# end
and (2):
#start
"""Project Euler problem 23 - quick version"""
from eulertools import factors
def ab(lim):
nums = []
for n in range(12, lim+1):
if sum(factors(n)[:-1])>n:
nums.append(n)
return nums
def nm(lst):
nums = []
for n in range(len(lst)):
for m in range(n, len(lst)):
v = lst[n]+lst[m]
if v > 28123: break
nums.append(v)
return nums
abundants = ab(28123)
nums = nm(abundants)
print sum(range(28124))-sum(list(set(nums)))
# end
the eulertools.factor() function looks like this:
# start
from math import sqrt, ceil
def factors(n):
"""Returns an ordered list of factors of n"""
sqrtn = sqrt(n)
facts = not sqrtn%1 and [int(sqrtn)] or []
for i in range(int(ceil(sqrtn))-1, 0, -1):
if n%i == 0:
facts = [i] + facts + [n/i]
return facts
# end
Now, when I run these two scripts at the command line they take about
the same time to run, as you'd expect - about 10 seconds on my old and
clunky laptop. However, if I run them in Spyder the first version,
where the loops are not enclosed in functions, takes several times
longer: almost a minute, in fact. Can anyone suggest a reason for this
discrepancy? Spyder is far and away the best IDE for Python I've
found, but this is just bizarre.
David Verelst
--
You received this message because you are subscribed to the Google Groups "spyder" group.
To post to this group, send email to spyd...@googlegroups.com.
To unsubscribe from this group, send email to spyderlib+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/spyderlib?hl=en.
Ricey
Debian Testing repository and Spyder 2.1.2 direct from Googlecode,
both with the same result: on running the version with the functions
the CPU load jumps to about 50% and stays there for 10 seconds, while
RAM use goes up by about 200MB (I have 4GB so that's not an issue),
and after 10 seconds I get the answer; while on running the other one,
without functions, the load goes to 50% for the first 2 seconds while
the first loop is processed, then drops off to almost idling for the
next 40+ seconds, although memory usage is similar to in the first
instance.
The code, by the way, is a solution to Project Euler problem 23:
projecteuler.net/problem=23 - so it's not terribly important in the
greater scheme of things. It's just a bit strange and I thought maybe
someone could shed some light on it.
Thanks for your time
Col
On Nov 23, 9:43 am, David Verelst <david.vere...@gmail.com> wrote:
> On my machine it takes in the spyder consoles (internal, external and
> Ipython) on average 4.8-5.0 seconds for both versions of you code. An
> independent python shell does it slightly faster, in 4 seconds. I am not
> sure as why there is this difference (sorry can't help you there).
>
> However, I noticed the algorithm uses a lot of memory. Storing a lot of
> numbers in a list is not a particular efficient way of dealing with large
> sets of numbers I guess? So that might be part of the problem (just
> thinking out loud here, not sure if its true). In the spyder's case you
> might have more memory in use (you ran the code a few times before in the
> same console, other variables hanging around, ...) and you hit the end of
> your RAM. Since your function keeps eating memory it gets slowed down by
> the slow hard disk.
>
> On a side note, I am not sure I understand what the meaning of the code is.
>
> Maybe you could also share some important details as python and spyder
> versions, operating system, etc.
>
> Regards,
>
> David
>
Pierre Raybaut
I think that the problem of the first code sample is that it's not
efficient at all and all these long (and very inefficient!) 'for'
loops tend to slow down the other threads running together with your
code. Actually there are two other threads running in the background
in a Spyder's console with standard settings, providing code
introspection features and the variable explorer support. To disable
these features and thus accelerate your code execution, you may
disable the monitor. But note that -in most of the cases- this should
not be necessary.
I checked with your code:
* the first code sample being inefficient:
* with monitor: it runs approx. two times slower in Spyder than outside
* without monitor: no difference between execution inside or outside Spyder
* with the second code sample (the efficient one), I see no
significative difference between all combinations (inside/outside
Spyder, with/without monitor)
If those performance limitations are reported in other situations, I
shall investigate further.
HTH,
Pierre
2011/11/23 Ricey <colin...@gmail.com>:
Pierre Raybaut
After more tests, it appears to be only related to the Variable
Explorer's automatic refresh feature. If you disable it, runtime will
be back to normal.
So, you may let the monitor enabled and try to disable the Variable
Explorer's autorefresh setting (Preferences > Variable Explorer).
-Pierre
On 26 nov, 16:05, Pierre Raybaut <pierre.rayb...@gmail.com> wrote:
> Hi,
>
> I think that the problem of the first code sample is that it's not
> efficient at all and all these long (and very inefficient!) 'for'
> loops tend to slow down the other threads running together with your
> code. Actually there are two other threads running in the background
> in a Spyder's console with standard settings, providing code
> introspection features and the variable explorer support. To disable
> these features and thus accelerate your code execution, you may
> disable the monitor. But note that -in most of the cases- this should
> not be necessary.
>
> I checked with your code:
> * the first code sample being inefficient:
> * with monitor: it runs approx. two times slower in Spyder than outside
> * without monitor: no difference between execution inside or outside Spyder
> * with the second code sample (the efficient one), I see no
> significative difference between all combinations (inside/outside
> Spyder, with/without monitor)
>
> If those performance limitations are reported in other situations, I
> shall investigate further.
>
> HTH,
> Pierre
>
> 2011/11/23 Ricey <colinvic...@gmail.com>:
Ricey
Spyder is fantastic!
~Col
Pierre Raybaut
Thanks to your report, I was able to investigate further and found out
why the Variable Explorer's refresh process is slowing down things so
much in this particular case:
http://code.google.com/p/spyderlib/source/detail?r=ae3269b7aceb8d282ff5b900d900d5106b5d2752
As mentioned in the revision cited above, I was able to solve the
problem and now there is no difference of performance between running
this code inside or outside Spyder. And you are no longer forced to
disable the Variable Explorer's "automatic refresh" feature. In fact,
you may even use it to monitor the progress of your script's execution
-- see this youtube video I just uploaded:
http://www.youtube.com/watch?v=Uf5nRTq59j4
(this screencast was recorded on a very old laptop, hence the poor 30s
runtime...)
Cheers,
Pierre