Success with Unladen Swallow

[garyrob]

I just want to mention that I am running a genetic algorithm in Python
using Unladen Swallow. Obviously GA's need to run fast.

Unladen Swallow is making it take about half as long to run a
generation, compared to CPython 2.5 (I don't have plain 2.6 currently
available to check against, but I'm guessing that most of the speedup
is due to Unladen).

Thanks! :)
Gary

P.S. depending on some experiments I'm running, this code might become
part of the music recommendation technology at http://www.flyfi.com.

[Collin Winter]

Hey Gary,

On Tue, Nov 10, 2009 at 12:47 PM, garyrob <gary...@gmail.com> wrote:
> I just want to mention that I am running a genetic algorithm in Python
> using Unladen Swallow. Obviously GA's need to run fast.
>
> Unladen Swallow is making it take about half as long to run a
> generation, compared to CPython 2.5 (I don't have plain 2.6 currently
> available to check against, but I'm guessing that most of the speedup
> is due to Unladen).

That's great news, glad to hear it's working well for you. Is this
code open-source (or open source-able)? We're always looking for more
real-world benchmarks.

Thanks,
Collin Winter

[Gary Robinson]

> That's great news, glad to hear it's working well for you. Is this
> code open-source (or open source-able)? We're always looking for more
> real-world benchmarks.

Oops, sorry for the slow response, I missed this follow-up until now. Unfortunately no, this code is not currently open-source-able.

Lately I've been running all my python code in unladen, with no problems whatsoever. Great work.

[Gary Robinson]

Oh, I should mention, that results of the recommendation processing running in unladen ARE now available at http://www.flyfi.com...

Thanks gain!
Gary

[Collin Winter]

On Thu, Dec 3, 2009 at 7:19 AM, Gary Robinson <gary...@gmail.com> wrote:
> Oh, I should mention, that results of the recommendation processing running in unladen ARE now available at http://www.flyfi.com...

Very cool.

If you find any hotspots in your application that you can reduce to
microbenchmarks, we'd be interested in adding them to our benchmark
suite so we can make Python faster for your uses.

Thanks,
Collin Winter

[garyrob]

> If you find any hotspots in your application that you can reduce to
> microbenchmarks, we'd be interested in adding them to our benchmark
> suite so we can make Python faster for your uses.

Well, now that you ask... ;)

The following code isn't actually from our FlyFi web site, it's from
the popular and successful Python-based SpamBayes email filter. The
chi2P function was written by Tim Peters. While it's not part of
FlyFi, it's typical of some of the statistical code we have in FlyFi.

It runs a little faster in python 2.5 than it does in unladen (20
seconds vs 22 on one of our Linux boxes; the unladen version was run
with "-j always").

from math import exp
from random import random
import time

def chi2P(chi, df):
"""
Return prob(chisq >= chi, with df degrees of freedom).
df must be even.
"""
assert df & 1 == 0
# XXX If chi is very large, exp(-m) will underflow to 0.
m = chi / 2.0
sum = term = exp(-m)
for i in range(1, df//2):
term *= m / i
sum += term
return min(sum, 1.0)

t = time.time()
for i in range(int(1e7)):
chi2P(1.0 / random(), 4)
print 'Took:', time.time() - t

[Collin Winter]

[+Skip]

On Wed, Dec 9, 2009 at 8:22 AM, garyrob <gary...@gmail.com> wrote:
>> If you find any hotspots in your application that you can reduce to
>> microbenchmarks, we'd be interested in adding them to our benchmark
>> suite so we can make Python faster for your uses.
>
> Well, now that you ask...  ;)
>
> The following code isn't actually from our FlyFi web site, it's from
> the popular and successful Python-based SpamBayes email filter. The
> chi2P function was written by Tim Peters. While it's not part of
> FlyFi, it's typical of some of the statistical code we have in FlyFi.

Cool, thanks! I'll look at making a microbenchmark out of this kernel.

Skip, are there any macro-ish benchmarks for SpamBayes? It would be
nice to have a larger-scale benchmark for a system like SpamBayes. I
looked through the SVN repo and Google, but I couldn't find anything.

Thanks,
Collin Winter

[Collin Winter]

On Wed, Dec 9, 2009 at 9:59 AM, Collin Winter <collin...@google.com> wrote:
> [+Skip]
>
> On Wed, Dec 9, 2009 at 8:22 AM, garyrob <gary...@gmail.com> wrote:
>>> If you find any hotspots in your application that you can reduce to
>>> microbenchmarks, we'd be interested in adding them to our benchmark
>>> suite so we can make Python faster for your uses.
>>
>> Well, now that you ask...  ;)
>>
>> The following code isn't actually from our FlyFi web site, it's from
>> the popular and successful Python-based SpamBayes email filter. The
>> chi2P function was written by Tim Peters. While it's not part of
>> FlyFi, it's typical of some of the statistical code we have in FlyFi.
>
> Cool, thanks! I'll look at making a microbenchmark out of this kernel.

Looking at this, I think the range of math operations here are covered
by the nbody benchmark I just added. I'd still be interested in a
large-scale SpamBayes benchmark, but for floating-point
microbenchmarks, I think we're good for the moment.

On that note, I'm finishing up some polishing work on ebo's inlining
patch, which should help most float-heavy apps. Gary, I'll ping you
when that patch goes in; I'll be curious how it impacts your app's
overall performance.

Thanks,
Collin Winter

[skip.mo...@gmail.com]

(Sorry for the possible/probable duplicates. Didn't see this turn up in my
inbox so I figured maybe Google Groups rejected it since it wasn't "From:"
my Gmail address.)

Collin> I'd still be interested in a large-scale SpamBayes benchmark,
Collin> but for floating-point microbenchmarks, I think we're good for
Collin> the moment.

I'm one of the SpamBayes developers. Do you have something specific in
mind when you say "large-scale"?

Skip

[sk...@pobox.com]

[Valery]

Hi Collin and Skip,

the training part of bayesian spam filter also has code like from
recent benchmark snippet:
http://groups.google.com/group/unladen-swallow/msg/cb928c20abfc3d39

Namely, code that counts the number of appearances of words in
(training) set of texts.

The calculation isn't fast, when set of texts contains 10 thousands of
1Kbyte texts...

@Skip, maybe you could confirm this and supply your benchmark test on
that?

Regards,
Valery

[garyrob]

> I'm one of the SpamBayes developers. Do you have something specific in
> mind when you say "large-scale"?

Just for proper introductions in this forum, I'm one of the SpayBayes
contributors too, in that I contributed a couple of the bayesian and
non-bayesian statistical ideas it uses... didn't write any of the code
though! :) OTOH I'm writing a ton of Python code for FlyFi.

[Collin Winter]

Hey Skip,

I was thinking of something like "run N messages through a trained
ham/spam classifier" or something like that.

Thanks,
Collin Winter

[sk...@pobox.com]

>> I'm one of the SpamBayes developers. Do you have something specific
>> in mind when you say "large-scale"?

Collin> I was thinking of something like "run N messages through a
Collin> trained ham/spam classifier" or something like that.

Ok, I can probably whip something up. Is there some structural convention
about this? You might want to run a certain number of passes to prime the
pump before starting the timer, right? Minimize actual I/O I presume?

Gary, good to see you pop up again. Can you build a SpamBayes installer for
Windows? <wink>

Skip

[Collin Winter]

Hey Skip,

2009/12/15 <sk...@pobox.com>:

>
>    >> I'm one of the SpamBayes developers.  Do you have something specific
>    >> in mind when you say "large-scale"?
>
>    Collin> I was thinking of something like "run N messages through a
>    Collin> trained ham/spam classifier" or something like that.
>
> Ok, I can probably whip something up.  Is there some structural convention
> about this?  You might want to run a certain number of passes to prime the
> pump before starting the timer, right?  Minimize actual I/O I presume?

Yep, there are some conventions about how to do it:
http://code.google.com/p/unladen-swallow/source/detail?r=926 is a
recent benchmark addition, and
http://code.google.com/p/unladen-swallow/source/browse/tests/performance/bm_django.py
is an example of benchmarking a library.
http://code.google.com/p/unladen-swallow/source/browse/tests/performance/bm_rietveld.py
is a somewhat larger, multi-library benchmark that uses canned data.

Overall:
- Do some priming runs.
- Collect N timing samples; I generally like each iteration to take
300-500ms to hide machine jitter.
- Minimize IO during each iteration.
- Change perf.py to expose the benchmark and process the data. There's
now a lot of abstraction so you don't have to write so much
boilerplate to add a new benchmark.

Thanks,
Collin Winter

[sk...@pobox.com]

Collin> Yep, there are some conventions about how to do it:
Collin> http://code.google.com/p/unladen-swallow/source/detail?r=926 is a
Collin> recent benchmark addition, and

Thanks. That will probably be sufficient (I'm close to having a running
benchmark now but need to tune it to get the time for each iteration in the
right ballpark), but I can't tell where to get the util module you imported.

Skip

[James Abbatiello]

On Wed, Dec 16, 2009 at 10:10 PM, <sk...@pobox.com> wrote:
> Thanks.  That will probably be sufficient (I'm close to having a running
> benchmark now but need to tune it to get the time for each iteration in the
> right ballpark), but I can't tell where to get the util module you imported.

http://code.google.com/p/unladen-swallow/source/browse/tests/performance/util.py

--
James Abbatiello

[sk...@pobox.com]

>> Thanks. That will probably be sufficient (I'm close to having a
>> running benchmark now but need to tune it to get the time for each
>> iteration in the right ballpark), but I can't tell where to get the
>> util module you imported.

James> http://code.google.com/p/unladen-swallow/source/browse/tests/performance/util.py

Duh. I didn't even see that the file display had a scrollbar.

And since tests is its own branch that would explain why I wasn't getting
any update from my top-level checkout. I actually have the file on my
computer now. I should be good to go.

Thanks,

S