Rewrite of bottleneck

[Keith Goodman]

I am working on a complete rewrite of bottleneck.

So far the rewrite contains only a few functions. Before I add more functions I'd like to know if the rewrite of bottleneck compiles on your system and if the unit tests pass.

All unit tests pass for me on

    ubuntu 14.04 64-bit with python 2.7.6
    ubuntu 14.04 64-bit with python 3.4.0

(Note that numpy 1.9.1 is required.)

The rewrite branch is here: https://github.com/kwgoodman/bottleneck/tree/rewrite

Here is a rough draft of the tentative release notes:

Bottleneck 1.0.0

This release is a complete rewrite of Bottleneck.

- Function-call overhead cut in half---a big speed up for small input arrays
- Arbitrary ndim input arrays accelerated; previously only 1d, 2d, and 3d
- Single API: bn.nansum instead of bn.nansum and nansum_2d_float64_axis0
- On 64-bit systems bn.nansum(int32) returns int32 instead of int64
- Can now install bottleneck with pip even if numpy is not already installed
- Source tarball is one-xxxx the size of the previous release
- Requires numpy 1.9.1

[Dougal Sutherland]

Tests pass for me on OSX 10.10 with 64-bit pythons 2.7.6 and 3.4.2, both with numpy 1.9.1.

In 3.4 I have a lot (thousands) of numpy warnings, though:

/usr/local/anaconda/envs/rewrite/lib/python3.4/site-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered

and

/usr/local/anaconda/envs/rewrite/lib/python3.4/site-packages/numpy/lib/nanfunctions.py:598: RuntimeWarning: Mean of empty slice

  warnings.warn("Mean of empty slice", RuntimeWarning)

These don't print for me with 2.7.

- Dougal

--
You received this message because you are subscribed to the Google Groups "bottle-neck" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bottle-neck+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christoph Deil]

I also tried with Python 3.4 on Mac OS X 10.10 and see the warnings Dougal mentioned:

https://gist.githubusercontent.com/cdeil/2c0e586a8296bb82b75a/raw/gistfile1.txt

As far as I can see the same RuntimeWarnings show up with the `master` branch.

Christoph

To unsubscribe from this group and stop receiving emails from it, send an email to bottle-neck...@googlegroups.com.

[Christoph Deil]

I think the compile time also went down an order of magnitude or two?

If that’s the case, mention this here?

[Keith Goodman]

On Tue, Dec 2, 2014 at 1:58 PM, Dougal Sutherland <dou...@gmail.com> wrote:

Tests pass for me on OSX 10.10 with 64-bit pythons 2.7.6 and 3.4.2, both with numpy 1.9.1.

Yay!

 

In 3.4 I have a lot (thousands) of numpy warnings, though:

/usr/local/anaconda/envs/rewrite/lib/python3.4/site-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered

and

/usr/local/anaconda/envs/rewrite/lib/python3.4/site-packages/numpy/lib/nanfunctions.py:598: RuntimeWarning: Mean of empty slice

  warnings.warn("Mean of empty slice", RuntimeWarning)

These don't print for me with 2.7.

I got rid of the warnings in python 2.7 with this commit:

https://github.com/kwgoodman/bottleneck/commit/afe3a6dd62c04d39918417df79d02b84f3641d87

I guess that doesn't work in python 3.4?

[Keith Goodman]

On Tue, Dec 2, 2014 at 2:09 PM, Christoph Deil <deil.ch...@googlemail.com> wrote:

emails from it, send an email to bottle-neck...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On Tue Dec 02 2014 at 4:07:05 PM Keith Goodman <kwgo...@gmail.com> wrote:

Here is a rough draft of the tentative release notes:

Bottleneck 1.0.0

This release is a complete rewrite of Bottleneck.

- Function-call overhead cut in half---a big speed up for small input arrays
- Arbitrary ndim input arrays accelerated; previously only 1d, 2d, and 3d
- Single API: bn.nansum instead of bn.nansum and nansum_2d_float64_axis0
- On 64-bit systems bn.nansum(int32) returns int32 instead of int64
- Can now install bottleneck with pip even if numpy is not already installed
- Source tarball is one-xxxx the size of the previous release

I think the compile time also went down an order of magnitude or two?

If that’s the case, mention this here?

Good idea. Done: https://github.com/kwgoodman/bottleneck/commit/e6ecf2b3d2afbbc925ef549fcab29e51222521d7

[Keith Goodman]

On Tue, Dec 2, 2014 at 2:03 PM, Christoph Deil <deil.ch...@googlemail.com> wrote:

I also tried with Python 3.4 on Mac OS X 10.10 and see the warnings Dougal mentioned:

https://gist.githubusercontent.com/cdeil/2c0e586a8296bb82b75a/raw/gistfile1.txt

I added

                         with warnings.catch_warnings():
                            warnings.simplefilter("ignore")

everywhere I could think of. Does that remove the unit test warnings in python 3.4?

I test python 3.4 with tox so I don't see the warnings.

 

[Dougal Sutherland]

Yep, no more warnings here on python 3.4.2 with def4790. I also checked 3.3.5 and 2.6.9, they also pass all tests without warnings.

Very happy that it now takes only a few minutes to compile and test on multiple platforms like this. :)

--

[Keith Goodman]

On Wed, Dec 3, 2014 at 12:03 PM, Dougal Sutherland <dou...@gmail.com> wrote:

Yep, no more warnings here on python 3.4.2 with def4790. I also checked 3.3.5 and 2.6.9, they also pass all tests without warnings.

That's great! Those warnings are a pain.

 

Very happy that it now takes only a few minutes to compile and test on multiple platforms like this. :)

It will slow down a bit (a factor of 3?) once I add all the functions to the rewrite branch. But should still be faster than before.

[Keith Goodman]

On Tue, Dec 2, 2014 at 1:07 PM, Keith Goodman <kwgo...@gmail.com> wrote:

I am working on a complete rewrite of bottleneck.

So far the rewrite contains only a few functions. Before I add more functions I'd like to know if the rewrite of bottleneck compiles on your system and if the unit tests pass.

I asked (off list) Christoph Gohlke [1] if the rewrite branch works on windows. He said all test pass for win-amd64-py3.4 with numpy 1.9.1.

 
[1] http://www.lfd.uci.edu/~gohlke/pythonlibs/#bottleneck

[Moritz Beber]

Hi,

I was wondering (I didn't look into your new implementation yet), if you are considering numpy ufuncs for your functions? At scipy they are trying to introduce them more and more including the possibility to pass in a mask for convenient missing value computations. Just wanted to know if you had considered that, are already doing that or if it doesn't really concern the scope of bottleneck.

Thanks,
Moritz

[Stephan Hoyer]

I did discuss using ufuncs when Keith brought up the rewrite of bottleneck originally. But it's not clear how much that helps, since not everything can be fit in the model of ufuncs or even gufuncs (bottleneck needs an axis argument, too).

I did make a separate project to try out using gufuncs and numba (instead of cython), which you might want to check out if you're interested in how well that approach works out: https://github.com/shoyer/numbagg

I'm pretty sure Keith is not using ufuncs here, although he is using the numpy iterator interface.

--

[Keith Goodman]

On Thu, Dec 4, 2014 at 1:16 PM, Moritz Beber <moritz...@gmail.com> wrote:

Hi,

I was wondering (I didn't look into your new implementation yet), if you are considering numpy ufuncs for your functions? At scipy they are trying to introduce them more and more including the possibility to pass in a mask for convenient missing value computations. Just wanted to know if you had considered that, are already doing that or if it doesn't really concern the scope of bottleneck.

Nope, I'm not using ufuncs for this rewrite. Maybe we'll use them in a future rewrite.

I'm using iterators from the numpy C API. I'm using the old-style numpy interators simply because the cython examples I found used the old iterators. I do wonder what this rewrite would look like with the new-style iterators.

[Keith Goodman]

On Tue, Dec 2, 2014 at 1:07 PM, Keith Goodman <kwgo...@gmail.com> wrote:

I am working on a complete rewrite of bottleneck.

So far the rewrite contains only a few functions. Before I add more functions I'd like to know if the rewrite of bottleneck compiles on your system and if the unit tests pass.

All unit tests pass for me on

    ubuntu 14.04 64-bit with python 2.7.6
    ubuntu 14.04 64-bit with python 3.4.0

Thank you to those who provided compile and unit test reports. It is a big help.

All reports were on 64-bit systems. So if anyone has a 32-bit system I'd been interested in hearing if the unit tests pass.

Also the speed of bottleneck relative to numpy can vary from system to system. So I'd be interested in seeing the benchmark numbers on your system.

On 64-bit ubuntu 14.04, python 2.7.6, intel i7-2600k:

In [1]: import bottleneck as bn
In [2]: bn.bench()
Bottleneck performance benchmark
    Bottleneck  1.0.0dev
    Numpy (np)  1.9.1
    Scipy (sp)  0.14.0
    Speed is NumPy or SciPy time divided by Bottleneck time
    NaN means one-third NaNs; float64 and axis=-1 are used

                 no NaN     no NaN     no NaN      NaN        NaN        NaN   
                 (10,)     (10,10)  (1000,1000)   (10,)     (10,10)  (1000,1000)
nansum           36.80      19.04       3.97      38.55      19.22       9.16
nanmean         142.71      72.57       5.07     140.46      71.36      10.10
nanstd          242.89      92.36       4.59     249.36      96.42       8.78
nanmax           29.41      13.55       1.07      29.61      13.79       2.75
move_mean        27.83      18.93      31.68      29.46      19.62      66.73
move_nanmean     69.04      51.69      65.21      72.44      53.34      65.82

Reference functions:
nansum         np.nansum
nanmean        np.nanmean
nanstd         np.nanstd
nanmax         np.nanmax
move_mean      sp.ndimage.convolve1d based, window=a.shape[-1] // 5
move_nanmean   sp.ndimage.convolve1d based, window=a.shape[-1] // 5

[Moritz Beber]

I recently installed your rewrite branch on my laptop, older Ubuntu 32bit system, and ran the tests and benchmarks. This is for commit 35a8c650dd7281c3ed7c8efc0d819a81e8eed83f.

The tests showed two failures. I have attached logs.

[Keith Goodman]

On Sun, Dec 14, 2014 at 7:32 AM, Moritz Beber <moritz...@gmail.com> wrote:

I recently installed your rewrite branch on my laptop, older Ubuntu 32bit system, and ran the tests and benchmarks. This is for commit 35a8c650dd7281c3ed7c8efc0d819a81e8eed83f.

The tests showed two failures. I have attached logs.

A 32-bit test it great.

I think you are running bottleneck master instead of the rewrite branch. I can tell that by the number of unit tests and by the functions in the benchmark.

[Moritz Beber]

Hi,

On Mon, Dec 15, 2014 at 5:39 PM, Keith Goodman <kwgo...@gmail.com> wrote:

I think you are running bottleneck master instead of the rewrite branch. I can tell that by the number of unit tests and by the functions in the benchmark.

Oops, you're completely correct. I was wondering why it said version 0.9 in the logs... Anyway, this time I have the right branch installed and reran the tests and benchmarks. As you will see there are 12 errors in the tests and an ImportError in the benchmark.

Cheers,
Moritz

[Keith Goodman]

The first test error is in fast_test.py. That file does not exist in the rewrite branch.

[Moritz Beber]

OK, sorry for the faulty reporting. I'm now on commit fb484e0 on the rewrite branch. I deleted previous installations of bottleneck. Still don't have the necessary partsort module but maybe that's intentional or I'm still doing something wrong.

[Keith Goodman]

On Mon, Dec 15, 2014 at 1:53 PM, Moritz Beber <moritz...@gmail.com> wrote:

OK, sorry for the faulty reporting. I'm now on commit fb484e0 on the rewrite branch. I deleted previous installations of bottleneck. Still don't have the necessary partsort module but maybe that's intentional or I'm still doing something wrong.

Can anyone reproduce what Moritz is seeing?

On 32-bit ubuntu 14.04 with python 2.7, all unit tests pass for me.