Fixed race condition

[Mark]

I tracked down and fixed the bug reported by Carlos, which turned out
to be a race condition. I also closed a few issues fixed by the switch
to nditer. That means there are no known issues I'm planning to look
at, so things should be clear to take it through a release cycle when
Gaetan has enough available time.

Cheers,
Mark

[Francesc Alted]

2011/10/31 Mark <mww...@gmail.com>:

I'll have a look at your changes. Meanwhile, I've tried the 'default'
branch and NumPy 1.6.0 and I still get a bunch of warnings/errors when
running the test suite (see below). Are you seeing them too, or it is
just me?

$ PYTHONPATH=. python numexpr/tests/test_numexpr.py
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Numexpr version: 1.4.3.dev277
NumPy version: 1.6.0
Python version: 2.6.1 (r261:67515, Feb 3 2009, 17:34:37)
[GCC 4.3.2 [gcc-4_3-branch revision 141291]]
Platform: linux2-x86_64
AMD/Intel CPU? True
VML available? False
Detected cores: 2
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
...............E...........E..'a + ones_like(b+c)' not implemented for
complex (scalar=0, opt=aggressive)
................................................................................'(a+1)**0'
not implemented for complex (scalar=0, opt=aggressive)
.'(a+1)**0.0' not implemented for complex (scalar=0, opt=aggressive)
......'a + ones_like(b+c)' not implemented for complex (scalar=0, opt=moderate)
................................................................................'(a+1)**0'
not implemented for complex (scalar=0, opt=moderate)
.'(a+1)**0.0' not implemented for complex (scalar=0, opt=moderate)
......'a + ones_like(b+c)' not implemented for complex (scalar=0, opt=none)
................................................................................'(a+1)**0'
not implemented for complex (scalar=0, opt=none)
.'(a+1)**0.0' not implemented for complex (scalar=0, opt=none)
......'a + ones_like(b+c)' not implemented for float32 (scalar=0,
opt=aggressive)
.....numexpr/tests/test_numexpr.py:1: RuntimeWarning: invalid value
encountered in arcsin
import new, sys, os
.numexpr/tests/test_numexpr.py:1: RuntimeWarning: invalid value
encountered in arccos
import new, sys, os
..numexpr/tests/test_numexpr.py:1: RuntimeWarning: overflow encountered in sinh
import new, sys, os
.numexpr/tests/test_numexpr.py:1: RuntimeWarning: overflow encountered in cosh
import new, sys, os
...numexpr/tests/test_numexpr.py:1: RuntimeWarning: invalid value
encountered in arccosh
import new, sys, os
.numexpr/tests/test_numexpr.py:1: RuntimeWarning: invalid value
encountered in arctanh
import new, sys, os
.numexpr/tests/test_numexpr.py:1: RuntimeWarning: divide by zero
encountered in log
import new, sys, os
..numexpr/tests/test_numexpr.py:1: RuntimeWarning: divide by zero
encountered in log10
import new, sys, os
.numexpr/tests/test_numexpr.py:1: RuntimeWarning: overflow encountered in exp
import new, sys, os
.numexpr/tests/test_numexpr.py:1: RuntimeWarning: overflow encountered in expm1
import new, sys, os
..............................................................................................'(a+1)**0'
not implemented for float32 (scalar=0, opt=aggressive)
.'(a+1)**0.0' not implemented for float32 (scalar=0, opt=aggressive)
......'a + ones_like(b+c)' not implemented for float32 (scalar=0, opt=moderate)
................................................................................................................'(a+1)**0'
not implemented for float32 (scalar=0, opt=moderate)
.'(a+1)**0.0' not implemented for float32 (scalar=0, opt=moderate)
......'a + ones_like(b+c)' not implemented for float32 (scalar=0, opt=none)
................................................................................................................'(a+1)**0'
not implemented for float32 (scalar=0, opt=none)
.......'a + ones_like(b+c)' not implemented for float64 (scalar=0,
opt=aggressive)
................................................................................................................'(a+1)**0'
not implemented for float64 (scalar=0, opt=aggressive)
.'(a+1)**0.0' not implemented for float64 (scalar=0, opt=aggressive)
......'a + ones_like(b+c)' not implemented for float64 (scalar=0, opt=moderate)
................................................................................................................'(a+1)**0'
not implemented for float64 (scalar=0, opt=moderate)
.'(a+1)**0.0' not implemented for float64 (scalar=0, opt=moderate)
......'a + ones_like(b+c)' not implemented for float64 (scalar=0, opt=none)
................................................................................................................'(a+1)**0'
not implemented for float64 (scalar=0, opt=none)
.'(a+1)**0.0' not implemented for float64 (scalar=0, opt=none)
......'a + ones_like(b+c)' not implemented for int (scalar=0, opt=aggressive)
.............numexpr/tests/test_numexpr.py:1: RuntimeWarning: divide
by zero encountered in arctanh
import new, sys, os
...................................................................................................'(a+1)**0'
not implemented for int (scalar=0, opt=aggressive)
.'(a+1)**0.0' not implemented for int (scalar=0, opt=aggressive)
......'a + ones_like(b+c)' not implemented for int (scalar=0, opt=moderate)
................................................................................................................'(a+1)**0'
not implemented for int (scalar=0, opt=moderate)
.'(a+1)**0.0' not implemented for int (scalar=0, opt=moderate)
......'a + ones_like(b+c)' not implemented for int (scalar=0, opt=none)
................................................................................................................'(a+1)**0'
not implemented for int (scalar=0, opt=none)
.'(a+1)**0.0' not implemented for int (scalar=0, opt=none)
......'a + ones_like(b+c)' not implemented for long (scalar=0, opt=aggressive)
................................................................................................................'(a+1)**0'
not implemented for long (scalar=0, opt=aggressive)
.'(a+1)**0.0' not implemented for long (scalar=0, opt=aggressive)
......'a + ones_like(b+c)' not implemented for long (scalar=0, opt=moderate)
................................................................................................................'(a+1)**0'
not implemented for long (scalar=0, opt=moderate)
.'(a+1)**0.0' not implemented for long (scalar=0, opt=moderate)
......'a + ones_like(b+c)' not implemented for long (scalar=0, opt=none)
................................................................................................................'(a+1)**0'
not implemented for long (scalar=0, opt=none)
.'(a+1)**0.0' not implemented for long (scalar=0, opt=none)
......'a + ones_like(b+c)' not implemented for complex (scalar=1,
opt=aggressive)
................................................................................'(a+1)**0'
not implemented for complex (scalar=1, opt=aggressive)
.'(a+1)**0.0' not implemented for complex (scalar=1, opt=aggressive)
......'a + ones_like(b+c)' not implemented for complex (scalar=1, opt=moderate)
................................................................................'(a+1)**0'
not implemented for complex (scalar=1, opt=moderate)
.'(a+1)**0.0' not implemented for complex (scalar=1, opt=moderate)
......'a + ones_like(b+c)' not implemented for complex (scalar=1, opt=none)
................................................................................'(a+1)**0'
not implemented for complex (scalar=1, opt=none)
.'(a+1)**0.0' not implemented for complex (scalar=1, opt=none)
......'a + ones_like(b+c)' not implemented for float32 (scalar=1,
opt=aggressive)
................................................................................................................'(a+1)**0'
not implemented for float32 (scalar=1, opt=aggressive)
.'(a+1)**0.0' not implemented for float32 (scalar=1, opt=aggressive)
......'a + ones_like(b+c)' not implemented for float32 (scalar=1, opt=moderate)
................................................................................................................'(a+1)**0'
not implemented for float32 (scalar=1, opt=moderate)
.'(a+1)**0.0' not implemented for float32 (scalar=1, opt=moderate)
......'a + ones_like(b+c)' not implemented for float32 (scalar=1, opt=none)
................................................................................................................'(a+1)**0'
not implemented for float32 (scalar=1, opt=none)
.'(a+1)**0.0' not implemented for float32 (scalar=1, opt=none)
......'a + ones_like(b+c)' not implemented for float64 (scalar=1,
opt=aggressive)
................................................................................................................'(a+1)**0'
not implemented for float64 (scalar=1, opt=aggressive)
.'(a+1)**0.0' not implemented for float64 (scalar=1, opt=aggressive)
......'a + ones_like(b+c)' not implemented for float64 (scalar=1, opt=moderate)
................................................................................................................'(a+1)**0'
not implemented for float64 (scalar=1, opt=moderate)
.'(a+1)**0.0' not implemented for float64 (scalar=1, opt=moderate)
......'a + ones_like(b+c)' not implemented for float64 (scalar=1, opt=none)
................................................................................................................'(a+1)**0'
not implemented for float64 (scalar=1, opt=none)
.'(a+1)**0.0' not implemented for float64 (scalar=1, opt=none)
......'a + ones_like(b+c)' not implemented for int (scalar=1, opt=aggressive)
...............................................................................................................'(a+1)**0'
not implemented for int (scalar=1, opt=aggressive)
.'(a+1)**0.0' not implemented for int (scalar=1, opt=aggressive)
......'a + ones_like(b+c)' not implemented for int (scalar=1, opt=moderate)
...............................................................................................................'(a+1)**0'
not implemented for int (scalar=1, opt=moderate)
.'(a+1)**0.0' not implemented for int (scalar=1, opt=moderate)
......'a + ones_like(b+c)' not implemented for int (scalar=1, opt=none)
...............................................................................................................'(a+1)**0'
not implemented for int (scalar=1, opt=none)
.'(a+1)**0.0' not implemented for int (scalar=1, opt=none)
......'a + ones_like(b+c)' not implemented for long (scalar=1, opt=aggressive)
...............................................................................................................'(a+1)**0'
not implemented for long (scalar=1, opt=aggressive)
.'(a+1)**0.0' not implemented for long (scalar=1, opt=aggressive)
......'a + ones_like(b+c)' not implemented for long (scalar=1, opt=moderate)
...............................................................................................................'(a+1)**0'
not implemented for long (scalar=1, opt=moderate)
.'(a+1)**0.0' not implemented for long (scalar=1, opt=moderate)
......'a + ones_like(b+c)' not implemented for long (scalar=1, opt=none)
...............................................................................................................'(a+1)**0'
not implemented for long (scalar=1, opt=none)
.'(a+1)**0.0' not implemented for long (scalar=1, opt=none)
......'a + ones_like(b+c)' not implemented for complex (scalar=2,
opt=aggressive)
................................................................................'(a+1)**0'
not implemented for complex (scalar=2, opt=aggressive)
.'(a+1)**0.0' not implemented for complex (scalar=2, opt=aggressive)
......'a + ones_like(b+c)' not implemented for complex (scalar=2, opt=moderate)
................................................................................'(a+1)**0'
not implemented for complex (scalar=2, opt=moderate)
.'(a+1)**0.0' not implemented for complex (scalar=2, opt=moderate)
......'a + ones_like(b+c)' not implemented for complex (scalar=2, opt=none)
................................................................................'(a+1)**0'
not implemented for complex (scalar=2, opt=none)
.'(a+1)**0.0' not implemented for complex (scalar=2, opt=none)
......'a + ones_like(b+c)' not implemented for float32 (scalar=2,
opt=aggressive)
................................................................................................................'(a+1)**0'
not implemented for float32 (scalar=2, opt=aggressive)
.'(a+1)**0.0' not implemented for float32 (scalar=2, opt=aggressive)
......'a + ones_like(b+c)' not implemented for float32 (scalar=2, opt=moderate)
................................................................................................................'(a+1)**0'
not implemented for float32 (scalar=2, opt=moderate)
.'(a+1)**0.0' not implemented for float32 (scalar=2, opt=moderate)
......'a + ones_like(b+c)' not implemented for float32 (scalar=2, opt=none)
................................................................................................................'(a+1)**0'
not implemented for float32 (scalar=2, opt=none)
.'(a+1)**0.0' not implemented for float32 (scalar=2, opt=none)
......'a + ones_like(b+c)' not implemented for float64 (scalar=2,
opt=aggressive)
................................................................................................................'(a+1)**0'
not implemented for float64 (scalar=2, opt=aggressive)
.'(a+1)**0.0' not implemented for float64 (scalar=2, opt=aggressive)
......'a + ones_like(b+c)' not implemented for float64 (scalar=2, opt=moderate)
................................................................................................................'(a+1)**0'
not implemented for float64 (scalar=2, opt=moderate)
.'(a+1)**0.0' not implemented for float64 (scalar=2, opt=moderate)
......'a + ones_like(b+c)' not implemented for float64 (scalar=2, opt=none)
................................................................................................................'(a+1)**0'
not implemented for float64 (scalar=2, opt=none)
.'(a+1)**0.0' not implemented for float64 (scalar=2, opt=none)
......'a + ones_like(b+c)' not implemented for int (scalar=2, opt=aggressive)
...............................................................................................................'(a+1)**0'
not implemented for int (scalar=2, opt=aggressive)
.'(a+1)**0.0' not implemented for int (scalar=2, opt=aggressive)
......'a + ones_like(b+c)' not implemented for int (scalar=2, opt=moderate)
...............................................................................................................'(a+1)**0'
not implemented for int (scalar=2, opt=moderate)
.'(a+1)**0.0' not implemented for int (scalar=2, opt=moderate)
......'a + ones_like(b+c)' not implemented for int (scalar=2, opt=none)
...............................................................................................................'(a+1)**0'
not implemented for int (scalar=2, opt=none)
.'(a+1)**0.0' not implemented for int (scalar=2, opt=none)
......'a + ones_like(b+c)' not implemented for long (scalar=2, opt=aggressive)
...............................................................................................................'(a+1)**0'
not implemented for long (scalar=2, opt=aggressive)
.'(a+1)**0.0' not implemented for long (scalar=2, opt=aggressive)
......'a + ones_like(b+c)' not implemented for long (scalar=2, opt=moderate)
...............................................................................................................'(a+1)**0'
not implemented for long (scalar=2, opt=moderate)
.'(a+1)**0.0' not implemented for long (scalar=2, opt=moderate)
......'a + ones_like(b+c)' not implemented for long (scalar=2, opt=none)
...............................................................................................................'(a+1)**0'
not implemented for long (scalar=2, opt=none)
.'(a+1)**0.0' not implemented for long (scalar=2, opt=none)
.......................
======================================================================
ERROR: test_boolean_operator (__main__.test_evaluate)
----------------------------------------------------------------------
Traceback (most recent call last):
File "numexpr/tests/test_numexpr.py", line 184, in test_boolean_operator
raise ValueError("should raise exception!")
ValueError: should raise exception!

======================================================================
ERROR: test_true_div (__main__.test_evaluate)
----------------------------------------------------------------------
Traceback (most recent call last):
File "numexpr/tests/test_numexpr.py", line 173, in test_true_div
assert_array_equal(evaluate("x/2", truediv=False), x / 2)
File "/home/faltet/vepython/lib/python2.6/site-packages/numexpr-1.4.3.dev-py2.6-linux-x86_64.egg/numexpr/necompiler.py",
line 624, in evaluate
context = getContext(kwargs)
File "/home/faltet/vepython/lib/python2.6/site-packages/numexpr-1.4.3.dev-py2.6-linux-x86_64.egg/numexpr/necompiler.py",
line 436, in getContext
raise ValueError("Unknown keyword argument '%s'" % map.popitem()[0])
ValueError: Unknown keyword argument 'truediv'

----------------------------------------------------------------------
Ran 5102 tests in 4.709s

FAILED (errors=2)

--
Francesc Alted

[Gaëtan de Menten]

Hmmmm. Sorry for the obvious question, but... Are you sure you ran the
latest source? It looks like you ran the latest test suite on an older
version.

-G.

[Francesc Alted]

2011/10/31 Gaëtan de Menten <gdem...@gmail.com>:

Sure, it was my fault indeed. I don't know exactly why, but I cannot
override an installed version of numexpr by just specifying the
PYTHONPATH, and also need to remove completely the numexpr installed
in `site-packages`. That's weird, because, of all the packages that I
use to develop, this only happens with numexpr :-/

By the way, I'm heating engines for doing a new release with the new
virtual machine based in NumPy's new iterator introduced in NumPy 1.6.
After the recent fixes by Mark, it seems that everything runs
smoothly now. My plan is to get a 2.0rc1 today, and 2.0 in a few
days. Do you agree with this, or you would like to include more work
in there?

--
Francesc Alted

[Gaëtan de Menten]

On Tue, 2011-11-01 at 11:59 +0100, Francesc Alted wrote:

> By the way, I'm heating engines for doing a new release with the new
> virtual machine based in NumPy's new iterator introduced in NumPy 1.6.
> After the recent fixes by Mark, it seems that everything runs
> smoothly now. My plan is to get a 2.0rc1 today, and 2.0 in a few
> days. Do you agree with this, or you would like to include more work
> in there?

I would have liked to:
1) implement issue #23
2) fix where() with a constant as the condition (ie part of issue #61).
I *think* it's only a matter of removing lines 174-176, but I'm not 100%
sure and I didn't have time to test it (hence I didn't remove them
directly).
3) fix the different benches. IIRC, a few failed to run correctly (I
don't know whether this is specific to the "newiter" branch code, nor if
any of the recent changes fixed them -- I doubt it though).

Unfortunately, I probably won't have enough time to do anything on
numexpr before December so I guess these are not worth delaying the
release if you are ready to do it.

As for the release notes, the only significant speed enhancement I made
was the one for very large expressions, for which the compilation phase
could be very slow due to the O(n^2) nature of
optimizeTempoariesAllocation. The better hash patch was already included
in numexpr 1.4.2.

Also, you might (or might not -- not sure if it was a deliberate
omission and I don't care much if you don't) want to mention something
about "code cleanup" and "the test suite being fixed" to the release
notes.

-Gaï¿œtan

[Francesc Alted]

On 1 Nov, 21:59, Gaëtan de Menten <gdemen...@gmail.com> wrote:
> On Tue, 2011-11-01 at 11:59 +0100, Francesc Alted wrote:
> > By the way, I'm heating engines for doing a new release with the new
> > virtual machine based in NumPy's new iterator introduced in NumPy 1.6.
> >  After the recent fixes by Mark, it seems that everything runs
> > smoothly now.  My plan is to get a 2.0rc1 today, and 2.0 in a few
> > days.  Do you agree with this, or you would like to include more work
> > in there?
>
> I would have liked to:
> 1) implement issue #23

I've had a look at this, and although that should be relatively easy
to do, I think this is better to be implemented with a new release
(2.1?).

> 2) fix where() with a constant as the condition (ie part of issue #61).
> I *think* it's only a matter of removing lines 174-176, but I'm not 100%
> sure and I didn't have time to test it (hence I didn't remove them
> directly).

That's is not so easy. Booleans are treated pretty differently than
integers in numexpr (they are a completely separated type), so fixing
this would be be easy (even not desirable?).

> 3) fix the different benches. IIRC, a few failed to run correctly (I
> don't know whether this is specific to the "newiter" branch code, nor if
> any of the recent changes fixed them -- I doubt it though).

I think I have fixed all benches for 2.0rc1 (please report if you find
some unsolved glitch yet).

> Unfortunately, I probably won't have enough time to do anything on
> numexpr before December so I guess these are not worth delaying the
> release if you are ready to do it.

Ok. Will try to release 2.0 by today or tomorrow.

> As for the release notes, the only significant speed enhancement I made
> was the one for very large expressions, for which the compilation phase
> could be very slow due to the O(n^2) nature of
> optimizeTempoariesAllocation.

It should be faster in other scenarios too:

For broadcasting:

>>> a = np.arange(1e3)
>>> b = np.arange(1e6).reshape(1e3, 1e3)

>>> timeit ne.evaluate("a*(b+1)") # 1.4.2
100 loops, best of 3: 16.4 ms per loop

>>> timeit ne.evaluate("a*(b+1)") # 2.0
100 loops, best of 3: 5.2 ms per loop

Operating with non-native dtypes:

>>> a = np.arange(1e6, dtype=">f8")
>>> b = np.arange(1e6, dtype=">f8")

>>> timeit ne.evaluate("a*(b+1)") # 1.4.2
100 loops, best of 3: 17.2 ms per loop

>>> timeit ne.evaluate("a*(b+1)") # 2.0
100 loops, best of 3: 6.32 ms per loop

With fortran-ordered arrays:

>>> a = np.arange(1e6).reshape(1e3, 1e3).copy('F')
>>> b = np.arange(1e6).reshape(1e3, 1e3).copy('F')

>>> timeit ne.evaluate("a*(b+1)") # 1.4.2
10 loops, best of 3: 32.8 ms per loop

>>> timeit ne.evaluate("a*(b+1)") # 2.0
100 loops, best of 3: 5.62 ms per loop

With a mix of 'non-native' arrays, and using broadcasting:

>>> a = np.arange(1e3, dtype='>f8').copy('F')
>>> b = np.arange(1e6, dtype='>f8').reshape(1e3, 1e3).copy('F')

>>> timeit ne.evaluate("a*(b+1)") # 1.4.2
10 loops, best of 3: 21.2 ms per loop

>>> timeit ne.evaluate("a*(b+1)") # 2.0
100 loops, best of 3: 5.22 ms per loop

So, yeah, the improvement is notable in many areas.

> The better hash patch was already included
> in numexpr 1.4.2.

Oops! Will remove that from release notes.

> Also, you might (or might not -- not sure if it was a deliberate
> omission and I don't care much if you don't) want to mention something
> about "code cleanup" and "the test suite being fixed" to the release
> notes.

Okay, I wouldn't want to miss anything important. Could you develop
more on this please?

Francesc

[Francesc Alted]

2011/11/4 Francesc Alted <fal...@gmail.com>:

> Ok.  Will try to release 2.0 by today or tomorrow.

Unfortunately, I've found a blocking issue for 2.0:

http://code.google.com/p/numexpr/issues/detail?id=64

I'm a bit puzzled on how to solve this. Mark?

--
Francesc Alted

[Gaëtan de Menten]

On Fri, 2011-11-04 at 05:20 -0700, Francesc Alted wrote:

> On 1 Nov, 21:59, Gaï¿œtan de Menten <gdemen...@gmail.com> wrote:

> > 2) fix where() with a constant as the condition (ie part of issue #61).
> > I *think* it's only a matter of removing lines 174-176, but I'm not 100%
> > sure and I didn't have time to test it (hence I didn't remove them
> > directly).
>
> That's is not so easy. Booleans are treated pretty differently than
> integers in numexpr (they are a completely separated type), so fixing
> this would be be easy (even not desirable?).

I didn't mean to fix issue #61 completely but rather, only allow a
constant as the condition.

> > Also, you might (or might not -- not sure if it was a deliberate
> > omission and I don't care much if you don't) want to mention something
> > about "code cleanup" and "the test suite being fixed" to the release
> > notes.
>
> Okay, I wouldn't want to miss anything important. Could you develop
> more on this please?

The relevant commit message:

Fixed the test suite: most of the TestExpressions tests were not run !!!
in fact, the first test was run over and over again, making it look like
all tests were executed and were successful.

Another possibly relevant change:

I fixed the ones_like function (though it is not documented in the
overview page -- I'm not sure why).

Gaï¿œtan