[SciPy-user] Pb with numpy.histogram

[LB]

Hi,

I've got strange results with numpy.histogram :

Here is its doc strings :
"""
Help on function histogram in module numpy.lib.function_base:

histogram(a, bins=10, range=None, normed=False)
Compute the histogram from a set of data.

:Parameters:
- `a` : array
The data to histogram. n-D arrays will be flattened.
- `bins` : int or sequence of floats, optional
If an int, then the number of equal-width bins in the given
range.
Otherwise, a sequence of the lower bound of each bin.
- `range` : (float, float), optional
The lower and upper range of the bins. If not provided, then
(a.min(),
a.max()) is used. Values outside of this range are allocated
to the
closest bin.
- `normed` : bool, optional
If False, the result array will contain the number of samples
in each bin.
If True, the result array is the value of the probability
*density*
function at the bin normalized such that the *integral* over
the range
is 1. Note that the sum of all of the histogram values will
not usually
be 1; it is not a probability *mass* function.

:Returns:
- `hist` : array (n,)
The values of the histogram. See `normed` for a description of
the
possible semantics.
- `lower_edges` : float array (n,)
The lower edges of each bin.
"""

and here is a snipplet of code :
>>> r = random.normal(8, 2, 500)
>>> r.min(), r.max()
(1.164117097856284, 13.069426390055149)
>>> ra
(3, 12)
>>> pdf, xpdf = histogram(r, nbins, range=ra, normed=False)
>>> pdf
array([ 1, 6, 5, 8, 30, 39, 53, 55, 61, 50, 45, 42, 32, 26, 17,
27])
>>> pdf.sum()
497

It seems I've lost 3 of my 500 random numbers !

>>> r[ r>= ra[1]]
array([ 12.00676288, 12.8381615 , 12.48380931, 12.55392835,
12.26153469, 12.92869504, 12.58290343, 12.03782311,
13.06942639, 12.06375346, 12.02970414, 12.53556779,
12.54203654, 12.02611864, 12.85113934, 12.64692817])

>>> r[ r<= ra[0]]
array([ 1.1641171 , 2.85873306, 2.92046745])

So this number match the number of experiments below the range given
to histogram.
This smells like a bug to me.
Is there something I've misunderstood in the utilisation of
numpy.histogram ?

For information
>>> numpy.__version__
'1.0.2'

Regards,

--
LB

_______________________________________________
SciPy-user mailing list
SciPy...@scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-user

[David Huard]

Hi LB,

I think histogram has had this weird behavior since the numeric era and a lot of code may break if we fix it. Basically, histogram discards the lower than range values as outliers but puts the higher than range values into the last bin.

I'm generally using my own histograming routines, I could send them your way if you're interested.

David

2007/9/27, LB < berth...@gmail.com>:

   Hi,

I've got strange results with numpy.histogram :

Here is its doc strings :
"""
Help on function histogram in module numpy.lib.function_base:

histogram(a, bins=10, range=None, normed=False)
    Compute the histogram from a set of data.

    :Parameters:
      - `a` : array
        The data to histogram. n-D arrays will be flattened.
      - `bins` : int or sequence of floats, optional
        If an int, then the number of equal-width bins in the given
range.
        Otherwise, a sequence of the lower bound of each bin.
      - `range` : (float, float), optional
        The lower and upper range of the bins. If not provided, then
(a.min(),

        a.max ()) is used. Values outside of this range are allocated

        12.54203654 ,  12.02611864,  12.85113934,  12.64692817])

[Alan G Isaac]

On Thu, 27 Sep 2007, David Huard apparently wrote:
> I'm generally using my own histograming routines, I could
> send them your way if you're interested.

If they share the SciPy license, please consider posting
them or adding them to your sandbox directory.

Cheers,
Alan Isaac

[David Huard]

2007/9/27, Alan G Isaac <ais...@american.edu>:

On Thu, 27 Sep 2007, David Huard apparently wrote:
> I'm generally using my own histograming routines, I could
> send them your way if you're interested.

If they share the SciPy license, please consider posting
them or adding them to your sandbox directory.

Alan,

You'll find the histogram.py module in scipy/sandbox/dhuard/

Cheers,

David

[Alan G Isaac]

On Thu, 27 Sep 2007, David Huard apparently wrote:
> You'll find the histogram.py module in scipy/sandbox/dhuard/

Thanks!
I notice you do not specify a license.
Is everything in the sandbox considered to be
under the SciPy license?

[Robert Kern]

Alan G Isaac wrote:
> On Thu, 27 Sep 2007, David Huard apparently wrote:
>> You'll find the histogram.py module in scipy/sandbox/dhuard/
>
> Thanks!
> I notice you do not specify a license.
> Is everything in the sandbox considered to be
> under the SciPy license?

As a matter of policy, no one should check something in there that isn't.
Assuming mistakes aren't made and committers understand what they are doing,
then yes, everything there is under the SciPy license.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

[Alan G Isaac]

>> Is everything in the sandbox considered to be
>> under the SciPy license?

On Thu, 27 Sep 2007, Robert Kern apparently wrote:
> As a matter of policy, no one should check something in there that isn't.
> Assuming mistakes aren't made and committers understand what they are doing,
> then yes, everything there is under the SciPy license.

Thanks!
Alan

[LB]

> I think histogram has had this weird behavior since the numeric era and a
> lot of code may break if we fix it. Basically, histogram discards the lower
> than range values as outliers but puts the higher than range values into the
> last bin.

I think this should be clearly explained in the doc string. The
current doc string says

"Values outside of this range are allocated to the closest bin".

This is wrong, can leads to bug and should be fixed.

numpy.histogram's behavior seems still weirds to me, and I don't see
why values
lower than range should always be discarded as outliers.
If the real problem is cosistency with older versions from the numeric
era,
what about adding a new keyword to the function, says "discard", which
could be
used to decide what to do with values outside the range :
- 'low' => values lower than the range are discarded, values
higher are added to the last bin
- 'up' => values higher than the range are discarded, values lower
are added to the first bin
- 'out' => values out of the range are discarded
- None => values outside of this range are allocated to the closest
bin

For compatibility reason, a default value of 'low' could be used.

> I'm generally using my own histograming routines, I could send them your way
> if you're interested.

Thanks, I will check the code you've put in the sandbow at home.

[massimo sandal]

David Huard ha scritto:

> Hi LB,
>
> I think histogram has had this weird behavior since the numeric era and
> a lot of code may break if we fix it.
> Basically, histogram discards the
> lower than range values as outliers but puts the higher than range
> values into the last bin.
>
> I'm generally using my own histograming routines, I could send them your
> way if you're interested.

I'm not a SciPy developer so my weight on the discussion is next to
zero, but I personally think that other people code breaking is not a
good reason to keep a buggy function.

We don't want code that relies on bugs like they were features: we're
not Microsoft, aren't we? ;)

People should be informed before updating, of course, and maybe an
oldhistogram() function could be maintained to allow for easy updating,
but if histogram() behavour is so flawed that people is forced to
rewrite their own routines, please fix it.

m.

--
Massimo Sandal
University of Bologna
Department of Biochemistry "G.Moruzzi"

snail mail:
Via Irnerio 48, 40126 Bologna, Italy

email:
massimo...@unibo.it

tel: +39-051-2094388
fax: +39-051-2094387

[David Huard]

2007/10/1, LB <berth...@gmail.com>:

> I think histogram has had this weird behavior since the numeric era and a
> lot of code may break if we fix it. Basically, histogram discards the lower
> than range values as outliers but puts the higher than range values into the
> last bin.
I think this should be clearly explained in the doc string. The
current doc string says
"Values outside of this range are allocated to the closest bin".
This is wrong, can leads to bug and should be fixed.

You're right. In fact, it said so at some point but it seems it has been edited out.
 

numpy.histogram's behavior seems still weirds to me, and I don't see
why values
lower than range should always be discarded as outliers.
If the real problem is cosistency with older versions from the numeric
era,
what about adding a new keyword to the function, says "discard", which
could be
used to decide what to do with values outside the range :
   - 'low'      => values lower than the range are discarded, values
higher are added to the last bin
   - 'up'       => values higher than the range are discarded, values lower
are added to the first bin
   - 'out'      => values out of the range are discarded
   - None       => values outside of this range are allocated to the closest
bin

For compatibility reason, a default value of 'low' could be used.

Good idea. Better yet would be to raise a deprecation warning and change the function in the next or second next release, and ideally replace it with something written in C to speed things up. The final decision is up to someone else than me, though.

Cheers,

David