infs and Nans

[Rahul Dave]

This is likely to be because of some deep misunderstanding on my part, so please pardon me if this is a very obvious question!

I have

import numpy as np

import statsmodels.

y=np.array([-11., -12.,  -8., -13., -25. , -9. ,-10. , -6. , -7. ,-17., -12.,  -9. , -9.,  -5.,  -7.

, -35., -14., -13. , -9., -15., -16., -29., -17. , -5., -14.,  -9. ,-14. ,-12.])

X=np.array([1289520000000000000, 1296086400000000000, 1308355200000000000,

 1317427200000000000, 1320883200000000000 ,1321747200000000000,

 1322524800000000000, 1327795200000000000, 1329868800000000000,

 1330905600000000000 ,1332547200000000000 ,1332806400000000000,

 1333324800000000000, 1335571200000000000 ,1335916800000000000,

 1336521600000000000 ,1337126400000000000 ,1338854400000000000,

 1339977600000000000, 1341964800000000000, 1343779200000000000,

 1344124800000000000, 1345420800000000000, 1347321600000000000,

 1347408000000000000, 1347926400000000000, 1348012800000000000,

 1348185600000000000])

lr=sm.nonparametric.lowess(y, X,frac=.1, it=3) tells me:

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-238-9d3b8fa136ed> in <module>()

----> 1 lr=sm.nonparametric.lowess(y, X,frac=.1, it=3)

/Users/rahul/anaconda/lib/python2.7/site-packages/statsmodels/nonparametric/smoothers_lowess.pyc in lowess(endog, exog, frac, it)

    124     for i in xrange(it):

    125         _lowess_robustify_fit(x_copy, y_copy, fitted,

--> 126                                             weights, k, n)

    127 

    128     out = np.array([x_copy, fitted]).T

/Users/rahul/anaconda/lib/python2.7/site-packages/statsmodels/nonparametric/smoothers_lowess.pyc in _lowess_robustify_fit(x_copy, y_copy, fitted, weights, k, n)

    267         total_weights.shape = (k,1)

    268 

--> 269         beta = lstsq(total_weights * X, y_i)[0]

    270 

    271         fitted[i] = beta[0] + beta[1] * x_copy[i]

/Users/rahul/anaconda/lib/python2.7/site-packages/scipy/linalg/basic.pyc in lstsq(a, b, cond, overwrite_a, overwrite_b, check_finite)

    501 

    502     if check_finite:

--> 503         a1,b1 = map(np.asarray_chkfinite, (a,b))

    504     else:

    505         a1,b1 = map(np.asarray, (a,b))

/Users/rahul/anaconda/lib/python2.7/site-packages/numpy/lib/function_base.pyc in asarray_chkfinite(a, dtype, order)

    588     if a.dtype.char in typecodes['AllFloat'] and not np.isfinite(a).all():

    589         raise ValueError(

--> 590                 "array must not contain infs or NaNs")

    591     return a

    592 

ValueError: array must not contain infs or NaNs

Is there some way to get a smoothing which does not blow up? I'm guessing somehow that the local linear regressions are getting some singularity...

Rahul

[josef...@gmail.com]

Is this with statsmodels 0.4.3? lowess has been replaced by a cython
implementation.

I don't have a 0.4.3 version available right now.

What you could try:

convert X to float X.astype(float),
take away the zeros, rescale X / 100000000000.

From the traceback I cannot see where we might run into overflow
problems, or integer problems.


statsmodels 0.5.0rc1 should be out any day now, or you could upgrade
from current master.

I will check if your example works with the new version.

Josef
>
> Rahul

[josef...@gmail.com]

same problem with new version, rescaling doesn't help

Can you open an issue on github?
I have no idea right now what might be causing this.

Josef


>
> Josef
>>
>> Rahul

[josef...@gmail.com]

frac=0.2 works

the sample size with frac=0.1 in the local neighborhood seems to be
too small. I don't know what checks are in there for a minimal valid
local regression.

Josef


>
> Josef
>
>
>>
>> Josef
>>>
>>> Rahul