4.1 Release [Was: Re: [pystatsmodels] charlton - statsmodels ?]

[Skipper Seabold]

On Sat, Jun 9, 2012 at 12:41 PM, Yaroslav Halchenko
<yarik...@gmail.com> wrote:
>
> On Sat, 09 Jun 2012, Skipper Seabold wrote:
>> > What are the plans with charlton statsmodels integration?
>> I'd like to merge formula-integration in ASAP.
>
> sorry guys -- what has happened with the release plans?  is any one
> coming?

I had a short window to do this, and it didn't happen. I'm more or
less available again, but I don't know the status of master /
maintenance anymore. What do things look like?

Skipper

[josef...@gmail.com]

I have 3 open bugfix PRs and 1 move/cleanup PR, issue #320 should get
a solution.
(I have a few more branches, Stata, R helper files, and maybe some minor stuff,
new gof_again needs a bit more work)

I have no clue for the remaining Debian errors, and cannot reproduce
them in my setup.

I started to update the CHANGES but not committed yet.

Josef

>
> Skipper

[Yaroslav Halchenko]

let me know when/what treeish should I check and then I would make sure
you could reproduce any remaining issue ;)

Thanks in advance!

On Sat, 09 Jun 2012, josef...@gmail.com wrote:
> > I had a short window to do this, and it didn't happen. I'm more or
> > less available again, but I don't know the status of master /
> > maintenance anymore. What do things look like?

> I have 3 open bugfix PRs and 1 move/cleanup PR, issue #320 should get
> a solution.
> (I have a few more branches, Stata, R helper files, and maybe some minor stuff,
> new gof_again needs a bit more work)

> I have no clue for the remaining Debian errors, and cannot reproduce
> them in my setup.

> I started to update the CHANGES but not committed yet.

> Josef


> > Skipper

--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik

[josef...@gmail.com]

On Sun, Jun 10, 2012 at 10:46 PM, Yaroslav Halchenko
<yarik...@gmail.com> wrote:
> let me know when/what treeish should I check and then I would make sure
> you could reproduce any remaining issue ;)
>
> Thanks in advance!

I think I will do some merging tomorrow.

scipy weekend is over, our network is getting busier and is asking for
review, so better get 0.4.1 out of the way.

Josef

[Skipper Seabold]

On Sun, Jun 10, 2012 at 10:53 PM, <josef...@gmail.com> wrote:
> On Sun, Jun 10, 2012 at 10:46 PM, Yaroslav Halchenko
> <yarik...@gmail.com> wrote:
>> let me know when/what treeish should I check and then I would make sure
>> you could reproduce any remaining issue ;)
>>
>> Thanks in advance!
>
> I think I will do some merging tomorrow.
>
> scipy weekend is over, our network is getting busier and is asking for
> review, so better get 0.4.1 out of the way.
>

Let me know. I'd like to do this asap as well. I have code I'm ready
to get back into master (mainly for the pyschological burden of
knowing "finished," tested code is out in a cold, lonely branch) but
now that we don't currently have separation between maintenance and
development I have to wait :).

Skipper

[josef...@gmail.com]

On Tue, Jun 12, 2012 at 9:20 AM, Skipper Seabold <jsse...@gmail.com> wrote:
> On Sun, Jun 10, 2012 at 10:53 PM,  <josef...@gmail.com> wrote:
>> On Sun, Jun 10, 2012 at 10:46 PM, Yaroslav Halchenko
>> <yarik...@gmail.com> wrote:
>>> let me know when/what treeish should I check and then I would make sure
>>> you could reproduce any remaining issue ;)
>>>
>>> Thanks in advance!
>>
>> I think I will do some merging tomorrow.

One day late.
I merged 7 branches, master tests clean on my (medium old) working
setup, but I didn't run the compatibility tests yet.

a doc bug that I saw yesterday is missing,
https://github.com/statsmodels/statsmodels/issues/305,
and the release related stuff.
I will update CHANGES, but might not have time until tonight.

rebase - force push - green button is fun
(as long as there are no merge conflicts or newly found test failures)

>>
>> scipy weekend is over, our network is getting busier and is asking for
>> review, so better get 0.4.1 out of the way.
>>
>
> Let me know. I'd like to do this asap as well. I have code I'm ready
> to get back into master (mainly for the pyschological burden of
> knowing "finished," tested code is out in a cold, lonely branch) but
> now that we don't currently have separation between maintenance and
> development I have to wait :).

I don't see any pull requests. :)
The only PR waiting for 0.4.1 to get out of the way that I see, is
lowess in cython.

Josef

>
> Skipper

[Skipper Seabold]

On Tue, Jun 12, 2012 at 1:18 PM, <josef...@gmail.com> wrote:
> On Tue, Jun 12, 2012 at 9:20 AM, Skipper Seabold <jsse...@gmail.com> wrote:
>> On Sun, Jun 10, 2012 at 10:53 PM,  <josef...@gmail.com> wrote:
>>> On Sun, Jun 10, 2012 at 10:46 PM, Yaroslav Halchenko
>>> <yarik...@gmail.com> wrote:
>>>> let me know when/what treeish should I check and then I would make sure
>>>> you could reproduce any remaining issue ;)
>>>>
>>>> Thanks in advance!
>>>
>>> I think I will do some merging tomorrow.
>
> One day late.
> I merged 7 branches, master tests clean on my (medium old) working
> setup, but I didn't run the compatibility tests yet.
>

Great thanks.

> a doc bug that I saw yesterday is missing,
> https://github.com/statsmodels/statsmodels/issues/305,
> and the release related stuff.
> I will update CHANGES, but might not have time until tonight.
>
> rebase - force push - green button is fun
> (as long as there are no merge conflicts or newly found test failures)
>
>>>
>>> scipy weekend is over, our network is getting busier and is asking for
>>> review, so better get 0.4.1 out of the way.
>>>
>>
>> Let me know. I'd like to do this asap as well. I have code I'm ready
>> to get back into master (mainly for the pyschological burden of
>> knowing "finished," tested code is out in a cold, lonely branch) but
>> now that we don't currently have separation between maintenance and
>> development I have to wait :).
>
> I don't see any pull requests. :)

Branches are sitting in my fork. These probably aren't appropriate for
a minor revision release and might require some discussion. Don't want
to hold things up.

> The only PR waiting for 0.4.1 to get out of the way that I see, is
> lowess in cython.
>

Does this need a look? I'm not that familiar with this code yet.

Skipper

[Yaroslav Halchenko]

On Tue, 12 Jun 2012, Skipper Seabold wrote:

> > The only PR waiting for 0.4.1 to get out of the way that I see, is
> > lowess in cython.

> Does this need a look? I'm not that familiar with this code yet.

am I reading it right that current master is ready for the final
spin before the release? ;)

[josef...@gmail.com]

On Tue, Jun 12, 2012 at 1:30 PM, Skipper Seabold <jsse...@gmail.com> wrote:
> On Tue, Jun 12, 2012 at 1:18 PM,  <josef...@gmail.com> wrote:
>> On Tue, Jun 12, 2012 at 9:20 AM, Skipper Seabold <jsse...@gmail.com> wrote:
>>> On Sun, Jun 10, 2012 at 10:53 PM,  <josef...@gmail.com> wrote:
>>>> On Sun, Jun 10, 2012 at 10:46 PM, Yaroslav Halchenko
>>>> <yarik...@gmail.com> wrote:
>>>>> let me know when/what treeish should I check and then I would make sure
>>>>> you could reproduce any remaining issue ;)
>>>>>
>>>>> Thanks in advance!
>>>>
>>>> I think I will do some merging tomorrow.
>>
>> One day late.
>> I merged 7 branches, master tests clean on my (medium old) working
>> setup, but I didn't run the compatibility tests yet.
>>
>
> Great thanks.

I just saw https://github.com/statsmodels/statsmodels/issues/302
again, you mentioned you have the fix.

>
>> a doc bug that I saw yesterday is missing,
>> https://github.com/statsmodels/statsmodels/issues/305,
>> and the release related stuff.
>> I will update CHANGES, but might not have time until tonight.
>>
>> rebase - force push - green button is fun
>> (as long as there are no merge conflicts or newly found test failures)
>>
>>>>
>>>> scipy weekend is over, our network is getting busier and is asking for
>>>> review, so better get 0.4.1 out of the way.
>>>>
>>>
>>> Let me know. I'd like to do this asap as well. I have code I'm ready
>>> to get back into master (mainly for the pyschological burden of
>>> knowing "finished," tested code is out in a cold, lonely branch) but
>>> now that we don't currently have separation between maintenance and
>>> development I have to wait :).
>>
>> I don't see any pull requests. :)
>
> Branches are sitting in my fork. These probably aren't appropriate for
> a minor revision release and might require some discussion. Don't want
> to hold things up.

You can always open a PR so it's easier to keep track what's on the
waiting list.
I thought it will be the main changes for 0.5

>
>> The only PR waiting for 0.4.1 to get out of the way that I see, is
>> lowess in cython.
>>
>
> Does this need a look? I'm not that familiar with this code yet.

Also for 0.5, not quite backwards compatible and might need exposure
through an rc to make sure the cython doesn't create problems on one
of the platforms/versions.
test coverage is good, but it needs the infrastructure to be build
inside statsmodels, as your cython code.
Your help for this would be good, because you know this part.

Josef

>
> Skipper

[josef...@gmail.com]

On Tue, Jun 12, 2012 at 1:36 PM, Yaroslav Halchenko
<yarik...@gmail.com> wrote:
>
> On Tue, 12 Jun 2012, Skipper Seabold wrote:
>
>> > The only PR waiting for 0.4.1 to get out of the way that I see, is
>> > lowess in cython.
>
>> Does this need a look? I'm not that familiar with this code yet.
>
> am I reading it right that current master is ready for the final
> spin before the release? ;)

about ready, one small bugfix in categorical is missing.

If you want to possibly save some work, then you could wait until I
have checked from python 2.5 to python 3.2, currently I know it's
clean only for one version.

Josef

[Skipper Seabold]

On Tue, Jun 12, 2012 at 1:38 PM, <josef...@gmail.com> wrote:
> On Tue, Jun 12, 2012 at 1:30 PM, Skipper Seabold <jsse...@gmail.com> wrote:
>> On Tue, Jun 12, 2012 at 1:18 PM,  <josef...@gmail.com> wrote:
>>> On Tue, Jun 12, 2012 at 9:20 AM, Skipper Seabold <jsse...@gmail.com> wrote:
>>>> On Sun, Jun 10, 2012 at 10:53 PM,  <josef...@gmail.com> wrote:
>>>>> On Sun, Jun 10, 2012 at 10:46 PM, Yaroslav Halchenko
>>>>> <yarik...@gmail.com> wrote:
>>>>>> let me know when/what treeish should I check and then I would make sure
>>>>>> you could reproduce any remaining issue ;)
>>>>>>
>>>>>> Thanks in advance!
>>>>>
>>>>> I think I will do some merging tomorrow.
>>>
>>> One day late.
>>> I merged 7 branches, master tests clean on my (medium old) working
>>> setup, but I didn't run the compatibility tests yet.
>>>
>>
>> Great thanks.
>
> I just saw https://github.com/statsmodels/statsmodels/issues/302
> again, you mentioned you have the fix.

Ah, right. I'll do this now.

Ok I'll have a look.

>
> Josef
>
>>
>> Skipper

[Skipper Seabold]

On Tue, Jun 12, 2012 at 1:58 PM, Skipper Seabold <jsse...@gmail.com> wrote:
> On Tue, Jun 12, 2012 at 1:38 PM,  <josef...@gmail.com> wrote:
>> On Tue, Jun 12, 2012 at 1:30 PM, Skipper Seabold <jsse...@gmail.com> wrote:
>>> On Tue, Jun 12, 2012 at 1:18 PM,  <josef...@gmail.com> wrote:
>>>> On Tue, Jun 12, 2012 at 9:20 AM, Skipper Seabold <jsse...@gmail.com> wrote:
>>>>> On Sun, Jun 10, 2012 at 10:53 PM,  <josef...@gmail.com> wrote:
>>>>>> On Sun, Jun 10, 2012 at 10:46 PM, Yaroslav Halchenko
>>>>>> <yarik...@gmail.com> wrote:
>>>>>>> let me know when/what treeish should I check and then I would make sure
>>>>>>> you could reproduce any remaining issue ;)
>>>>>>>
>>>>>>> Thanks in advance!
>>>>>>
>>>>>> I think I will do some merging tomorrow.
>>>>
>>>> One day late.
>>>> I merged 7 branches, master tests clean on my (medium old) working
>>>> setup, but I didn't run the compatibility tests yet.
>>>>
>>>
>>> Great thanks.
>>
>> I just saw https://github.com/statsmodels/statsmodels/issues/302
>> again, you mentioned you have the fix.
>
> Ah, right. I'll do this now.

https://github.com/statsmodels/statsmodels/pull/311

[josef...@gmail.com]

On Tue, Jun 12, 2012 at 2:14 PM, Skipper Seabold <jsse...@gmail.com> wrote:
> On Tue, Jun 12, 2012 at 1:58 PM, Skipper Seabold <jsse...@gmail.com> wrote:
>> On Tue, Jun 12, 2012 at 1:38 PM,  <josef...@gmail.com> wrote:
>>> On Tue, Jun 12, 2012 at 1:30 PM, Skipper Seabold <jsse...@gmail.com> wrote:
>>>> On Tue, Jun 12, 2012 at 1:18 PM,  <josef...@gmail.com> wrote:
>>>>> On Tue, Jun 12, 2012 at 9:20 AM, Skipper Seabold <jsse...@gmail.com> wrote:
>>>>>> On Sun, Jun 10, 2012 at 10:53 PM,  <josef...@gmail.com> wrote:
>>>>>>> On Sun, Jun 10, 2012 at 10:46 PM, Yaroslav Halchenko
>>>>>>> <yarik...@gmail.com> wrote:
>>>>>>>> let me know when/what treeish should I check and then I would make sure
>>>>>>>> you could reproduce any remaining issue ;)
>>>>>>>>
>>>>>>>> Thanks in advance!
>>>>>>>
>>>>>>> I think I will do some merging tomorrow.
>>>>>
>>>>> One day late.
>>>>> I merged 7 branches, master tests clean on my (medium old) working
>>>>> setup, but I didn't run the compatibility tests yet.
>>>>>
>>>>
>>>> Great thanks.
>>>
>>> I just saw https://github.com/statsmodels/statsmodels/issues/302
>>> again, you mentioned you have the fix.
>>
>> Ah, right. I'll do this now.
>
> https://github.com/statsmodels/statsmodels/pull/311

I'll check tonight, since it's about strings there is some chance that
python 3.2 complains.

Otherwise, I extra failure on python 2.5 numpy 1.4, most likely too
high precision in test.
clean on py 2.6, 2.7, and 3.2

Josef

[josef...@gmail.com]

I fixed this one, but the last two merges don't look good, I guess the
categorical fix

these two show up in all versions. dtype mismatch i8 versus i4
(my python 2.5 2.6 2.7 are 32bit, python 3.2 is 64bit)

======================================================================
FAIL: statsmodels.tools.tests.test_tools.test_rec_issue302
----------------------------------------------------------------------
Traceback (most recent call last):
File "E:\Josef\testing\tox\py27a\lib\site-packages\nose\case.py",
line 197, in runTest
self.test(*self.arg)
File "E:\Josef\testing\tox\py27a\lib\site-packages\statsmodels\tools\tests\test_tools.py",
line 314, in test_rec_issue302
assert_array_equal(actual, expected)
File "E:\Josef\testing\tox\py27a\lib\site-packages\numpy\testing\utils.py",
line 707, in assert_array_equal
verbose=verbose, header='Arrays are not equal')
File "E:\Josef\testing\tox\py27a\lib\site-packages\numpy\testing\utils.py",
line 636, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Arrays are not equal

(mismatch 100.0%)
x: rec.array([(10, 1.0, 0.0), (11, 0.0, 1.0)],
dtype=[('group', '<i4'), ('group_10', '<f8'), ('group_11', '<f8')])
y: rec.array([(10L, 1.0, 0.0), (11L, 0.0, 1.0)],
dtype=[('group', '<i8'), ('group_10', '<f8'), ('group_11', '<f8')])

======================================================================
FAIL: statsmodels.tools.tests.test_tools.test_issue302
----------------------------------------------------------------------
Traceback (most recent call last):
File "E:\Josef\testing\tox\py27a\lib\site-packages\nose\case.py",
line 197, in runTest
self.test(*self.arg)
File "E:\Josef\testing\tox\py27a\lib\site-packages\statsmodels\tools\tests\test_tools.py",
line 322, in test_issue302
assert_array_equal(actual, expected)
File "E:\Josef\testing\tox\py27a\lib\site-packages\numpy\testing\utils.py",
line 707, in assert_array_equal
verbose=verbose, header='Arrays are not equal')
File "E:\Josef\testing\tox\py27a\lib\site-packages\numpy\testing\utils.py",
line 636, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Arrays are not equal

(mismatch 100.0%)
x: rec.array([(10, 12, 1.0, 0.0), (11, 13, 0.0, 1.0)],
dtype=[('group', '<i4'), ('whatever', '<i4'), ('group_10',
'<f8'), ('group_11', '<f8')])
y: rec.array([(10L, 12L, 1.0, 0.0), (11L, 13L, 0.0, 1.0)],
dtype=[('group', '<i8'), ('whatever', '<i8'), ('group_10',
'<f8'), ('group_11', '<f8')])



this one, I think, shows up every once in a while, only python 2.7

======================================================================
FAIL: statsmodels.tsa.tests.test_ar.TestARMLEConstant.test_predict
----------------------------------------------------------------------
Traceback (most recent call last):
File "E:\Josef\testing\tox\py27a\lib\site-packages\nose\case.py",
line 197, in runTest
self.test(*self.arg)
File "E:\Josef\testing\tox\py27a\lib\site-packages\statsmodels\tsa\tests\test_ar.py",
line 126, in
test_predict
DECIMAL_4)
File "E:\Josef\testing\tox\py27a\lib\site-packages\numpy\testing\utils.py",
line 452, in assert_almost_equal
return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "E:\Josef\testing\tox\py27a\lib\site-packages\numpy\testing\utils.py",
line 800, in assert_array_almost_equal
header=('Arrays are not almost equal to %d decimals' % decimal))
File "E:\Josef\testing\tox\py27a\lib\site-packages\numpy\testing\utils.py",
line 636, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 4 decimals

(mismatch 0.323624595469%)
x: array([ 48.324518 , 12.66334061, 26.19586592, 30.74311623,
35.68553023, 48.62067299, 67.61598597, 16.00193946,
13.37601498, 11.23054447, 9.48774695, 9.05886735,...
y: array([ 48.3243, 12.6633, 26.1958, 30.743 , 35.6854, 48.6206,
67.6159, 16.0018, 13.376 , 11.2305, 9.4877, 9.0588,
8.7644, 11.2268, 20.9667, 23.5576, 36.9585, 50.0107,...

----------------------------------------------------------------------
Ran 1470 tests in 97.341s

FAILED (SKIP=24, failures=3)


on python 3.2 I get 12 errors like this, my guess is string versus bytes issue

======================================================================
ERROR: Failure: ValueError (field named race_black not found.)
----------------------------------------------------------------------
Traceback (most recent call last):
File "E:\Josef\testing\tox\py32a\lib\site-packages\nose-1.1.2-py3.2.egg\nose\failure.py",
line 37,
in runTest
raise self.exc_class(self.exc_val).with_traceback(self.tb)
File "E:\Josef\testing\tox\py32a\lib\site-packages\nose-1.1.2-py3.2.egg\nose\loader.py",
line 495,
in makeTest
return self._makeTest(obj, parent)
File "E:\Josef\testing\tox\py32a\lib\site-packages\nose-1.1.2-py3.2.egg\nose\loader.py",
line 554,
in _makeTest
return MethodTestCase(obj)
File "E:\Josef\testing\tox\py32a\lib\site-packages\nose-1.1.2-py3.2.egg\nose\case.py",
line 346, in __init__
self.inst = self.cls()
File "E:\Josef\testing\tox\py32a\lib\site-packages\statsmodels\genmod\tests\test_glm.py",
line 238, in __init__
self.res2 = Lbw()
File "E:\Josef\testing\tox\py32a\lib\site-packages\statsmodels\genmod\tests\results\results_glm.py",
line 691, in __init__
data['race_black'], data['race_other'], data['smoke'],
File "E:\Josef\testing\tox\py32a\lib\site-packages\numpy\core\records.py",
line 457, in __getitem__
obj = ndarray.__getitem__(self, indx)
ValueError: field named race_black not found.


Josef

[Skipper Seabold]

Can you try replacing 'i8' with int?

Odd. Just lower precision to 3 I guess.

Just installed Python 3.2. Will have a look. This has never showed up
before? That's not a new test.

Skipper

[josef...@gmail.com]

the test specifies '<i8' as correct answer, Is that what you get on
your version combination.
For my versions it looks like it should just be '<i4'

I will run the tests again after looking at the 3.2 issue

I'm looking at this now, it's a consequence of the str call in
categorical I guess.
I'm in 3.2 and try to figure out if it's supposed to be string,
unicode (encoding of string) or bytes.

Josef

>
> Skipper

[Skipper Seabold]

It's 32-bit for 64-bit I'm pretty sure. Just using int should fix it.

Ah, makes sense.

> Josef
>
>>
>> Skipper

[josef...@gmail.com]

but then it's also platform or compiler dependent, I get also '<i4' on
64 bit python 3.2 (gohlke binaries)
I will try the int

Josef

[josef...@gmail.com]

I found the source of the problem, but not yet the solution

from csv file

>>> data.dtype.names
('id', 'low', 'age', 'lwt', 'race', 'smoke', 'ptl', 'ht', 'ui', 'ftv', 'bwt')

after calling categorical, concatenates string and bytes

>>> data2.dtype.names
('id', 'low', 'age', 'lwt', 'smoke', 'ptl', 'ht', 'ui', 'ftv', 'bwt',
"race_b'black'", "race_b'other'", "race_b'white'")

>>> data2.dtype.names[-1]
"race_b'white'"

content of array is bytes

>>> data['race'][:5]
rec.array([b'black', b'other', b'white', b'white', b'white'],
dtype='|S5')

>>> type(data['race'][0])
<class 'numpy.bytes_'>

Josef

[josef...@gmail.com]

numpy provided conversion function

>>> from statsmodels.compatnp.py3k import asbytes, asstr
>>> asstr(data['race'][0])
'black'
>>> 'race_' + asstr(data['race'][0])
'race_black'

>>> [asstr(ii) for ii in np.unique(data['race'])]
['black', 'other', 'white']

I wonder if numpy has a documentation for python 3.x (but not enough
today to go look for it)

Josef

[Skipper Seabold]

On Tue, Jun 12, 2012 at 9:58 PM, <josef...@gmail.com> wrote:
> numpy provided conversion function
>
>>>> from statsmodels.compatnp.py3k import asbytes, asstr
>>>> asstr(data['race'][0])
> 'black'
>>>> 'race_' + asstr(data['race'][0])
> 'race_black'
>
>>>> [asstr(ii) for ii in np.unique(data['race'])]
> ['black', 'other', 'white']
>
> I wonder if numpy has a documentation for python 3.x (but not enough
> today to go look for it)
>

I thought it might but I really don't know enough about py3 to open my mouth...

Skipper

[josef...@gmail.com]

On Tue, Jun 12, 2012 at 9:59 PM, Skipper Seabold <jsse...@gmail.com> wrote:
> On Tue, Jun 12, 2012 at 9:58 PM,  <josef...@gmail.com> wrote:
>> numpy provided conversion function
>>
>>>>> from statsmodels.compatnp.py3k import asbytes, asstr
>>>>> asstr(data['race'][0])
>> 'black'
>>>>> 'race_' + asstr(data['race'][0])
>> 'race_black'
>>
>>>>> [asstr(ii) for ii in np.unique(data['race'])]
>> ['black', 'other', 'white']

the copy of numpy's asstr doesn't convert numbers to strings, so I
adjusted it as asstr2.

master runs without errors and failures in all my tox environments,
minimum still python 2.5, numpy 1.4.1, scipy 0.7.
I don't have any more changes to the code planned (unless we find some
bugs of course)

docs build with a few warnings, but would benefit from manual
checking. (I never see rst errors, because they run off the screen)

Skipper, can you update the documentation on the web tonight?

One or two days of adding release related stuff (changes, notes), and
some checking of the docs and 0.4.1 could be out before the end of the
week. (I haven't run my 0.4.0 backwards compatibility tests yet, but
don't expect any problems there.)

Josef

[Skipper Seabold]

On Tue, Jun 12, 2012 at 11:55 PM, <josef...@gmail.com> wrote:
> On Tue, Jun 12, 2012 at 9:59 PM, Skipper Seabold <jsse...@gmail.com> wrote:
>> On Tue, Jun 12, 2012 at 9:58 PM,  <josef...@gmail.com> wrote:
>>> numpy provided conversion function
>>>
>>>>>> from statsmodels.compatnp.py3k import asbytes, asstr
>>>>>> asstr(data['race'][0])
>>> 'black'
>>>>>> 'race_' + asstr(data['race'][0])
>>> 'race_black'
>>>
>>>>>> [asstr(ii) for ii in np.unique(data['race'])]
>>> ['black', 'other', 'white']
>
> the copy of numpy's asstr doesn't convert numbers to strings, so I
> adjusted it as asstr2.
>
> master runs without errors and failures in all my tox environments,
> minimum still python 2.5, numpy 1.4.1, scipy 0.7.
> I don't have any more changes to the code planned (unless we find some
> bugs of course)
>
> docs build with a few warnings, but would benefit from manual
> checking. (I never see rst errors, because they run off the screen)
>
> Skipper, can you update the documentation on the web tonight?
>

Will do.

> One or two days of adding release related stuff (changes, notes), and
> some checking of the docs and 0.4.1 could be out before the end of the
> week.  (I haven't run my 0.4.0 backwards compatibility tests yet, but
> don't expect any problems there.)
>

Ok. Let me know when you want the binaries built. I can do this
today/tonight or it will have to be Monday.

Skipper

[josef...@gmail.com]

Monday, then.

I'd like to give Yaroslav a chance to test them on Debian/Ubuntu
before releasing.
And, I found one more bug in an untested code path of a test.

Josef

>
> Skipper

[Yaroslav Halchenko]

On Thu, 14 Jun 2012, josef...@gmail.com wrote:
> Monday, then.

> I'd like to give Yaroslav a chance to test them on Debian/Ubuntu
> before releasing.
> And, I found one more bug in an untested code path of a test.

FWIW -- seems to build fine BUT the same old issue I have mentioned
before on ubuntu 11.10 amd64:

reading sources... [ 24%] generated/statsmodels.genmod.generalized_linear_model.GLMResults.t_test
reading sources... [ 24%] generated/statsmodels.graphics.boxplots.beanplot
/usr/lib/pymodules/python2.7/matplotlib/sphinxext/plot_directive.py:322: PlotWarning: Exception running plot plots/graphics_boxplot_beanplot.py

Traceback (most recent call last):

File "/usr/lib/pymodules/python2.7/matplotlib/sphinxext/plot_directive.py", line 319, in render_figures
run_code(plot_path, function_name, plot_code, context=context)
File "/usr/lib/pymodules/python2.7/matplotlib/sphinxext/plot_directive.py", line 225, in run_code
os.chdir(path)
OSError: [Errno 2] No such file or directory: 'plots'

warnings.warn(s, PlotWarning)

Exception occurred:
File "/usr/lib/pymodules/python2.7/docutils/statemachine.py", line 233, in run
context, state, transitions)
File "/usr/lib/pymodules/python2.7/docutils/statemachine.py", line 454, in check_line
return method(match, context, next_state)
File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2281, in explicit_markup
nodelist, blank_finish = self.explicit_construct(match)
File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2293, in explicit_construct
return method(self, expmatch)
File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2035, in directive
directive_class, match, type_name, option_presets)
File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/states.py", line 2086, in run_directive
result = directive_instance.run()
File "/usr/lib/pymodules/python2.7/docutils/parsers/rst/__init__.py", line 370, in run
self.state, self.state_machine)
File "/usr/lib/pymodules/python2.7/matplotlib/sphinxext/plot_directive.py", line 483, in plot_directive
options, state_machine)
File "/usr/lib/pymodules/python2.7/matplotlib/sphinxext/plot_directive.py", line 400, in _plot_directive
shutil.copyfile(plot_path, os.path.join(destdir, fname))
File "/usr/lib/python2.7/shutil.py", line 81, in copyfile
with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: u'plots/graphics_boxplot_beanplot.py'
The full traceback has been saved in /tmp/sphinx-err-7QpCeB.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
Either send bugs to the mailing list at <http://groups.google.com/group/sphinx-dev/>,
or report them in the tracker at <http://bitbucket.org/birkenfeld/sphinx/issues/>. Thanks!
make[1]: *** [html] Error 1
make[1]: Leaving directory `/tmp/buildd/statsmodels-0.4.0+git1162-g602ec99/docs'
make: *** [doc-stamp] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
E: Failed autobuilding of package


if I find a moment I will try to figure it out...

full logs are at
http://neuro.debian.net/_files/_buildlogs/statsmodels/0.4.0+git1162-g602ec99
disregard those FAILED on older ubuntus (due to missing recent
numpy/scipy's) and I guess I forgotten to fix requirement of python >=2.6 so it
fails on debian stable while trying to test against 2.5 (will fix and retest)

: # Run unittests here against installed statsmodels
cd build/ && \
PYTHONPATH=`/bin/ls -d /tmp/buildd/statsmodels-0.4.0+git1162-g602ec99/debian/python-statsmodels/usr/lib/python2.5/*/` \
MPLCONFIGDIR=/tmp/buildd/statsmodels-0.4.0+git1162-g602ec99/build \
python2.5 /usr/bin/nosetests -s -v statsmodels

Traceback (most recent call last):

File "/usr/bin/nosetests", line 8, in <module>
load_entry_point('nose==0.11.1', 'console_scripts', 'nosetests')()
File "/usr/lib/pymodules/python2.5/nose/core.py", line 113, in __init__
argv=argv, testRunner=testRunner, testLoader=testLoader)
File "/usr/lib/python2.5/unittest.py", line 767, in __init__
self.parseArgs(argv)
File "/usr/lib/pymodules/python2.5/nose/core.py", line 164, in parseArgs
self.createTests()
File "/usr/lib/pymodules/python2.5/nose/core.py", line 178, in createTests
self.test = self.testLoader.loadTestsFromNames(self.testNames)
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 442, in loadTestsFromNames
return unittest.TestLoader.loadTestsFromNames(self, names, module)
File "/usr/lib/python2.5/unittest.py", line 565, in loadTestsFromNames
suites = [self.loadTestsFromName(name, module) for name in names]
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 394, in loadTestsFromName
discovered=discovered)
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 316, in loadTestsFromModule
tests.extend(self.loadTestsFromDir(module_path))
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 165, in loadTestsFromDir
entry_path, discovered=True)
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 394, in loadTestsFromName
discovered=discovered)
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 316, in loadTestsFromModule
tests.extend(self.loadTestsFromDir(module_path))
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 165, in loadTestsFromDir
entry_path, discovered=True)
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 394, in loadTestsFromName
discovered=discovered)
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 316, in loadTestsFromModule
tests.extend(self.loadTestsFromDir(module_path))
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 157, in loadTestsFromDir
entry_path, discovered=True)
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 394, in loadTestsFromName
discovered=discovered)
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 305, in loadTestsFromModule
test_classes + test_funcs)
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 304, in <lambda>
tests = map(lambda t: self.makeTest(t, parent=module),
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 511, in makeTest
return self.loadTestsFromTestClass(obj)
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 475, in loadTestsFromTestClass
for case in filter(wanted, dir(cls))]
File "/usr/lib/pymodules/python2.5/nose/loader.py", line 521, in makeTest
return MethodTestCase(obj)
File "/usr/lib/pymodules/python2.5/nose/case.py", line 328, in __init__
self.inst = self.cls()
File "/usr/lib/python2.5/site-packages/statsmodels/genmod/tests/test_glm.py", line 238, in __init__
File "/usr/lib/python2.5/site-packages/statsmodels/genmod/tests/results/results_glm.py", line 693, in __init__
File "/usr/lib/python2.5/site-packages/statsmodels/tools/tools.py", line 301, in add_constant
TypeError
make[1]: *** [python-test2.5] Error 1
make[1]: Leaving directory `/tmp/buildd/statsmodels-0.4.0+git1162-g602ec99'
make: *** [binary-arch] Error 2
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2

[josef...@gmail.com]

/tmp/sphinx-err-7QpCeB.log
does the error log say what's the absolute path of the not found file
plots/graphics_boxplot_beanplot.py
I don't know if the file is not where it is supposed to be, or if
sphinx is looking in the wrong place.

>
>
> if I find a moment I will try to figure it out...
>
> full logs are at
> http://neuro.debian.net/_files/_buildlogs/statsmodels/0.4.0+git1162-g602ec99
> disregard those FAILED on older ubuntus (due to missing recent
> numpy/scipy's) and I guess I forgotten to fix requirement of python >=2.6 so it
> fails on debian stable while trying to test against 2.5 (will fix and retest)

I'm successfully running it on python 2.5, it should still be compatible.

the message below breaks on a warning, This add_constant deprecation
warning is really a pain.
I don't know what might be wrong, maybe the two line text, I know I
had some problems in the past.
explicit string concatenation helped sometimes (I'm vague on this)

Otherwise you could try to delete the warning in
tools.tools.add_constant, and run the tests on py 2.5 to see if
everything else works.

Thanks,

Josef

[Yaroslav Halchenko]

On Sat, 16 Jun 2012, josef...@gmail.com wrote:
> >  File "/usr/lib/pymodules/python2.7/matplotlib/sphinxext/plot_directive.py", line 225, in run_code
> >    os.chdir(path)
> > OSError: [Errno 2] No such file or directory: 'plots'

> /tmp/sphinx-err-7QpCeB.log
> does the error log say what's the absolute path of the not found file
> plots/graphics_boxplot_beanplot.py
> I don't know if the file is not where it is supposed to be, or if
> sphinx is looking in the wrong place.

now that I have disabled building for 2.5 (sorry -- I just lack
time/motivation to dig into it atm), build succeeds on debian stable
(squeeze) and fails similar way during doc build:

OSError: [Errno 2] No such file or directory: 'plots'

and the tail of the sphinx's log file has:

return BaseDirective.run(self)
File "/tmp/buildd/statsmodels-0.4.0+git1162-g602ec99/docs/sphinxext/numpy_ext/numpydoc.py", line 162, in run
return base_directive.run(self)
File "/usr/lib/pymodules/python2.6/sphinx/directives/__init__.py", line 164, in run
self.state.nested_parse(self.content, self.content_offset, contentnode)
File "/usr/lib/pymodules/python2.6/docutils/parsers/rst/states.py", line 284, in nested_parse
node=node, match_titles=match_titles)
File "/usr/lib/pymodules/python2.6/docutils/parsers/rst/states.py", line 195, in run
results = StateMachineWS.run(self, input_lines, input_offset)
File "/usr/lib/pymodules/python2.6/docutils/statemachine.py", line 233, in run
context, state, transitions)
File "/usr/lib/pymodules/python2.6/docutils/statemachine.py", line 454, in check_line
return method(match, context, next_state)
File "/usr/lib/pymodules/python2.6/docutils/parsers/rst/states.py", line 2281, in explicit_markup
nodelist, blank_finish = self.explicit_construct(match)
File "/usr/lib/pymodules/python2.6/docutils/parsers/rst/states.py", line 2293, in explicit_construct
return method(self, expmatch)
File "/usr/lib/pymodules/python2.6/docutils/parsers/rst/states.py", line 2035, in directive
directive_class, match, type_name, option_presets)
File "/usr/lib/pymodules/python2.6/docutils/parsers/rst/states.py", line 2086, in run_directive
result = directive_instance.run()
File "/usr/lib/pymodules/python2.6/docutils/parsers/rst/__init__.py", line 370, in run
self.state, self.state_machine)
File "/usr/lib/pymodules/python2.6/matplotlib/sphinxext/plot_directive.py", line 342, in plot_directive
lines, state_machine.input_lines.source(0))
File "/usr/lib/pymodules/python2.6/docutils/statemachine.py", line 392, in insert_input
source='internal padding after '+source,
File "/usr/lib/pymodules/python2.6/docutils/statemachine.py", line 1193, in __radd__
raise TypeError('adding ViewList to a non-ViewList')
TypeError: adding ViewList to a non-ViewList


that is with

# Sphinx version: 1.0.7
# Python version: 2.6.6
# Docutils version: 0.7 release
# Jinja2 version: 2.5.5

[josef...@gmail.com]

nd-60 has python-matplotlib (0.99.3-1) but doesn't build docs AFAICS
nd-70 has python-matplotlib-data_1.1.1~rc1-2_all.deb no error
building docs
nd11.10+1_i386 has python-matplotlib-data_1.0.1-3_all.deb but
doesn't build docs AFAICS
nd11.10+1_amd64 haspython-matplotlib-data_1.0.1-3_all.deb
docs build fails with OSEroor

Skipper, maybe you understand better

possibly related to this, which was after matplotlib 1.0.1 according
to release (files) dates
https://github.com/matplotlib/matplotlib/commit/41f46a9f0968e4f64e02582c005b3a447b0688e8

my guess is that our docs build requires the
matplotlib/sphinxext/plot_directive.py of matplotlib version at least
> 1.0.1-3

three possibilities:
if it's a path-directory issue, maybe copying the plots directory to
one level higher up again, might work
If there are more problems with the plot_directive, we could provide
our own version (again)
ignore it: Debian question: Do the docs have to be buildable on the
target platform. I never assumed we run into that requirement.
(nose, sphinx, matplotlib for docs is pretty much whatever we use, and
not included in compatibility checks)


extra bug:
distributions, sdist, are missing the themes directory and causes
error in building docs. Copying the themes folder and html builds fine
in my py 2.6 version combination.

Thanks for testing

Josef

[Yaroslav Halchenko]

On Sun, 17 Jun 2012, josef...@gmail.com wrote:
> ignore it: Debian question: Do the docs have to be buildable on the
> target platform. I never assumed we run into that requirement.
> (nose, sphinx, matplotlib for docs is pretty much whatever we use, and
> not included in compatibility checks)

yes -- docs must be buildable BUT you are all fine for Debian
(sid) upload. That is why if you can release Monday -- please do so
that we could upload into Debian proper before the freeze (if it
wouldn't be too late already ;) )

All the above doc-building problems are related to backport builds --
for NeuroDebian we are going a few steps further and try to provide
backports for all recent debian/ubuntu releases. That is why it makes
it harder at times to provide such backport-friendly packaging so
we could just reuse it across all releases. And clearly we are trying
to avoid coming up with some adhoc solutions e.g. build docs on sid and
somehow ship for all other releases. If docs fail to build for older
release and we don't find an easy way to fix -- I would just either
patch for that specific failure or patch to remove building of docs for
that release altogether. But, once again, ideally -- they should be
buildable ;)

[josef...@gmail.com]

On Sun, Jun 17, 2012 at 9:39 PM, Yaroslav Halchenko
<yarik...@gmail.com> wrote:
>
> On Sun, 17 Jun 2012, josef...@gmail.com wrote:
>> ignore it:  Debian question: Do the docs have to be buildable on the
>> target platform.  I never assumed we run into that requirement.
>> (nose, sphinx, matplotlib for docs is pretty much whatever we use, and
>> not included in compatibility checks)
>
> yes -- docs must be buildable BUT you are all fine for Debian
> (sid) upload.  That is why if you can release Monday -- please do so
> that we could upload into Debian proper before the freeze (if it
> wouldn't be too late already ;) )

Thanks, I try to get the release ready.
(I needed some fun with coding this weekend.
http://jpktd.blogspot.ca/2012/06/qr-and-sequential-regression-with-fixed.html

)

>
> All the above doc-building problems are related to backport builds --
> for NeuroDebian we are going a few steps further and try to provide
> backports for all recent debian/ubuntu releases.  That is why it makes
> it harder at times to provide such backport-friendly packaging so
> we could just reuse it across all releases.  And clearly we are trying
> to avoid coming up with some adhoc solutions e.g. build docs on sid and
> somehow ship for all other releases.  If docs fail to build for older
> release and we don't find an easy way to fix -- I  would just either
> patch for that specific failure or patch to remove building of docs for
> that release altogether.  But, once again, ideally -- they should be
> buildable ;)

just to understand this backporting
Are you also backporting a newer version of matplotlib, and would it
help to put in a doc build requires version?

I will need to add doc builds to my tox virtualenvs, to see what the
version requirements are and to make sure we stick to them, assuming
it's possible with tox.

Josef

[Skipper Seabold]

Yes. This commit gets rid of the error for me, though this isn't the error I saw when I made that patch. Is this something we need to fix on our end? I will if we really need to.
 

three possibilities:
if it's a path-directory issue, maybe copying the plots directory to
one level higher up again, might work

It's not a path issue I don't think. I tried to just make a symbolic link to see if that fixed it.
 

If there are more problems with the plot_directive, we could provide
our own version (again)

I'd rather not have multiple versions floating around again. NumPy also patched for this error

http://projects.scipy.org/numpy/changeset/8680
 

ignore it:  Debian question: Do the docs have to be buildable on the
target platform.  I never assumed we run into that requirement.
(nose, sphinx, matplotlib for docs is pretty much whatever we use, and
not included in compatibility checks)


extra bug:
distributions, sdist, are missing the themes directory and causes
error in building docs. Copying the themes folder and html builds fine
in my py 2.6 version combination.

Pushed a patch.
 

[Yaroslav Halchenko]

On Sun, 17 Jun 2012, josef...@gmail.com wrote:
> > patch for that specific failure or patch to remove building of docs for
> > that release altogether.  But, once again, ideally -- they should be
> > buildable ;)

> just to understand this backporting
> Are you also backporting a newer version of matplotlib, and would it
> help to put in a doc build requires version?

no -- we are trying to avoid backporting of core packages -- only
leaf packages or packages used by others but the ones which we maintain
as well (e.g. nibabel) so we have better clue what is going on so we do
not brake the "stable" system installations. Some times we do
backport some (e.g. ipython 0.11 for Debian stable) since they are
needed but then it requires uglish resolutions so they get installed in
addition to stock versions provided by those stable releases of Debian.

[josef...@gmail.com]

I'm done, my last planned change just went into master.

tests run without error or failures on py 2.5 to 2.7 and 3.2

Josef