Re: [pystatsmodels] Regression model interface

[Skipper Seabold]

On Tue, Jun 19, 2012 at 5:34 PM, Brendan Barnwell <bren...@gmail.com> wrote:
> I've been looking around at statsmodels and I'm especially interested in the
> regression facilities.  I currently use R for such tasks, but it'd be great
> if I could find a Python solution, since I like Python a lot more.
>
> It seems clear, though, that the interface to the statsmodels regression
> tools is much rougher than that provided by R's glm or a package like lmer.
> What I'm trying to figure out is to what extent the actual stats aren't
> implemented in statsmodels, versus just not being wrapped in a nice
> interface.
>
> For instance, "Regression with Discrete Independent Variable" makes it sound
> like it could be used to do some of the stuff I want (binary logistic
> regression), but the documentation is very sparse and doesn't actually
> explain what format you need to put your data in to get it to work.  Is the
> machinery there to support something like R's ability to just do
> glm(OneVar~(OtherVar+MoreVar+ThirdVar)^2, and it's just that there's no nice
> formula interface?  Or would this require actually crunching the data into
> some other format expected by statsmodels?  Basically what I'm wondering is,
> as someone who knows math and Python but doesn't know the details of the
> statistical computations involved in these regressions, could I write a
> wrapper to provide a nicer interface?  Or are there "real work" that needs
> to be done on the statistical side?
>

The machinery is there to compute the model. There's just not yet a
formula interface to create your design matrix for you. You have to do
this by hand.

We are working on merging in a formula interface though. I will
actually probably make the pull request in the next day or so, and I
hope to have it released by mid-July. You can get a sense for how it
will look to work with these models here

https://github.com/jseabold/statsmodels/blob/formula-integration/statsmodels/formula/example_interactions.py

Any feedback as we discuss the merge would be appreciated and welcome.

Generally though we just expect that you data is in the form Y ~ X
where X is a 2d array that contains the columns of your variables and
Y is your (binary, for your case) response variable. Please let us
know anywhere you find the documentation or examples to be lacking.
Here are some workable examples using maximum likelihood and GLM for
fitting binary response models with formulae.

http://statsmodels.sourceforge.net/devel/examples/generated/example_discrete.html
http://statsmodels.sourceforge.net/devel/examples/generated/example_glm.html

> Also, does statsmodels provide any way to do mixed-effects models?

There is some code in the sandbox to do mixed-effects regression but
it needs to be cleaned up. Josef may know more the status here.

Skipper

[josef...@gmail.com]

mixed effects models is a big topic, some versions are working, much
is still missing.

Unless we get the support of a statistician, my plan is to move into
the econometrics literature with this,
closer to R plm for linear models, and straightforward maximum
likelihood for discrete models.
(panel data and repeated measures)

Josef

>
> Skipper

[Brendan Barnwell]

On Tuesday, June 19, 2012 2:55:05 PM UTC-7, jseabold wrote:

We are working on merging in a formula interface though. I will
actually probably make the pull request in the next day or so, and I
hope to have it released by mid-July. You can get a sense for how it
will look to work with these models here

https://github.com/jseabold/statsmodels/blob/formula-integration/statsmodels/formula/example_interactions.py

Any feedback as we discuss the merge would be appreciated and welcome.

Generally though we just expect that you data is in the form Y ~ X
where X is a 2d array that contains the columns of your variables and
Y is your (binary, for your case) response variable. Please let us
know anywhere you find the documentation or examples to be lacking.
Here are some workable examples using maximum likelihood and GLM for
fitting binary response models with formulae.

http://statsmodels.sourceforge.net/devel/examples/generated/example_discrete.html
http://statsmodels.sourceforge.net/devel/examples/generated/example_glm.html

Thanks, that looks interesting.  I do have some suggestions on the documentation.  Like, I was looking at http://statsmodels.sourceforge.net/stable/discretemod.html .

The main thing is that the documentation and examples don't show what the functions actually do, nor what the data is.  Seeing code that just says "spector_data.exog = sm.add_constant(spector_data.exog)" is not very informative.  At the least, the examples should have a little intro paragraph saying "this code computes a linear regression to determine how variables X and Y predict Z".  It would also be helpful to have a "head()" call or the like in there just to show that the data are.  When I'm browsing examples, I don't want to just see code that says "spector_data = sm.datasets.spector.load()", because that doesn't tell me what format the data is in, and one of the reasons I'm browsing the docs is to see what format I would have to put my data in to make it work.  I'm not going to be loading sample data sets.

The documentation also doesn't really explain what the function arguments are supposed to be.  It just says "Logit(endog, exog)".  That doesn't explain what endog and exog are, nor what format they need to be in.  (It's possible that this is explained on a different doc page for a different sort of model, but if so that should be linked from each page that assumes that knowledge.)  Also, it doesn't even explain that "endog", and "exog" mean "endogenous" and "exogenous" variables.  I figured that out, but that terminology is unfamilar to me.  Looking it up I saw some descriptions of economics stuff, but at least in my context (linguistics) it's more common to use terms like "independent and dependent variables".  I'm not saying that the page needs to be a thesaurus of all possible terminology, but it should at least write out in full words the meaning of each argument so you know what values to pass to it.

So those are some things that I think would make the docs more useful to someone looking at it from the perspective of "can/should I use this to do the stats I want to do?"

> Also, does statsmodels provide any way to do mixed-effects models?

There is some code in the sandbox to do mixed-effects regression but
it needs to be cleaned up. Josef may know more the status here.

Judging from his response it's not quite there yet, so I will have to take a rain check and stick to R for now, as mixed effects models are more or less essential for what I'm doing.  I'm glad to see that a Python stats toolkit is developing though, so I'll keep my eyes peeled for news.

Thanks,
Brendan

[josef...@gmail.com]

On Fri, Jun 22, 2012 at 9:18 PM, VincentAB <vincen...@gmail.com> wrote:
> Although I wouldn't put things as strongly as Brendan, I share some of his
> feelings toward the statsmodels docs. From the perspective of someone who
> might like to contribute improvements to the docs and examples, I was
> wondering what your vision is on this. Do you mainly see the website docs as
> a "rich man page", or are you aiming for something more in line with Quick
> R: http://www.statmethods.net/

both,
Man pages in the style of numpy and scipy are a minimum requirement
and what I'm used to.
More tutorial or Quick R documentation would be very desired.

>
> My view is that there is a definite need for a Quick statsmodels, but I'm
> wondering if such a thing should live on its own webpage or if you thing the
> statsmodels docs will eventually evolve in that direction. In the former
> case, I'll have to punt (too much work!). In the latter case, however, I can
> see many new adopters begin able to put together useful examples that nicely
> complement the man pages.

I was thinking along the latter after Brendan's comments.

New contributions to improve the documentation to make statsmodels
more accessible to newcomers would be very welcome. Vincent Davis was
making similar comments maybe 2 years ago, and started on a tutorial
(and laid the foundation for the summary method).

The main problem is manpower.
As long as it's mainly Skipper and I doing development, maintenance
and documentation, it's rather limited what we can accomplish in our
time. We have 4 GSOC students which are busy with their own projects.
I'm pretty happy if I can even keep the "man pages" up-to-date and
improving. Those are pretty much designed for an intermediate python
and statistics level. Large parts of my work between 0.4.0 and 0.4.1
was updating or improving this part.
(for example adding plots to docstrings:
http://statsmodels.sourceforge.net/devel/generated/statsmodels.graphics.boxplots.beanplot.html#statsmodels.graphics.boxplots.beanplot
)

There are parts where I tried to provide more of an overview, but I
didn't spend the time adding examples to docstrings or rst files
http://www.statmethods.net/stats/rdiagnostics.html
ours:
http://statsmodels.sourceforge.net/devel/stats.html#residual-diagnostics-and-specification-tests
http://statsmodels.sourceforge.net/devel/diagnostic.html#diagnostics
(my blog is more for work in progress than final results, e.g.
http://jpktd.blogspot.ca/2012/01/influence-and-outlier-measures-in.html
)

open issues (filed by myself):
https://github.com/statsmodels/statsmodels/issues?labels=comp-docs&state=open

In 0.4.0, we added examples that ipython and sphinx renders to html,
e.g. http://statsmodels.sourceforge.net/devel/examples/generated/example_discrete.html
In this case also: I went through the examples, that have been written
as scripts before, to make sure they run and render without problems,
but I didn't go through the content to check whether the style is
really appropriate for the new format and has enough information for
newcomers.
Submission/contribution of examples by users would be very useful.
I've seen some tutorials for statsmodels on the web.


We have most of the infrastructure for the documentation now. We are
still missing a real "Getting Started" tutorial section, and many of
the module docstrings (big ones regression, discrete, genmod, ...) or
topics pages are very "skinny"

missing overview and recommendation of what to read for newcomers.
for example endog exog is explained/hidden here
http://statsmodels.sourceforge.net/devel/dev/package_overview.html#design
basic structure and import paths
http://statsmodels.sourceforge.net/devel/importpaths.html

missing examples in docstrings all over the place

one lacking part of the documentation is cross-linking
https://github.com/statsmodels/statsmodels/issues/278
to make it easier to find the relevant information.


So, the area where the documentation can be improved is pretty wide,
and we are open to adding more sections, tutorials, examples,
improvements, ...

numpy and scipy managed to get improved documentation through a large
community effort over the last several years.

We don't have an online editor for statsmodels, but, in the times of
git and github:

Pull requests are very welcome.

Josef

>
> Vincent

[Skipper Seabold]

To echo and add:

We would love to have more user friendly and tutorial-style
documentation. We are currently writing a tutorial for SciPy 2012
which will go into the documentation. Eventually I think we'll have a
much more extensive, readable documentation (in some form), but right
now the documentation is mostly scraped from the code with some more
here and there.

We do have a github repo for the project as a whole. Contributions and
pull requests could be made against this repo, if desired, while the
source docs stay more auto-generated manual style from the source
code. I'm not sure yet.

https://github.com/statsmodels/statsmodels-website

Thanks for the feedback though. Feedback and contributions are very
welcome right now. Whereas the R community has at least 100s of people
active in the community (outside of core and package developers) we
are but a handful of developers, mostly working on writing code so
that we have to something to advertise (eventually). Having people who
are willing to engage with us and some of the source to write docs is
and will continue to be greatly helpful. Either way hearing what you
like / dislike is also very important.

Cheer,

Skipper

[VincentAB]

Thanks for your replies.

So I'm still not entirely sure where fleshed out examples/tutorials should go. One thing I find confusing about the current docs is that things are all over the place (e.g. when you click on the docs link, the website's top menu bar suddenly has a new option called "examples", but many entries in the docs also have examples section that are often empty. Strange duplication). Should examples be place as .py scripts in the examples folders to be rendered automatically in html form, or should they be integrated directly to a rst doc file in a format like the one at http://statsmodels.sourceforge.net/devel/regression.html:

intro/math model
man page
references
examples (with actual examples here, not just a placeholder or cross-links)

Vincent

[VincentAB]

Quick idea: Could the script examples be rendered as html directly in the relevant section of the docs instead of being cross-linked?

[Skipper Seabold]

On Sat, Jun 23, 2012 at 10:41 AM, VincentAB <vincen...@gmail.com> wrote:
> Thanks for your replies.
>
> So I'm still not entirely sure where fleshed out examples/tutorials should
> go. One thing I find confusing about the current docs is that things are all
> over the place (e.g. when you click on the docs link, the website's top menu
> bar suddenly has a new option called "examples", but many entries in the
> docs also have examples section that are often empty. Strange duplication).

How do you mean duplication? The examples sections in the docs come
from the examples of the docstrings. These are meant to be quick start
examples, much like those in R or Stata docs at the bottom of any cran
page or stata help.

The Examples header points to scripts that are meant to be longer,
tutorial style examples - but mostly time constraints mean we haven't
cleaned these up and made them as useful as they could be.

> Should examples be place as .py scripts in the examples folders to be
> rendered automatically in html form, or should they be integrated directly
> to a rst doc file in a format like the one at
> http://statsmodels.sourceforge.net/devel/regression.html:
>

The examples/ folder contains many Python scripts that are
automagically rendered into .rst files under the Examples header. This
rendering is probably sub-optimal (it inserts code into
ipython_directive sphinx code blocks) and definitely fragile. At this
point I am thinking that most of these will be subsumed by IPython
notebook scripts since they can be both literate and runnable without
any extra work on our part. Actually...

https://github.com/statsmodels/statsmodels/issues/328

So maybe it's best to make tutorial documentation contributions as
IPython notebooks?

> intro/math model
> man page
> references
> examples (with actual examples here, not just a placeholder or cross-links)

You can certainly contribute in this way too. We have documentation
for VAR that is a .rst page and not a runnable script.

http://statsmodels.sourceforge.net/devel/vector_ar.html#var

Does that help clear some things up? Mainly things feel scattered
because they are at this point. Working on the docs is a large
undertaking done mostly in spurts when I find myself with a few days
to hunker down and focus on it.

[Skipper Seabold]

On Sat, Jun 23, 2012 at 11:01 AM, VincentAB <vincen...@gmail.com> wrote:
> Quick idea: Could the script examples be rendered as html directly in the
> relevant section of the docs instead of being cross-linked?
>

Hmm. You mean having an include directive at the actual landing pages
for each module? Like discrete would have an include for
examples/example_discrete.py?

I don't see why not. If we have many, many examples (wishful thinking)
though it might make the page loading take a while. What would be the
advantage over just having better links to example from the docs
pages?

[josef...@gmail.com]

On Sat, Jun 23, 2012 at 11:06 AM, Skipper Seabold <jsse...@gmail.com> wrote:
> On Sat, Jun 23, 2012 at 11:01 AM, VincentAB <vincen...@gmail.com> wrote:
>> Quick idea: Could the script examples be rendered as html directly in the
>> relevant section of the docs instead of being cross-linked?
>>
>
> Hmm. You mean having an include directive at the actual landing pages
> for each module? Like discrete would have an include for
> examples/example_discrete.py?
>
> I don't see why not. If we have many, many examples (wishful thinking)
> though it might make the page loading take a while. What would be the
> advantage over just having better links to example from the docs
> pages?

Since I'm too slow in replying, just on this

I don't like very long pages, it makes browsing more difficult, so I
would put examples/tutorials mostly on separate pages unless they are
short.

I don't know whether providing ipython notebooks will substitute for
the examples. I still want them as rendered html in the online docs
and htmlhelp.
(I tried ipython notebooks only for a few hours. For *my* taste they
are too much white space, and make reading code difficult. I'm looking
forward to the conversion facilities for notebooks.)

Josef

[Skipper Seabold]

On Sat, Jun 23, 2012 at 11:16 AM, <josef...@gmail.com> wrote:
> On Sat, Jun 23, 2012 at 11:06 AM, Skipper Seabold <jsse...@gmail.com> wrote:
>> On Sat, Jun 23, 2012 at 11:01 AM, VincentAB <vincen...@gmail.com> wrote:
>>> Quick idea: Could the script examples be rendered as html directly in the
>>> relevant section of the docs instead of being cross-linked?
>>>
>>
>> Hmm. You mean having an include directive at the actual landing pages
>> for each module? Like discrete would have an include for
>> examples/example_discrete.py?
>>
>> I don't see why not. If we have many, many examples (wishful thinking)
>> though it might make the page loading take a while. What would be the
>> advantage over just having better links to example from the docs
>> pages?
>
> Since I'm too slow in replying, just on this
>
> I don't like very long pages, it makes browsing more difficult, so I
> would put examples/tutorials mostly on separate pages unless they are
> short.
>
> I don't know whether providing ipython notebooks will substitute for
> the examples. I still want them as rendered html in the online docs
> and htmlhelp.
> (I tried ipython notebooks only for a few hours. For *my* taste they
> are too much white space, and make reading code difficult. I'm looking
> forward to the conversion facilities for notebooks.)
>

Notebooks are exportable to HTML and you can configure viewing
experience to your heart's content via CSS AFAIK.

Skipper

[josef...@gmail.com]

On Sat, Jun 23, 2012 at 11:39 AM, Skipper Seabold <jsse...@gmail.com> wrote:
> On Sat, Jun 23, 2012 at 11:16 AM,  <josef...@gmail.com> wrote:
>> On Sat, Jun 23, 2012 at 11:06 AM, Skipper Seabold <jsse...@gmail.com> wrote:
>>> On Sat, Jun 23, 2012 at 11:01 AM, VincentAB <vincen...@gmail.com> wrote:
>>>> Quick idea: Could the script examples be rendered as html directly in the
>>>> relevant section of the docs instead of being cross-linked?
>>>>
>>>
>>> Hmm. You mean having an include directive at the actual landing pages
>>> for each module? Like discrete would have an include for
>>> examples/example_discrete.py?
>>>
>>> I don't see why not. If we have many, many examples (wishful thinking)
>>> though it might make the page loading take a while. What would be the
>>> advantage over just having better links to example from the docs
>>> pages?
>>
>> Since I'm too slow in replying, just on this
>>
>> I don't like very long pages, it makes browsing more difficult, so I
>> would put examples/tutorials mostly on separate pages unless they are
>> short.

In terms of (re-)organizing the documentation, it might be good to
keep introductory information, tutorials, examples a bit separate from
the module reference, "man pages".

From my experience with scipy or, now, learning Stata: At the
beginning I went through the tutorials and a lot of background
information, but after some time, I just wanted to get the specific
information for a function or method.

Now that I'm more familiar with it, I like the documentation style of
Stata. Each model has a pretty concise description page with the
essentials in a consistent format, but everything has hyperlinks to
the detailed explanations and related methods.
SAS has for each procedure several sections with introductory
information and examples, and then one technical section, that
explains what the procedure and all it's options actually do.
UCLA stats has a lot of nice tutorials but often organized by topic,
and might cover a range of models.

So, depending on the content and the intended audience, I still think
it is useful to have examples in different parts of the documentation.

Josef

[Fernando Perez]

On Sat, Jun 23, 2012 at 8:16 AM, <josef...@gmail.com> wrote:
> I don't know whether providing ipython notebooks will substitute for
> the examples. I still want them as rendered html in the online docs
> and htmlhelp.
> (I tried ipython notebooks only for a few hours. For *my* taste they
> are too much white space, and make reading code difficult. I'm looking
> forward to the conversion facilities for notebooks.)

The conversion code is in semi-usable shape and actually only needs a
*small* amount of work:

https://github.com/ipython/nbconvert

I've already tested conversions to rst for Sphinx and it works
reasonably (but by no means perfectly) well. Unfortunately I'm maxed
out for the next few weeks and won't make any progress on this, but at
Scipy I'm hoping to convince some of our sprinting folks to make a
push on it. If that doesn't happen, I'll likely just double down and
finish the code after Scipy'12, because it really is becoming pretty
important to have it done.

With nbconvert done the way I want it, it will be *trivial* to have
make rules in your Sphinx makefile to ensure that out of every
notebook you get:

1. A clean rst file for sphinx to build as part of the docs.
2. A link to a .py file with just the code for those who want to run
just the code as a script.
3. A second .py link for a script that has all the notebook *text* as
well, but wrapped in comment blocks. This will be a more verbose
python script but it will make a good, readable version in pure .py
form.

So your users would be able to simply choose which one they prefer:
the notebook itself, to read and copy/paste pieces from the web page,
or to download either a compact or a verbose pure .py script they can
run just like any other script.

Hopefully that setup will make for a good workflow for projects: they
can author notebooks (which I really think is the most convenient way
to polish and iterate on narrative example codes), but do not force an
IPython dependency at all on their users.

This can happen sooner rather than later with more outside help, but
we're committed to making it happen no matter what.

Cheers,

f

[justin]

Is there any code (either in Statsmodels or in the sandbox) that does
linear regression with parameter restrictions? I browsed around but
couldn't find anything.

Thanks,

Justin

[Wes McKinney]

There's an implementation from a few years ago in
pandas.sandbox.stats.rls, check it out. Likely needs to be spruced up
given API changes in statsmodels.

https://github.com/pydata/pandas/blob/master/pandas/sandbox/stats/rls.py

- Wes

[josef...@gmail.com]

there is a copy of the pandas version in statsmodels.sandbox.rls.py
It is in working condition. Runs without errors, but I don't know
about the status.

Josef


>
> - Wes

[justin]

Thanks Wes and Josef, I will check it out.

[VincentAB]

Josef,

Does this mean you would oppose the consolidation of the various OLS examples.py, for instance?

Vincent

[josef...@gmail.com]

On Sat, Jun 23, 2012 at 5:32 PM, VincentAB <vincen...@gmail.com> wrote:
> Josef,
>
> Does this mean you would oppose the consolidation of the various OLS
> examples.py, for instance?

The only parts I have a strong opinion about are the module
references/manpages and the examples folders inside the source tree,
because I'm using those all the time.

I'd also like to keep the overview pages for the model categories
(subdirectories of statsmodels) like
http://statsmodels.sourceforge.net/devel/tsa.html regression, genmode
reasonably short.

For the rest I'm pretty open, and can be easily convinced of
improvements. I also wouldn't "oppose" them if they are not in my
preferred style/format/organization, if others find it useful.

One open question is where do you want to put examples and
documentation that covers several areas, contrast between OLS and RLM,
OLS and diagnostics or specification searches, count or logit model
with discrete versus genmod.

I'm mainly trying to get an overview how a restructuring of the
documentation might look like.

Josef

[VincentAB]

My inclination would be to:

1. WLS vs OLS in the WLS section
2. RLM vs OLS in the RLS section
3. Merge/cleanup OLS, minimal OLS, and Joint hypothesis tests
4. Discrete can probably stand on its own, but there should be a subsection for individual models so that the words "Probit", "Multinomial Logit" pop out in (a possibly "folded") table of content. I think this may be good for marketing to new users).

1 & 2 seem reasonable because users will only care about the comparison if they think OLS may be insufficient in the first place.

Vincent

[josef...@gmail.com]

On Sat, Jun 23, 2012 at 9:53 PM, VincentAB <vincen...@gmail.com> wrote:
> My inclination would be to:
>
> 1. WLS vs OLS in the WLS section
> 2. RLM vs OLS in the RLS section
> 3. Merge/cleanup OLS, minimal OLS, and Joint hypothesis tests
> 4. Discrete can probably stand on its own, but there should be a subsection
> for individual models so that the words "Probit", "Multinomial Logit" pop
> out in (a possibly "folded") table of content. I think this may be good for
> marketing to new users).
>
> 1 & 2 seem reasonable because users will only care about the comparison if
> they think OLS may be insufficient in the first place.

sounds good to me, maybe with an overview page over examples
"if you think that your OLS might not be the right model after all,
head over to ..."


just as background:
some of the examples had different purposes when they were just
scripts and not rendered documentation. For example, minimal OLS, or
similar for other models was often my starting point for writing new
example files for different topics or just for quickly checking a
model.
tut_xxx.py were intended as more introductory examples
ex_xxx, and examples_xxx were often intended for special topics.
(try_xxx are usually my dirty working scripts, that haven't been cleaned up yet)

this was or is a useful distinction for standalone example scripts,
but is often not the best structure for a consistent documentation.

Thank you for working on this.

[VincentAB]

I sent in a pull request, but it's the first time I try to contribute something using git/github, so let me know if I can improve on my workflow.

Best,

Vincent