Examples in ipython notebooks

[VincentAB]

Hi,

Some members of this list have previously expressed interest in publishing statsmodels examples in the form of ipython notebooks.

This morning I converted all the examples currently published on the statsmodels web site to notebook form. Here is an example (from my recent pull request) in which I use a few different features (e.g. math, hyperlinks, markdown stuff, etc.):

http://nbviewer.ipython.org/3481922

I can make the other files available as well if there is interest.

Vincent

[joe bradley]

Hi folks,

I'm a brand new to the module, but I work in the notebook all the time, and I really like this.  The notebook setup makes trying out test code and getting one's feet on the ground very easy, so thanks Vincent.

Best,

Joe Bradley

Physicist, Lawrence Livermore National Lab

[josef...@gmail.com]

Very nice. (I didn't know about nbviewer yet.)

I fast possibility would be to add a wiki page with the links to the
gists and the nbviewer pages.
I think statsmodels wiki can be edited by anyone, and it would be an
easy way to keep track of notebooks in gists.

But I think we should also include them as part of the documentation,
example list. No need to wait for automatic conversion if you already
did it.

Thanks,

Josef

>>
>> Vincent
>
>

[VincentAB]

Cool.

I'm not sure if the wiki is very useful for this (even as a stopgap measure) since there is little content on there and I doubt it's getting many page views.

Here's what I propose to do:

1) Create a "ipynb" subfolder in "statsmodels/examples" and put all the notebooks in that subfolder
2) Edit the examples index.rst: Create 2 sections html and notebook. The html section will display the current TOC. The notebook section will display a bulletpoint list of links to .ipynb files.
3) Pull request

I don't yet understand very well how the docs build yet, so someone will probably have to show me how to get sphinx to copy .ipynb to a proper location in "build/html". When I know this location, I can adjust the relative links I inserted in 2 above.

Sounds good?

Vincent

[josef...@gmail.com]

sounds good. I don't know about the sphinx doc build yet
(Also I don't know how we could render our own html for notebooks if
we want to.)

However,
add a *prominent* link to a wiki page
let users add a link to their own gist notebooks and benefit directly
from the nbviewer rendering.

The first is part of the "official" documentation and needs to go
through PRs, the second is an easy way for users or anyone to
advertise their notebooks.

Josef

>
> Vincent
>

[VincentAB]

Thanks Joe,  I'm glad this is going to be helpful to some.

Josef,

PR446 includes the changes proposed above.

You changed my mind about the wiki. It's a brilliant idea to let users add their own examples without having to go through "official" channels. So I gave the wiki a quick facelift:

Home: https://github.com/statsmodels/statsmodels/wiki
Examples: https://github.com/statsmodels/statsmodels/wiki/Examples:-user-contributions

I also put together a Notebook for the Getting Started page I proposed in PR396. You can see it in its full glory here: http://nbviewer.ipython.org/3485014

Best,

Vincent

[trelsco]

Brilliant use of notebook.  More, more...

[josef...@gmail.com]

working my way through the pull request queue, I'd like to merge

https://github.com/statsmodels/statsmodels/pull/446

I'm thinking of merging it pretty much as it is now. Right now it will
be just another form of the examples for users.

Vincent added them also here
https://github.com/statsmodels/statsmodels/wiki/Examples:-user-contributions
where they can be looked at with an nbviewer (but requires storing a
duplicate as gist).

two question for notebooks in the source (I'm not up-to-date or don't
find the relevant emails anymore)

1) is there a simple way to smoke test the notebooks, just to check
whether they run without an exception?

2) future: is there an easy way to render them during the doc build,
so they can be fully integrated with the documentation?

bonus question:
Is there a reason against storing the full notebook format instead of
the minimal markup python version?

Thanks,

Josef

[VincentAB]

1) Testing: I don't know

2) The future seems bright. F. Perez writes: "I've already tested conversions to rst for Sphinx and it works reasonably (but by no means perfectly) well"

The conversation is here: https://groups.google.com/d/msg/pystatsmodels/ua8eVY3cuSg/rOvPiqawMj0J

nbconvert: https://github.com/ipython/nbconvert

3) Not sure what you mean. The .ipynb are the full notebook files. They're always just JSON.

[Dave Hirschfeld]

On Wednesday, September 19, 2012 7:19:21 PM UTC+1, josefpktd wrote:

working my way through the pull request queue, I'd like to merge

https://github.com/statsmodels/statsmodels/pull/446

I'm thinking of merging it pretty much as it is now. Right now it will
be just another form of the examples for users.

Vincent added them also here
https://github.com/statsmodels/statsmodels/wiki/Examples:-user-contributions
where they can be looked at with an nbviewer (but requires storing a
duplicate as gist).

two question for notebooks in the source (I'm not up-to-date or don't
find the relevant emails anymore)

1) is there a simple way to smoke test the notebooks, just to check
whether they run without an exception?

2) future: is there an easy way to render them during the doc build,
so they can be fully integrated with the documentation?

bonus question:
Is there a reason against storing the full notebook format instead of
the minimal markup python version?

Thanks,

Josef

I don't think you need to have duplicate gists - if the ipynb file is in your repository you can view it on GitHub, click the `Raw` button and past the raw url into the nbviewer. As an example the nbviewer url to view the `ex_arma2.ipynb` file from Vincent's branch above is:

http://nbviewer.ipython.org/urls/raw.github.com/vincentarelbundock/statsmodels/8dab3fe7d0d4023308bebbe2688f6a2c5fbe1442/examples/ipynb/ex_arma2.ipynb 

I'm auto-rendering notebooks as part of our doc-build process on Windows. At present I simply have a doc-build.bat file which will run the sphinx doc build then will call nbconvert on each of our notebooks. It might be a slightly hacky solution but it works well in practice with the rendered notebooks being automatically linked to from our documentation.

-Dave

[josef...@gmail.com]

On Wed, Sep 19, 2012 at 9:56 PM, VincentAB <vincen...@gmail.com> wrote:
> 1) Testing: I don't know
>
> 2) The future seems bright. F. Perez writes: "I've already tested
> conversions to rst for Sphinx and it works reasonably (but by no means
> perfectly) well"
>
> The conversation is here:
> https://groups.google.com/d/msg/pystatsmodels/ua8eVY3cuSg/rOvPiqawMj0J
>
> nbconvert: https://github.com/ipython/nbconvert

Thanks for the link

>
> 3) Not sure what you mean. The .ipynb are the full notebook files. They're
> always just JSON.

http://ipython.org/ipython-doc/dev/interactive/htmlnotebook.html#importing-or-executing-a-notebook-as-a-normal-python-file

only restriction seems to be no ipython magic

The pull request is merged with only a minor edit

Josef
inline

[josef...@gmail.com]

good to know, I didn't figure this out when I tried last time

If we point the wiki pages to the notebooks in master, then users can
point out if they don't work. (crowd testing :) (but we might get
into version conflicts with master)

>
> I'm auto-rendering notebooks as part of our doc-build process on Windows. At
> present I simply have a doc-build.bat file which will run the sphinx doc
> build then will call nbconvert on each of our notebooks. It might be a
> slightly hacky solution but it works well in practice with the rendered
> notebooks being automatically linked to from our documentation.

is this solution somewhere public?

I updated https://github.com/statsmodels/statsmodels/issues/328
(I'm not patient enough right now to figure this out.)

Thanks,

Josef

>
> -Dave

[VincentAB]

A simple solution could be to not have the examples on the main website at all. The menu bar link could simply point users to the wiki examples page.

On the wiki, there could be 2 sections,

1) "official" examples with ipynb file and nbviewer rendering which links directly to an .ipynb file on statsmodels github repo (with raw like Dave suggested).
2) User-supplied examples.

This would have the advantage of bypassing completely the need to render notebooks as html. We could just use nbconvert to save notebooks as simple .py scripts for users who don't want to deal with notebooks or html nbviewer.

Vincent

[VincentAB]

Of course, there would still have to be a way to test notebooks.

[josef...@gmail.com]

I didn't look carefully enough at the pull request notebooks.

I don't like these notebooks any more in the source in the current
form. These are notebooks with output and png's.

I just wanted the stripped notebooks in the source tree.
Getting the output and plots in the source tree is too heavy and will
change with any cosmetic changes.

Also I *guess* now the nbviewer is just displaying the json content, I
thought it's calculating the output.

So, when we use notebooks with stripped output in the source, then we
have to create the output before adding them to the html docs.

(Obviously notebooks are not part of my usual workflow.)

Josef

[VincentAB]

No worries, this is a trivial change. I will do it tonight. Would you prefer a new pull request, or would you like me to change the files in my current repo branch?

Vincent

[josef...@gmail.com]

I just cleared the output of interactions notebook, file size goes
from 408KB to 20 KB.

Your branch is already merged after a rebased, so you should not add
anything more to that. It would make it difficult to distinguish new
from already merged changes.

I just have to do "All Output - clear" ?

I can do it directly since I'm currently on master with these changes.

Vincent, as far as I can see I have merged all your Pull Requests.
Tonight should be the scheduled update of the website. Then you could
check if you still see anything wrong in the documentation.

Thanks,

Josef


>
> Vincent

[VincentAB]

Sounds good. I'll have a look tonight.

Vincent