Executing notebooks with parameters

[Antonio Ingargiola]

Hi Joivans,

lately I found myself using more and more batch execution of Jupyter/IPython notebooks to coordinate different parts of data analysis. I use a "master notebook" instead of a script so I can show links to the executed notebooks (both to generate an index and for quick debug) and some result summaries (HTML tables or plots).

Once you start going this route, you quickly realize that would be super neat to pass "arguments" to notebooks (similarly to functions) to avoid having to duplicate notebooks only to process multiple data files or to use a different set of analysis parameters.

I want to share a small wrapper function to achieve exactly this (on a rudimentary level) using nbconvert:

https://github.com/tritemio/nbrun

You can find more info and limitations in the README. Even with all its limitations, this function has been terribly useful for me. 

So I'm sharing it in case others want to use it.

Any comments and ideas are welcome.

Antonio

[Matthias Bussonnier]

Hi Antonio, 

I think you might be interested in looking at 

- https://github.com/paulgb/runipy

and 

- https://github.com/jbn/BatchNotebook

in particular this issue:

https://github.com/paulgb/runipy/issues/17

Also since this PR [1], you should be able to make this just another exporter for nbconvert and do jupyter nbconvert --to nbrun.Somthing --options myNotebook.ipynb

Thanks !

-- 

M

[1]: https://github.com/jupyter/nbconvert/pull/185

Antonio

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/8a966d18-63dd-463b-b9ae-7cd98e914265%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Kluyver]

On 19 January 2016 at 21:46, Matthias Bussonnier <bussonnie...@gmail.com> wrote:

I think you might be interested in looking at 

- https://github.com/paulgb/runipy

and 

- https://github.com/jbn/BatchNotebook

Also:
https://github.com/takluyver/nbparameterise

The key feature of nbparameterise is that you have a notebook that still makes sense when you open it and run it normally, rather than it referring to external things like environment variables. Passing parameters in works by replacing values in the first cell and re-running it.

Thomas

[Antonio Ingargiola]

On Tuesday, January 19, 2016 at 1:55:38 PM UTC-8, takowl wrote:

On 19 January 2016 at 21:46, Matthias Bussonnier <bussonnie...@gmail.com> wrote:

I think you might be interested in looking at 

- https://github.com/paulgb/runipy

and 

- https://github.com/jbn/BatchNotebook

I used runiy in the past, but the argument passing is through environment variables which is very cumbersome IMHO. With "my" (or Thomas) approach I can pass strings, numbers, and list/dict of strings and numbers easily.

 

Also:
https://github.com/takluyver/nbparameterise

The key feature of nbparameterise is that you have a notebook that still makes sense when you open it and run it normally, rather than it referring to external things like environment variables. Passing parameters in works by replacing values in the first cell and re-running it.

Thomas, I wasn't aware of nbparametrize (or maybe I saw it in the past then forgot about it?). This is the same concept I implemented in nbrun, only I just insert a second cell, instead of rewriting the first one. In this way I don't need to manipulate an existing cell and the first cell indicates default values.

Looking at nbparameterize code, here some other differences:

-  I'm saving executed notebooks as as ipynb, not HTML. I like to be able to jump directly to a failing notebook by clicking on the link printed on the "master notebook".

- I serialize an input dict of arguments to a string containing a series of assignment using repr(). Quick and dirty way to pass a wide range of basic python types (must have a complete literal representation).

- nbrun is an much less sophisticated than nbparametrize, it contains only a single .py with two small functions (only one "public").

I also conventionally document the input arguments of a template notebook (a notebook to be parametrized) in a markdown cell, see:

https://github.com/tritemio/nbrun/blob/master/example-template-notebook.ipynb

Overall I'm pretty happy with the result which required a ridiculously low effort on my part (thanks to nbconvert doing all the work). I'd like to see something like this becoming a more standard functionality, for example in nbconvert.

Antonio

[Thomas Kluyver]

On 20 January 2016 at 02:10, Antonio Ingargiola <trit...@gmail.com> wrote:

-  I'm saving executed notebooks as as ipynb, not HTML. I like to be able to jump directly to a failing notebook by clicking on the link printed on the "master notebook".

The web form and conversion to HTML in nbparamaterise is mainly intended as a demo application. I consider the important thing to be the API which does two important steps:

1. Pass in a notebook, get back a description of possible parameters, based on the contents of the first cell.

2. Pass in values for the parameters, get back a copy of the notebook with those replaced in the first cell, and the code re-executed. Then you can do whatever you want with that notebook, including saving it to an ipynb file or running it through nbconvert.

I think I should make this separation between the core API and the demo app clearer.

Thomas

[Thomas Kluyver]

On 20 January 2016 at 11:22, Thomas Kluyver <tak...@gmail.com> wrote:

I think I should make this separation between the core API and the demo app clearer.

I've spent some time cleaning this up today. nbparameterise is now a package on PyPI, and the webapp part is example code in the repository with it.

There's also an example of running a notebook in in batch mode, looping over input values and saving a copy of the notebook for each:
https://github.com/takluyver/nbparameterise/blob/master/examples/batch.py

Thomas

[Antonio Ingargiola]

Thanks, now I understand better how nbparametrize works. Is it currently possible to pass tuple/list/dict as parameter values?

Antonio

[Thomas Kluyver]

On 20 January 2016 at 20:54, Antonio Ingargiola <trit...@gmail.com> wrote:

Thanks, now I understand better how nbparametrize works. Is it currently possible to pass tuple/list/dict as parameter values?

Great :-). To be honest, I think looking at it again has helped me understand how it should work.

At the moment it only finds strings, numbers and True/False as parameters - though I don't think it actually checks what's passed back in, so you might be able to give it other types. That's partly because I was designing it around a simple web form, and partly because I was thinking about supporting other languages, and those basic types are pretty universal.

It wouldn't be too hard to add the basic collection types (if you're familiar with Python ASTs, see the code at [1]). But I reckon a lot of the use cases for parameterised notebooks work best with simple types - it's much harder to design a web form or a command line interface that takes collection types.

[1] https://github.com/takluyver/nbparameterise/blob/master/nbparameterise/code_drivers/python3.py

Thomas

[Andrei Plamadă]

You could try https://github.com/hz-inova/run_jnb/ as well. 

It should be straightforward.

Andrei