Using nbconvert API with a specific kernel in a conda env

[Paul Hobson]

Hey folks, 

I've been searching all morning, but coming up empty. The question is this:

Is there a way to using the nbconvert python API with a kernel installed in a conda environment?

Background:

I'm trying to write a documentation build script for a module that converts notebooks to RST, and then compiles HTML docs with Sphinx.

I have installed the module I'm documenting into an isolated conda environment called "probscale".

So with this trivial notebook (note the kernelspec):

a_test.ipynb:

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.04809765 -0.65510951 -0.06389409 -0.78956431  1.12613107]\n"
     ]
    }
   ],
   "source": [
    "import numpy\n",
    "print(numpy.random.normal(size=5))"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:probscale]",
   "language": "python",
   "name": "conda-env-probscale-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

When I run:

    

if 1:

    import nbformat

    from nbconvert import preprocessors

    nbfile = 'a_test.ipynb'
    meta = {
        'metadata': {
            'path': '.',
        }

    }
    with open(nbfile, 'r') as nbf:
        nbook = nbformat.read(nbf, as_version=4)
       
    runner = preprocessors.ExecutePreprocessor()
    runner.preprocess(nbook, meta)
   
    with open(nbfile, 'w') as nbf:
        nbformat.write(nbook, nbf)

I get a KeyError and a NoSuchKernal error

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
C:\Users\phobson\Miniconda3\envs\probscale\lib\site-packages\jupyter_client\kernelspec.py in get_kernel_spec(self, kernel_name)
    172         try:
--> 173             resource_dir = d[kernel_name.lower()]
    174         except KeyError:

KeyError: 'conda-env-probscale-py'

[snip]

C:\Users\phobson\Miniconda3\envs\probscale\lib\site-packages\jupyter_client\kernelspec.py in get_kernel_spec(self, kernel_name)
    173             resource_dir = d[kernel_name.lower()]
    174         except KeyError:
--> 175             raise NoSuchKernel(kernel_name)
    176 
    177         return self._get_kernel_spec_by_name(kernel_name, resource_dir)

NoSuchKernel: No such kernel named conda-env-probscale-py

I tried adding the kernel spec from the notebook to metadata passed to runner.preprocess(nbook, meta), but that produced the same result.

Any thoughts or guidance would be much appreciated.

Cheers,

-Paul

[Matthias Bussonnier]

Hi Paul,

As far as I can tell this is one of the limit of the conda extension
shiped with conda that autolist the kernelspecs. It does not work with
nbconvert.
You need to activate the env and manually install the spec using
`python -m ipykernel install <--options>`, and set the notebook to use
this kernelspec using the notebook UI.
Then, it should work.

Cheers,
--
M

> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jupyter+u...@googlegroups.com.
> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jupyter/cbee4894-2f86-44f0-a130-2cf63081e47c%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Paul Hobson]

Matthias,

Many thanks for the quick response and helpful solution.

Indeed, running:

$ # source activate <envname> # non-windows

$ activate <envname> # windows

(envname) $ python -m ipykernel install --name <envname>

And then manually changing the kernel in each of the notebooks before running my script solved it.

> email to jupyter+unsubscribe@googlegroups.com.

> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jupyter/cbee4894-2f86-44f0-a130-2cf63081e47c%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CANJQusW2j%2B%3DSGrr%3DodjNEsXgTHwGHHNxQd0LCsksE2K8x4iqTw%40mail.gmail.com.

[Matthias Bussonnier]

Great !

Sorry about the inconvenience.
I'm not sure how things would interact, but if you install the spec
with the right `--name` you might not have to switch each notebook.

Cheers,
--
M

>> > email to jupyter+u...@googlegroups.com.

>> > To post to this group, send email to jup...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/jupyter/cbee4894-2f86-44f0-a130-2cf63081e47c%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Project Jupyter" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to jupyter+u...@googlegroups.com.

>> To post to this group, send email to jup...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/jupyter/CANJQusW2j%2B%3DSGrr%3DodjNEsXgTHwGHHNxQd0LCsksE2K8x4iqTw%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to jupyter+u...@googlegroups.com.

> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jupyter/CADT3MECr_J1kgHaFBOdv22wrR44kzUDzSZzE2_H%3DOkMchzmWAA%40mail.gmail.com.

[Thomas Kluyver]

If ipykernel is installed where you're running the script, I think it should also be possible to tell nbconvert to use it directly, something like this:

runner = preprocessors.ExecutePreprocessor(kernel_name='python3')  # or 'python2', as appropriate


>> > email to jupyter+unsubscribe@googlegroups.com.

>> > To post to this group, send email to jup...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/jupyter/cbee4894-2f86-44f0-a130-2cf63081e47c%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Project Jupyter" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to jupyter+unsubscribe@googlegroups.com.

>> To post to this group, send email to jup...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/jupyter/CANJQusW2j%2B%3DSGrr%3DodjNEsXgTHwGHHNxQd0LCsksE2K8x4iqTw%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to jupyter+unsubscribe@googlegroups.com.

> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jupyter/CADT3MECr_J1kgHaFBOdv22wrR44kzUDzSZzE2_H%3DOkMchzmWAA%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CANJQusULtS_3UzoqSadiZyVv4EVA0e36uCYhYs9PsxrJiR5Lyg%40mail.gmail.com.

[Paul Hobson]

Yes, is this a piece that I left out of my response to Matthias.

This is what I have in my script:

runner = preprocessors.ExecutePreprocessor(kernel_name='probscale') # name of the env

Where "probscale" is the name of the kernel (and my conda env)

>> > email to jupyter+u...@googlegroups.com.

>> > To post to this group, send email to jup...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/jupyter/cbee4894-2f86-44f0-a130-2cf63081e47c%40googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Project Jupyter" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to jupyter+u...@googlegroups.com.

>> To post to this group, send email to jup...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/jupyter/CANJQusW2j%2B%3DSGrr%3DodjNEsXgTHwGHHNxQd0LCsksE2K8x4iqTw%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to jupyter+u...@googlegroups.com.

> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jupyter/CADT3MECr_J1kgHaFBOdv22wrR44kzUDzSZzE2_H%3DOkMchzmWAA%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.