Kernel gateway and turning notebooks into RESTful APIs

[Peter Parente]

Hi all,

I wanted to share a few blog posts we just made about turning notebooks into REST APIs / web services / microservices / (pick your favorite term) using the incubating kernel gateway project. The first is an introduction and the other two describe specific demo applications. The demo assets are all on GitHub and linked from within the posts.

1. Jupyter Notebooks as RESTful Microservices (http://blog.ibmjstart.net/2016/01/28/jupyter-notebooks-as-restful-microservices/)

2. Powering applications with a Notebook Microservice (http://blog.ibmjstart.net/2016/01/28/powering-your-application-with-a-notebook-microservice/)

3. Notebook Microservice And Swagger (http://blog.ibmjstart.net/2016/01/28/notebook-microservice-and-swagger/)

I appreciate any thoughts you’re willing to share.

Best,

Pete

[Peter Parente]

And if you're reading this on the Google Groups web site, the auto-linking breaks all the links, so here they are again without parentheses around them. :(

* http://blog.ibmjstart.net/2016/01/28/jupyter-notebooks-as-restful-microservices/

* http://blog.ibmjstart.net/2016/01/28/powering-your-application-with-a-notebook-microservice/

* http://blog.ibmjstart.net/2016/01/28/notebook-microservice-and-swagger/

[Michael Gilbert]

Those links all result in 404

[Kyle Kelley]

I was able to delete the parens at the very end to get to, e.g.

http://blog.ibmjstart.net/2016/01/28/notebook-microservice-and-swagger/

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/e4da5bfa-33b5-47e0-b109-f079c6e0bfb4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kyle Kelley (@rgbkrk; lambdaops.com)

[Kyle Kelley]

Also, awesome! I'm really happy about this and look forward to reading all your posts for the kernel gateway!

[Michael Kit Gilbert]

Sorry I should have mentioned that even the new ones you posted get 404s. The individual link you just sent does work though. I don't mean to nitpick, I just thought you might want to know.

You received this message because you are subscribed to a topic in the Google Groups "Project Jupyter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jupyter/w9SGV5N2jBs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jupyter+u...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CA%2BtbMaV0BPDFg-YR%2B%2BzKqJOQ0HazyWvEWSaGctcdhGpabFdeMA%40mail.gmail.com.

[Fernando Perez]

Here are plaintext versions of all three posts:

http://blog.ibmjstart.net/2016/01/28/powering-your-application-with-a-notebook-microservice/

http://blog.ibmjstart.net/2016/01/28/jupyter-notebooks-as-restful-microservices/

http://blog.ibmjstart.net/2016/01/28/notebook-microservice-and-swagger/

Peter, thank so much both for the great work of the entire team, and for documenting it for the community in this way!

Cheers,

f

[Peter Parente]

Thanks, Fernando. Apparently I need to learn how to copy/paste or use a better client!

[Fabian Eduardo Diaz Lizcano]

Hello All,

Is posible publish many services of the many notebook using jupyter kernel gateway for the same port all for example I have 3 notebook and for each notebook there are 3 services Can I publish all for the same port?.

[Peter Parente]

Hi Fabian

Is posible publish many services of the many notebook using jupyter kernel gateway for the same port all for example I have 3 notebook and for each notebook there are 3 services Can I publish all for the same port?.

You can only specify one notebook at present. It might be possible to add support for the case you mention. It'll take a bit of thinking about the design: a separate kernel pool per notebook, evaluating all of the notebooks in a single pool, specifying multiple notebooks/specs on the command line, handling conflicts, etc. Alternatively, it might turn out to be simpler to put a proxy in front of separate servers or use a web framework instead of a notebook for the more complex cases.

If you have a strong use case for this support, please open a PR or issue on the kernel gateway repo.

Cheers,

Pete

[Pav A]

Peter, I have experience of using kernel-gateway in the http-mode. The ability to create endpoints with one comment is super-cool, however I felt the case was self-limiting as one had to specify one notebook as seed_uri per kernel-gateway process. 

1. I feel http-mode belongs to a separate repo as the use case is very different from kernel-gateway.

2. kg-http-mode should really support a mode where you can specify the seed_uri via an API - while the service is running. This way you could do a requests.get to a cell of a notebook running in Jupyter notebook server - for each of running kernels, not just one.

3. nb2kg is even more powerful extension that allows offloading computational power to a dedicated box. It is the key component that ensures the notebooks and the dashboards server reference exactly the same environment. An example in https://github.com/jupyter-incubator/dashboards_setup/blob/master/docker_deploy/Dockerfile.notebook is suboptimal as it installs required dependencies twice - once for the kernels container, and second time for the Jupyter notebook server. What if your environment a bit more complex and makes use of environment variables too? I think a better use case of making the notebook server and the gateway referencing the same kernel environment is presented here: https://github.com/rs2/pyconuk-2016-jupyter-workshop/blob/master/docker/Dockerfile.notebook

nb2kg really needs to be available on pypi/conda-forge. Happy to give a hand, I think it's a really powerful project.

[Peter Parente]

 

1. I feel http-mode belongs to a separate repo as the use case is very different from kernel-gateway.

We discussed splitting the modes out long ago and decided that there's wasn't really a benefit to maintain another repo with its own packaging and release cycle at the time. If having two packages in the repo is causing problems, then we should reconsider it. 

 

2. kg-http-mode should really support a mode where you can specify the seed_uri via an API - while the service is running. This way you could do a requests.get to a cell of a notebook running in Jupyter notebook server - for each of running kernels, not just one.

This sounds like a good PR. There's some complications to solve with the kernel pool concept where N kernels are running to service requests for a single notebook simultaneously. I don't think there's any blockers, just work that has to be done to maintain a kernel pool per notebook.

 

3. nb2kg is even more powerful extension that allows offloading computational power to a dedicated box. It is the key component that ensures the notebooks and the dashboards server reference exactly the same environment. An example in https://github.com/jupyter-incubator/dashboards_setup/blob/master/docker_deploy/Dockerfile.notebook is suboptimal as it installs required dependencies twice - once for the kernels container, and second time for the Jupyter notebook server. What if your environment a bit more complex and makes use of environment variables too? I think a better use case of making the notebook server and the gateway referencing the same kernel environment is presented here: https://github.com/rs2/pyconuk-2016-jupyter-workshop/blob/master/docker/Dockerfile.notebook

nb2kg really needs to be available on pypi/conda-forge. Happy to give a hand, I think it's a really powerful project.

@jtyberg wrote the nb2kg as a proof-of-concept. It's definitely a cool little extension. I don't have cycles to spend on it, but if you want to submit PRs I'm sure someone can review and merge them. If you think it really deserves to be its own repo and package, maybe we can move it to jupyter-incubator or make it a top-level Jupyter project if there's enough consensus for people to maintain it?

Cheers,

Pete

[Justin Tyberg]

I've heard from others who are interested in nb2kg.  I don't mind submitting a jupyter-incubator proposal, but I'm not sure it's gained enough traction to meet the new sub-project proposal criteria.

https://github.com/jupyter/governance/blob/master/newsubprojects.md#criteria-for-official-subprojects