pluggable backends

[pablo...@gmail.com]

Hi,

I've started reading the code to see how I can plug a custom OT backend but it seems that the notebooks has dependency on google drive.
https://github.com/jupyter/colaboratory/blob/master/colaboratory/resources/colab/notebook.html#L69

Are there plans to make google drive optional and add an API to use a different backend?

Thanks

[Matthias Bussonnier]

Hi, 

I've started reading the code to see how I can plug a custom OT backend but it seems that the notebooks has dependency on google drive.
https://github.com/jupyter/colaboratory/blob/master/colaboratory/resources/colab/notebook.html#L69

Colaboratory is **not** IPython notebook.  

The IPython notebook is here http:/:github.com/ipython/ipython

And we are working on integrating google drive as an extension here :

https://github.com/jupyter/jupyter-drive

We are slowly mergingg colaboratory into main IPython itself, 

so in the end, the repo you looked at should not really exist beyond historical reason, or 

for a fully integrated NaCl app in chrome.

Are there plans to make google drive optional and add an API to use a different backend?

I suppose that by OT, you mean Operational Transform, so yes, there are plan. 

But it ned a significant refactor of javascript, and help would be welcomed.

We were thinking of integrating Drive only in a first pass, but if you are interesting in adding something 

else, have your feedback contribution at the same time would be really appreciated. 

May I ask which backend you had in mind ? 

-- 

Matthias

Thanks

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/e8e51fd3-a4d9-4224-9dd5-0b54d3678fda%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pablo...@gmail.com]

On Wednesday, December 3, 2014 8:57:45 AM UTC+2, bussonniermatthias wrote:

Hi, 

I've started reading the code to see how I can plug a custom OT backend but it seems that the notebooks has dependency on google drive.
https://github.com/jupyter/colaboratory/blob/master/colaboratory/resources/colab/notebook.html#L69

Colaboratory is **not** IPython notebook.  

The IPython notebook is here http:/:github.com/ipython/ipython

And we are working on integrating google drive as an extension here :

https://github.com/jupyter/jupyter-drive

We are slowly mergingg colaboratory into main IPython itself, 

so in the end, the repo you looked at should not really exist beyond historical reason, or 

for a fully integrated NaCl app in chrome.

Colaboratory uses google closure while the iptyhon-notebook doesn't.
Isn't it a complete rewrite of the notebook?
What is the scope of Jupyter and what's the scope of colaboratory?

Are there plans to make google drive optional and add an API to use a different backend?

I suppose that by OT, you mean Operational Transform, so yes, there are plan. 

But it ned a significant refactor of javascript, and help would be welcomed.

We were thinking of integrating Drive only in a first pass, but if you are interesting in adding something 

else, have your feedback contribution at the same time would be really appreciated. 

May I ask which backend you had in mind ? 

Yes, I meant Operational Transformation.
This is the cleanest OT client/server implementation I found.
It has CodeMirror support.
I didn't check the Python server but I expect it to work.
https://github.com/Operational-Transformation/ot.js
https://github.com/Operational-Transformation/ot.py
 
What I'm trying to use ot.js to sync changes to the CodeMirror cells through a server.
The server will also act as a proxy to the kernel. All users will be connected to a single kernel through that proxy.
When a user executes a cell, the result will be sent to all users.

For this should I start with the colaboratory repo or the ipython repo?

[Matthias BUSSONNIER]

Hey, 

<Put  on  développer hat that need to be cautious>

Colaboratory uses google closure while the iptyhon-notebook doesn't.
Isn't it a complete rewrite of the notebook?
What is the scope of Jupyter and what's the scope of colaboratory?

Collaboratory was written internally by google without our knowledge at the beginning. 

It was a complete rewrite of the notebook in pure javascript to run in browser + google 

drive + real-time collaboration. It also deal with some issues like security using iframes. 

it was also a playground for trying tech like NaCl, to have the full python stack in browser. 

But lets put that aside. 

The amount of code duplication was really high, so we will bring most of colaboratory feature into the notebook itself.

If everything work as planned, We should't need colaboratory anymore, or it would "just" become flavor of the notebook 

which is a chrome app. 

Jupyter, the org, is meant to regroup any component that are language agnostic and currently or

were  part of IPython

- The notebook

- The qt console

- the terminal console @ process zmq

- nbconvert. 

- nbviewer. 

- Google drive Backend integration

- … 

Not part of Jupyter:

- IPython kernel, 

- Magics. 

- Traitlets.

Roughly if yo u have a project the question :

Could it be part of Jupyter ?

is equivalent to 

Can it be used regardless of wether the end user code in Python, Haskell, Julia, or R ?

Yes, I meant Operational Transformation.
This is the cleanest OT client/server implementation I found.
It has CodeMirror support.
I didn't check the Python server but I expect it to work.
https://github.com/Operational-Transformation/ot.js
https://github.com/Operational-Transformation/ot.py

Ah, nice ! thanks for the link, I'll have a look.

 
What I'm trying to use ot.js to sync changes to the CodeMirror cells through a server.
The server will also act as a proxy to the kernel. All users will be connected to a single kernel through that proxy.
When a user executes a cell, the result will be sent to all users.

I would be careful, as I think we should separate "user as document edit access" from "user have execute access". 

Also where would the file be saved ? Central storage? One of the users ? 

For this should I start with the colaboratory repo or the ipython repo?

I would say Either IPython and/or Jupyter-drive. or even create a jupyter-otjs repo. 

We are trying to push IPython 3.0 through the door soon, so significant refactor will happened post 3.0, 

so you might fell a little push-back until mid-january when IPython 3.0 will be released. 

After that we'll start chopping IPython repo into pieces to move the right pieces to jupyter.

If you plan on working on OT, we could definitively work together to get a common OT abstraction layer 

that either support gdrive to.js or firebase. The abstraction layout should be in Jupyter/IPython-notebook itself,

while ot-backend would be plug able extensions. 

How does that sound ? 

Should we like start  a technical conversation here and now ? Or open an issue somewhere to dicuss ? 

<Now change hat for enthusiast user>

Having a OT layer on the server would buyer awesome, and would print some advantages over google drive, being : 

  - you can close all the browsers and still have result appended to notebook document !

  - Usage on labs with high security demands where gdrive is out of questions. 

Looking forward to get more people working on that ! You should probably join one of the google hangout we have every weeks, 

though I guess the next few ones will be annoying about finishing up IPython 3.0. 

Feel free to ask any question on where to find what on IPython, as a first stem I would just o ahead and modify source, we can fire out how to clean that and 

make abstractions/subproject later. 

Cheers, 

-- 

M

-- 

Matthias

Thanks

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/e8e51fd3-a4d9-4224-9dd5-0b54d3678fda%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/f4c97f09-f4f2-4bdb-a2bb-7f32a0366cb5%40googlegroups.com.

[pablo...@gmail.com]

As I see it, there are several parts:

- Create a proxy server that receive operations (OT) and the usual notebook protocol.
The proxy will send operations to the OT server and the notebook protocol to Jupyter.
It will also be responsible for saving the notebook and handling permissions.
- Listen to CodeMirror events and send OT operations to the proxy.
- Make the notebook UI modular in case it will be embedded in an interface that will take care of permissions (edit/execute), creating and saving notebooks.

Are there events in the notebook when a cell is created/deleted so we can hook OT?
How can I use the existing Websocket connection to send OT operations?
How can I send the messages to the proxy server and from the proxy server to jupyter/iPython?
Should I start with the ipython notebook code or wait for it to be moved to Jupyter?
https://github.com/ipython/ipython/tree/master/IPython/html/static/notebook

Thank you for the detailed explanation about Juptyer and colaboratory.
It might be helpful to add this to the Juptyer website.
 

[Matthias Bussonnier]

Le 8 déc. 2014 à 10:03, pablo...@gmail.com a écrit :

As I see it, there are several parts:

- Create a proxy server that receive operations (OT) and the usual notebook protocol.
The proxy will send operations to the OT server and the notebook protocol to Jupyter.
It will also be responsible for saving the notebook and handling permissions.

Probably, not sure how yet, Google drive will need direct js<-> drive communication, 

but a "proxy" that could be in current tornado server make sens. 

- Listen to CodeMirror events and send OT operations to the proxy.

Yes, and beyond code mirror, we could listen to cell insertion deletion, move, execution. 

- Make the notebook UI modular in case it will be embedded in an interface that will take care of permissions (edit/execute), creating and saving notebooks.

That could be/will be doable to; but let's think at this as orthogonal to OT. 

Are there events in the notebook when a cell is created/deleted so we can hook OT?

Yes, though I believe we should do that the other way around, have a model with event and

reflect changes in the model in the object. 

How can I use the existing Websocket connection to send OT operations?

Websoket go (almost) strait to kernel, we might need one more websocket or ajax call. 

How can I send the messages to the proxy server and from the proxy server to jupyter/iPython?

That have to be designed. 

Should I start with the ipython notebook code or wait for it to be moved to Jupyter?
https://github.com/ipython/ipython/tree/master/IPython/html/static/notebook

Move to jupyter will take some time (help welcome to cut the repo in many project) 

so fee free to hack on IPython. 

Thank you for the detailed explanation about Juptyer and colaboratory.
It might be helpful to add this to the Juptyer website.

Yes, like many docs, we have to find time for that :-)

-- 

M

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/fd923fd0-0df0-414f-b9de-9e813804d44b%40googlegroups.com.

[Doug Blank]

For those of us that are a few steps behind you two, here is an interesting article (although *some* say it is "tl;dr"):

http://www.codecommit.com/blog/java/understanding-and-applying-operational-transformation

Ironically, It mentions a "Jupiter collaboration system" [1], an early project with some OTs. 

Here is a link to the start of the OT code:

http://operational-transformation.github.io/index.html

 The idea of "pair programming" could be used in a Jupyter+OT system, which has shown to be effective for learning and effective code development [2, 3].

Looking forward to seeing a new Jupyter collaboration---at a couple of levels!

-Doug


[1] - http://doi.acm.org/10.1145/215585.215706

[2] - http://en.wikipedia.org/wiki/Pair_programming

[3] - http://collaboration.csc.ncsu.edu/laurie/pair.html

[pablo...@gmail.com]

A similar collaborative iPtyhon notebook concept but probably not open sourced
http://sagemath.blogspot.co.il/2013/09/ipython-notebooks-in-cloud-with.html

Doug, this is an interesting read.
A full understanding of OT is not required when using the ot.js library.
It has a CodeMirror adapter.

I hoped for a more friendlier API and modularity both on the client and server :)
I guess that's part of what Jupyter is going to improve.

Implementing this should be relatively easy for someone familiar with the notebook and server code.
I'll start experimenting with the client and server and see how it goes but it might take few weeks or probably months.

[Doug Blank]

One thing that Operational Transforms should give Jupyter (if I understand this), beyond all of the obvious, is that the notebook could finally have a complete undo/redo stack of editing commands. I think that this is critical to being widely accepted as a  platform for use in the classroom. Without it, many educators just won't touch it.

I think targeting a few months from now is probably about right. But I hope that something like this can be a focus of IPython 4 (if that is what comes next).

-Doug

[Thomas Kluyver]

On 8 December 2014 at 15:48, <pablo...@gmail.com> wrote:

A similar collaborative iPtyhon notebook concept but probably not open sourced
http://sagemath.blogspot.co.il/2013/09/ipython-notebooks-in-cloud-with.html

We're in regular contact with William Stein, the force behind SageMathCloud - I think the code for his live collaboration layer is open source, but it's not really integrated with IPython in a way that we can easily pick up.

Matthias is going to be working on live collaboration in the notebook, but it will be some time before it's ready to use. We'll need to redesign things so that the server has a copy of the document. That will have other benefits besides simultaneous editing, but it will take some work.

Thomas

[Matthias Bussonnier]

Hey, 

Me and Kester Tong (cc) will have a meeting on this Monday 10am NY time. 

Probably google hangout I guess. 

We'll discuss the current plan until the release of IPython 3.0 for the current Google drive integration, 

and then plan the things for real-time operational transform migration. 

If anyone want to join, please say so here so that we can send google invite to your adress at the time 

of the meeting. 

Cheers, 

-- 

M

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAOvn4qhqgDFeO-tQT_gHO89m7ET2HvKYD%3DqgxMyTJGi%3DX1Sg_g%40mail.gmail.com.