The biggest picture

[Edward K. Ream]

Imo, the most important direction for Leo is the best possible integration with existing programs and tools, such as:

- The Language Server Protocol.
- Better emulation of org-mode features, such as agendas and tables.

These are important directions for Leo, and I'll be working on both.

But imo the most important long-term direction for Leo is the best possible integration with the Jupyter project. Jupyter is where the action is, with a talented team of devs and a budget in the millions of dollars. Leo has features lacking in Jupyter, and vice versa. Better integration with Jupyter is my top priority in the coming year.

Edward

[Terry Brown]

On Thu, 14 Sep 2017 06:41:49 -0700 (PDT)
"Edward K. Ream" <edre...@gmail.com> wrote:

> Imo, the most important direction for Leo is the best possible
> integration with existing programs and tools, such as:
>

> - The Language Server Protocol. <http://langserver.org/>

> - Better emulation of org-mode features

> <http://orgmode.org/features.html>, such as agendas and tables.

>
> These are important directions for Leo, and I'll be working on both.
>
> But imo the most important long-term direction for Leo is the best

> possible integration with the Jupyter <http://jupyter.org/>project.

> Jupyter is where the action is, with a talented team of devs and a
> budget in the millions of dollars. Leo has features lacking in
> Jupyter, and vice versa. Better integration with Jupyter is my top
> priority in the coming year.

I've thought about this too. I suspect you've studied Jupyter much
more than I have so this may be "well, duh" for you, but here's where
I'm at currently.

Jupyter has cells which have an input and output component, source
(text) and result. Different cells can be handled by different
languages (R, Python, etc) in the same script, and variables passed
between them. Results are saved and can be recalculated on demand.

So the leo_edit_pane component would be great for displaying cells
because of its source / rendered split approach. A stack of
leo_edit_pane components is *visually* a Jupyter notebook.

But while there are challenges there, I think the interface end of the
problem is not the place to start.

I think the first question is - do we want to emulate or copy the
Jupyter approach in Leo (as is unavoidably the case with org mode), or
do we want to integrate Leo with Jupyter, so that Leo becomes an
alternate interface to real Jupyter notebooks. I'm convinced we want
to go the second route, that's the only way to get all the power of
Jupyter in terms of its integrations with other systems.

I know Leo can read, so some degree, Jupyter notebooks, but I'm
wondering if we need to back off a step and connect Leo to the "Jupyter
kernel" so that reading notebooks, and executing cells and caching
results etc. is all done by Jupyter.

The big difference here is that when Leo reads a notebook and makes it
part of the Leo outline everything's in the Leo system, vs. displaying
a Jupyter notebook, but maintaining links back to the Jupyter
environment, for cell execution etc. Actual implementation might be
some kind of hybrid between reading in to Leo and interacting with
Jupyter. As opposed to the other extreme, storing all state in Jupyter
and reading it constantly.

So, if I was working on Leo / Jupyter integration I'd start by writing
a non-Leo Python script that programatically created a Jupyter notebook
(using Jupyter libraries) which incorporates at least three languages,
say Python, markdown, and R or javascript, renders cells, observes data
exchange between languages, etc. From that Jupyter API perspective, I
think full featured integration into Leo will be easier.

Anyway, just thinking aloud, and maybe all "well, duh" for those who've
thought more about it than I have.

Cheers -Terry

[Edward K. Ream]

On Thu, Sep 14, 2017 at 9:21 AM, Terry Brown

<terry...@gmail.com> wrote:

I've thought about this too.  I suspect you've studied Jupyter much
more than I have so this may be "well, duh" for you, but here's where
I'm at currently.

​I'm no expert. The basis of my interest is that Jupyter is cool, under intense development, and very widely used.

So the leo_edit_pane component would be great for displaying cells
because of its source / rendered split approach.  A stack of
leo_edit_pane components is *visually* a Jupyter notebook.

​I agree.​

But while there are challenges there, I think the interface end of the
problem is not the place to start.

​I agree.

​

I think the first question is - do we want to emulate or copy the
Jupyter approach in Leo (as is unavoidably the case with org mode), or
do we want to integrate Leo with Jupyter, so that Leo becomes an
alternate interface to real Jupyter notebooks.  I'm convinced we want
to go the second route, that's the only way to get all the power of

Jupyter in terms of its integration with other systems.

​Yes.  Emulating jupyter is hopeless because we will never be able to keep up.  Leo's jupyter ​importer is only a first step towards the second route.

I know Leo can read, so some degree, Jupyter notebooks, but I'm
wondering if we need to back off a step and connect Leo to the "Jupyter

​ ​

kernel" so that reading notebooks, and executing cells and caching

​ ​

results etc. is all done by Jupyter.

​That's a great idea.  It should be fairly easy to do.  Actually, I suspect I've already done it somewhere ;-)

The big difference here is that when Leo reads a notebook and makes it

​ ​

part of the Leo outline everything's in the Leo system, vs. displaying

​ ​

a Jupyter notebook, but maintaining links back to the Jupyter

​ ​

environment, for cell execution etc.  Actual implementation might be

​ ​

some kind of hybrid between reading in to Leo and interacting with

​ ​

Jupyter.  As opposed to the other extreme, storing all state in Jupyter

​ ​

and reading it constantly.

​Very interesting.  ​I think your goal is right on target, namely to use Jupyter for as much as possible.

So, if I was working on Leo / Jupyter integration I'd start by writing
a non-Leo Python script that programatically created a Jupyter notebook

​ ​

(using Jupyter libraries) which incorporates at least three languages,

​ ​

say Python, markdown, and R or javascript, renders cells, observes data

​ ​

exchange between languages, etc.  From that Jupyter API perspective, I

​ ​

think full featured integration into Leo will be easier.

​I'm always a big fan of this kind of rapid prototyping.  I know from writing the jupyter importer that jupyter notebooks are just json files. So the first question that comes to mind is whether using the jupyter api is really easier than writing json "by hand".  Having said that, perhaps you are thinking that using the jupyter api in the prototype is an essential part of the prototype.

​

Anyway, just thinking aloud, and maybe all "well, duh" for those who've

​ ​

thought more about it than I have.

​Not at all a "duh" for me.  These are great ideas.

Edward

[Terry Brown]

On Thu, 14 Sep 2017 10:42:06 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

> ​I'm always a big fan of this kind of rapid prototyping.  I know from
> writing the jupyter importer that jupyter notebooks are just json
> files. So the first question that comes to mind is whether using the
> jupyter api is really easier than writing json "by hand".  Having
> said that, perhaps you are thinking that using the jupyter api in the
> prototype is an essential part of the prototype.

Right - my thought is that we want to be able to execute any Jupyter
cell, be it in Python or javascript or R or Go or whatever. And making
that happen, from software installation to invocation to exposing
variables across environments, should be Jupyter's problem, not ours.

In that context, I don't see that Leo needs to parse notebook JSON. It
needs to make Jupyter read and manipulate notebooks, and having done
that, I'm sure it can get/set the source part of cells and get/display
the result part of cells.

Cheers -Terry

[Edward K. Ream]

On Thu, Sep 14, 2017 at 11:03 AM, Terry Brown <terry...@gmail.com> wrote:

Right - my thought is that we want to be able to execute any Jupyter
cell, be it in Python or javascript or R or Go or whatever.  And making
that happen, from software installation to invocation to exposing
variables across environments, should be Jupyter's problem, not ours.

​Ok.

​Funny how my memory works.  I remembered that I have a j.bat file that starts the jupyter server.  It just executes "jupyter notebook".  This serves jupyter pages on http://localhost:8888/

So a simple http client should be able to access the data as the jupyter web page at http://localhost:8888/tree does.

There may be other ways to access this data, but I would imaging that the web-based approach would be most natural.

What do you think?

Edward

[Terry Brown]

On Thu, 14 Sep 2017 11:21:03 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

Don't know how much of Jupyter's API it exposes through the web. I
think interacting with it programatically via the user facing HTML,
while possible, would be fiddly and slow. I guess it could have a JSON
over HTTP kind of API that might be quicker, purely speculation, may
not exist. But from Leo's point of view I was assuming using a Python
API.

import jupyter
nb = jupyter.load_notebook('...')
nb.cells[3].execute()
p.b = nb.cells[3].result.as_text()

or whatever, just making stuff up, it's easier that way ;-)

Cheers -Terry

[john lunzer]

I was always under the impression that ZeroMQ was the way that Jupyter front-ends communicated to the kernels. I'm not entirely sure how the web-interface fits into that.

That said, this all sounds reasonable. There is really no way to compete with Jupyter, and also no reason to compete. Leveraging all the great work being done makes sense. 

[Edward K. Ream]

On Thu, Sep 14, 2017 at 11:39 AM, Terry Brown <terry...@gmail.com> wrote:

But from Leo's point of view I was assuming using a Python
API.

​Ok.  Just checking. Are you planning to do a prototype?

Edward

[Offray Vladimir Luna Cárdenas]

Happy to see this direction. I think that the best approach to make Leo
and Jupyter collaborate without one becoming the other.

As I have said before, the idea is to go from files to services and for
that, I would use the Jupyter Kernel protocol to communicate with Leo,
making nodes in Leo behave as Jupyter cells, which means that they can
send calculations to the Jupyter Kernel and get results back that would
be "output" nodes in Leo. I would start with the Jupyter Client
documentation [1], making Leo behave as one of such clients.

[1] http://jupyter-client.readthedocs.io/en/latest/

Cheers,

Offray

[Edward K. Ream]

On Thu, Sep 14, 2017 at 12:02 PM, Offray Vladimir Luna Cárdenas <off...@riseup.net> wrote:

Happy to see this direction. I think that the best approach to make Leo

​ ​

and Jupyter collaborate without one becoming the other.

​I'd call it the only approach ;-)​

As I have said before, the idea is to go from files to services and for
that, I would use the Jupyter Kernel protocol to communicate with Leo,

​ ​

making nodes in Leo behave as Jupyter cells, which means that they can

​ ​

send calculations to the Jupyter Kernel and get results back that would

​ ​

be "output" nodes in Leo. I would start with the Jupyter Client

​ ​

documentation [1], making Leo behave as one of such clients.

[1] http://jupyter-client.readthedocs.io/en/latest/

​Thanks for an excellent high-level summary.

Edward

[Terry Brown]

On Thu, 14 Sep 2017 11:56:58 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

I think my meager Leo time would be better spent on leo_edit_pane and
maybe the QDock thing, so no, not any time soon. John and Offray's
pointers look very valuable though.

Cheers -Terry

[Edward K. Ream]

On Thursday, September 14, 2017 at 1:04:10 PM UTC-5, Terry Brown wrote:

>>Are you planning to do a prototype?

> I think my meager Leo time would be better spent on leo_edit_pane and maybe the QDock thing, so no, not any time soon.

I plan to do a prototype soon.  A simple "hello world" communication between a kernel and Leo should be a good start.

Edward

[tfer]

There is an interesting multi-kernal front end other than Juypter you might want to check out, Beaker.  It is a Java app with a wrinkle of it's own, since let's you bring in kernals from different languages, it bridges the languages with a common data format that can be quickly be made into the native format of the language of the current cell.

Tom

[Edward K. Ream]

On Mon, Sep 18, 2017 at 10:03 PM, 'tfer' via leo-editor <leo-e...@googlegroups.com> wrote:

There is an interesting multi-kernal front end other than Juypter you might want to check out, Beaker.  It is a Java app with a wrinkle of it's own, since let's you bring in kernals from different languages, it bridges the languages with a common data format that can be quickly be made into the native format of the language of the current cell.

​How will this help Leo interface with Jupyter notebooks?

Edward

[Offray Vladimir Luna Cárdenas]

I don't think it will. At least in the short term. I would prioritize Jupyter kernel communication via the protocols I linked in other mail, making Leo a Jupyter client, and after that I would try to implement a abstract data layer that could store computations from several languages.

Cheers,

Offray