A better way to send code to language processors

[Edward K. Ream]

This post shows a button that sends code to rust. Thomas adapted it for Graal/JS here.

My original code contained a hack, which Thomas naturally used. The hack just removes @language directives.

My example was misleading. There is a much better way. Instead of the hack:

lines = p.b.split('\n')
prog = '\n'.join([
    line for line in lines if not line.startswith('@language')
])

scripts should do this instead:

s = c.atFileCommands.stringToString(
    p, p.b, forcePythonSentinels=False, sentinels=False)
prog = s.replace("\r\n", "\n")  # Use brute force.

This removes all Leo directives and also supports @language and section references.

I found this code in one of the helpers of g.getScript. It's not so easy to use g.getScript directly, as you can see for yourself if you are interested.

Edward

[Marcel Franke]

Both solutions seems to be flawed as none of them is filtering by language.

Is this uncommon in leo-community? I often have nodes with multiple mixed languages or mixed plaintext and code.

Both of those solutions would mix this plaintext with code or code of different languages together it seems.

[Edward K. Ream]

On Mon, Mar 23, 2020 at 11:09 AM 'Marcel Franke' via leo-editor <leo-e...@googlegroups.com> wrote:

> Both solutions seems to be flawed as none of them is filtering by language.

Glad you brought this up.  Using c.atFileCommands.stringToString means implies that everything in p.b applies to the desired language. In contrast, using g.getScript calls g.extractExecutableString, which does the filtering you desire.

Depending on your outline, and your outline's settings, using stringToString may allow the user to avoid @language directives, while using  g.extractExecutableString is more flexible. There are pros and cons each way.

> Is this uncommon in Leo community? I often have nodes with multiple mixed languages or mixed plaintext and code.

It's common within LeoDocs.leo. There, @language rst and @language python are often intermixed.

Edward

[Thomas Passin]

I often use mixtures of code and non-code blocks.  If there are different kinds of code blocks, though, there can be problems if you want to execute them.  If you execute each code block with a different language processor, you each block has to be independent of the others.  They can't share constants, function returns, class definitions, etc.  If that's the case, you might as well put each block of a different programming language into a different node, and not mix them.

Also, you couldn't break code in the same language across several code blocks, because how could the processor know if you want one to extend the previous one or be independent?

I had to face this issue as I worked on Viewrendered3.  Currently, you can render mix and colorize blocks with different languages, but for execution, all the code blocks need to be for the same language (right now limited to only python, but that should be changed in the future).  My desire was to support literate programming (that is, to be able to insert commentary in a nicely readable way), so I also had to provide for mixing code and non-code blocks in a node.  I also wanted the code to be executable.  In the current implementation of VR3, all code blocks in a node or subtree get merged before execution.

I would be willing to entertain requests to change this behavior, but I'd have to be convinced that it would be worth the effort!

[Marcel Franke]

Am Dienstag, 24. März 2020 01:32:29 UTC+1 schrieb Thomas Passin:

I often use mixtures of code and non-code blocks.  If there are different kinds of code blocks, though, there can be problems if you want to execute them.  If you execute each code block with a different language processor, you each block has to be independent of the others.  They can't share constants, function returns, class definitions, etc.  If that's the case, you might as well put each block of a different programming language into a different node, and not mix them.

 For an easy setup, yes. Complex setups could use shared datastores or just pipe results between blocks as input for the next block. Jupyter Notebooks and org-modes babel are doing this.

Also, you couldn't break code in the same language across several code blocks, because how could the processor know if you want one to extend the previous one or be independent?

 Just mark them. Add something like a @script-directive or utilize @other. Of course does the processor need information about the structure, but delivering this is simple.

I had to face this issue as I worked on Viewrendered3.  Currently, you can render mix and colorize blocks with different languages, but for execution, all the code blocks need to be for the same language (right now limited to only python, but that should be changed in the future).  My desire was to support literate programming (that is, to be able to insert commentary in a nicely readable way), so I also had to provide for mixing code and non-code blocks in a node.  I also wanted the code to be executable.  In the current implementation of VR3, all code blocks in a node or subtree get merged before execution.

 If you wanna build a complex workflow from an outline of mixed languages, then you need the tools and information for this. Leo itself does not offer them.

Notebboks have quite solved this problems and should give you an understanding on how to solve this.

So what exactly is Viewrendered3? Is this some kind of notebook-interface for an leo-outline?

[Thomas Passin]

Well, yes, in my response I avoided more complicated possibilities.  And I wrote about Viewrendered3 because I recently completed this plugin (really a beta version).  VR3, an extension of the Viewrendered  plugin, aims to render various node types, such as Markdown and Restructured Text.  As a convenience, it allows for rendering and execution of subtrees, and for inserting printed output into the rendering.

The point was to have a relatively simple rendering ability.  It was never intended to have complete notebook-like functionality.  For that, you'd want to start with such a goal in mind, and it would be much more complex. I prefer to avoid running servers if it's not absolutely necessary, and in this case I was able to avoid one.  For simplicity - for the user - I chose not to add new kinds of Leo-style directives. And I didn't want to extend MD or RsT with new commands or syntax, because the existing processors for those languages wouldn't understand them.

For notebook functionality, go for it!  Although I think that Edward's direction of a two-way bridge between Leo and, say, Jupyter, will be a better way to go.  Let the notebooks be good at what they are good at,  and let Leo be good at what it does well.