How to add support for other languages?

[Louis Abraham]

I am totally new to this awesome project (lured there by pyodide).

Some time ago I played with iocamljs (demo) but the interface was kind of hackish, and it was not possible (I think) to save modifications directly as the web page had to read an .ipynb file.

I would very much like to make OCaml work with iodide, and I'm ready to devote some of my time to this project.

I am not very familiar with react but I don't think it is important for what I want to do.

Is there some API to implement? Could somebody give me guidance for this project?

[Michael Droettboom]

Thanks for reaching out.  This sounds like an interesting project.

Ultimately, you're going to need create a Javascript function that takes a string of OCaml code, runs it, and returns something useful.  That's the hard part.  Pyodide works by compiling CPython to WebAssembly using emscripten.  I don't know if a similar approach would work for OCaml or not  There may have been other folks who have "cross-compiled" OCaml before who can give you some pointers.

Once you have that, the interface to integrate it with Iodide is fairly simple.  You just need a bit of metadata in JSON that you run in a "plugin" cell in iodide.  Here's the metadata for pyodide:

```

{
  "languageId": "py",  // a unique identifier for the language
  "displayName": "python",  // name to display to the user
  "codeMirrorMode": "python",  // the name of the language in the CodeMirror editor
  "keybinding": "p",  // keybinding to create language cell
  "url": "https://iodide.io/pyodide-demo/pyodide.js",  // URL to Javascript that bootstraps the Wasm
  "module": "pyodide",  // The module (namespace) in which to find the evaluator
  "evaluator": "runPython",  // The name of the evaluator function
  "pluginType": "language"
}

```

Feel free to reach out again here or on our gitter channel with further questions.

Mike

--
You received this message because you are subscribed to the Google Groups "iodide-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iodide-dev+...@googlegroups.com.
To post to this group, send email to iodid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/iodide-dev/33e0011d-d41d-475f-8ed8-4fa5b2e9865c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Michael Droettboom

Data Engineer

Mozilla Corporation

[Louis Abraham]

Thank you for this quick reply.

Ultimately, you're going to need create a Javascript function that takes a string of OCaml code, runs it, and returns something useful.  That's the hard part.  Pyodide works by compiling CPython to WebAssembly using emscripten.  I don't know if a similar approach would work for OCaml or not  There may have been other folks who have "cross-compiled" OCaml before who can give you some pointers.

In fact, the project I quoted, iocamljs, works by writing an IPython kernel in OCaml and compiling it for the browser with js_of_ocaml.

So, I think it's not very complicated to plug js_of_ocaml into iodide.

Aren't there some more advanced features to implement like autocompletion (in the style of https://langserver.org/)?

Best,

Louis

[Michael Droettboom]

Yes, we need to implement autocompletion for custom languages, but Iodide doesn't have that yet.

I've filed an issue for that here:

https://github.com/iodide-project/iodide/issues/804

Mike

[Louis Abraham]

Hi, 

I have made good progress on the

project. I have a functional REPL, but also some doubts on the right way to handle the output. 

There are 3 output streams:

- the eval output 

- stdout (different from the eval)

- stderr

Is there a way to separate the three in the output?

If there is a way to output html, I can hack something to put tags like <eval>, <stdout>, <stderr>, and create a CSS styling (gray for eval, black for stdout, red for stderr). 

Best, 

Louis 

Sent from my mobile. Please excuse the brevity, spelling and ponctuation.

[Michael Droettboom]

Iodide currently has the following APIs:

- `iodide.output.text`: Outputs a string.  Unfortunately, there's no way to distinguish between out and err here.  (But that's probably a simple thing to improve).

- `iodide.output.element`: Creates a DOM element in the output frame and returns it to the caller.  You could then attach whatever DOM nodes you want and basically support anything.

How you want to handle this is basically up to you.  You could file a pull request to Iodide to fix (1), or just use (2) which would be a little less convenient, but work today.

Cheers,

Mike

[Louis Abraham]

How can I build the latest version?

I tried `git clone` then `npm install` and `npm run build` but it seems the resulting iodide.dev.js does not work like https://iodide-project.github.io/dist/iodide.pyodide-20180623.js

Best,

Louis

Le 30 août 2018 à 00:16, Michael Droettboom <mdroe...@mozilla.com> a écrit :

Thanks.  I'll update the pyodide example.  It exists in a couple of places, and I neglected to update it.

No, the newlines things is a bonafide bug.  I just filed a PR here: https://github.com/iodide-project/iodide/pull/823

On Wed, Aug 29, 2018 at 6:00 PM Louis Abraham <abrah...@gmail.com> wrote:

Oh thanks... I'll try later when I have a computer :)

Not an excuse but I think I used the pyodide notebook as template, you might want to update it.

Is the newlines behavior when the evaluator returns a string normal then? 

Sent from my mobile. Please excuse the brevity, spelling and ponctuation.

Le mer. 29 août 2018 23:53, Michael Droettboom <mdroe...@mozilla.com> a écrit :

Seeing your example now, I think this is due to you using an old version of iodide. Update your tag from pyodide-20180420 to pyodide-20180623 and iodide.output.text should be available to you.

Mike

On Wed, Aug 29, 2018 at 4:40 PM Louis Abraham <abrah...@gmail.com> wrote:

You were very clear.

I forgot to say I had errors when trying to call `iodide.output.text`. 

Then I tried to print the objects and it didn't work.

Best,

Louis

Sent from my mobile. Please excuse the brevity, spelling and ponctuation.

Le 29 août 2018 10:14 PM, "Michael Droettboom" <mdroe...@mozilla.com> a écrit :

Sorry, that's my fault for not being clear.  `iodide.output.text("String")` should output that string to the output area (rather than to the Javascript console).

Mike

On Wed, Aug 29, 2018 at 3:59 PM Louis Abraham <abrah...@gmail.com> wrote:

I must admit I am by no means a js expert, let alone when it is called from an OCaml code compiled in js.

However, when I call `console.log('iodide')` or `console.log('window.iodide')`, I see an object.

But when I call `console.log('iodide.output')`, it displays "undefined".

Where is my error?

For the moment, I'll just output a string (which is already challenging).

IMO, another solution would be to add an output handler (like https://github.com/iodide-project/pyodide/blob/master/src/pyodide.js#L216).

Cheers,

Louis

[David Sarma]

Following this as a guide, I've added a few other languages:

Ruby (using Opal): https://codepen.io/ds604/pen/6cef7f732750cb2bde469b5ff387a320

Lua (using Fengari): https://codepen.io/ds604/pen/d9791eed4e1ce19e11fb0f3c71000d72

Assemblyscript (the Typescript-like language that compiles to WASM): https://codepen.io/ds604/pen/432d293df115432859f0af62e69d2e64

Gist with these three: https://gist.github.com/ds604/b552cc4b0bc43dbff997c955eb2b6bba

I'm not sure if these belong as demo notebooks, since they're not really developed beyond just making the eval functions for language plugin work, but maybe they might be helpful for other people. 

I'm meaning to add self-host Clojurescript as well. (In part I'm following along with what's offered by Klipse: https://github.com/viebel/klipse)

##

As an aside (maybe this belongs in a different discussion, but I saw that ease of use for different use cases is mentioned as a primary goal), I went through the few different means of saving out the notebooks and displaying them. What I found worked best for me, rather than Exporting the notebook (having it download to the computer), I prefer the Export to Clipboard option. I like to use Codepen, so that I have visual feedback on the contents of the notebooks. So I made a pared down iodide page, where i paste in the clipboard contents, and it extracts the jsmd part and renders the notebook (it's in the codepen links i have above). But I guess that's a little bit of a workaround to if the "Export notebook" were exporting the HTML to the clipboard (as opposed to downloading the HTML to the computer).

Anyway, great project! The R Markdown-like JSMD format, numpy and matplotlib running client-side, access to all things javascript in a notebook format... it's finally all the good things in one place!

[Michael Droettboom]

David,

Thanks so much for this.  This is fantastic -- increasing our language support is always a good thing.

I'm not opposed to these becoming demo notebooks.  I think they are good enough starting points that making them easier to find would be beneficial.  Would you want to file PRs against the `iodide-project/iodide-examples` repository?  I'd also like to see a [documentation page](https://github.com/iodide-project/docs/issues/10) listing all of the languages that work, along with some notes about their status / speed / limitations etc.

Can you confirm: The Ruby and AssemblyScript examples you posted convert their outputs to Javascript objects, but the Lua one just writes to the console.  Is that correct?  Not a dealbreaker, but if we build a documentation page, it would be good to include that info to manage expectations.

In recent versions of github master, iodide now has support for defining languages in a "built-in" way so that the user doesn't have to bring over the language plugin content themselves.  As these proof-of-concepts mature, it would be great to move these there.  Of course, the way we're currently doing it [may not scale to a lot of languages](https://github.com/iodide-project/iodide/issues/1065).

As for your question about clipboard export/import -- I think that's a great idea.  We are planning in the medium-term future to move to a cell-less editing environment where the jsmd is edited directly.  That should naturally allow what you describe, so that feature will probably come out of that anyway.  In any event, I [created an issue for this](https://github.com/iodide-project/iodide/issues/1064).

Thanks again,

Mike

To view this discussion on the web visit https://groups.google.com/d/msgid/iodide-dev/a74d7d64-d2f5-4e3c-84bf-54cc7d905969%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[brendan c.]

Indeed, amazingly cool David, thank you for contributing!! :-D

To unsubscribe from this group and stop receiving emails from it, send an email to iodide-dev+unsubscribe@googlegroups.com.

To post to this group, send email to iodid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/iodide-dev/a74d7d64-d2f5-4e3c-84bf-54cc7d905969%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Sarma]

Hi Michael, 

Thanks for the response, that's helpful to know. I filed the PR with the two notebooks, one for Ruby and the other for AssemblyScript. I held off on including the Lua notebook since I also realized that I was still linking to my own copy of Fengari.

As to level of interoperability:

AssemblyScript results in a wasm module, so those functions have full interoperability with Javascript.

I have not actually used Opal or Fengari extensively, but my understanding is that Opal offers some level of interop, but it appears to be not quite a seamless experience: http://funkworks.blogspot.com/2015/06/accessing-javascript-from-opal.html. In my experience (of just checking to see how much Ruby works out of the box), I tried to run

"hello!".chars.map{|char| "%08d" % char.unpack("C").first.to_s(2).to_i}

This did not work, because the unpack function is missing. So it would seem that some portions of what someone expecting to just start using Ruby on the web might be not quite there.

Fengari (https://fengari.io/) actually seems to be the more developed in terms of being a full implementation of Lua (as someone using it in another environment would expect), but also providing access to the DOM, and interoperability with Javascript objects: https://github.com/fengari-lua/fengari-interop. I haven't really put in much time trying it out though, so my knowledge is somewhat cursory.

-David