Live editing, operational transforms, Phoenix channels

[Stian Håklev]

I wonder if anyone here is interested in live collaborative editing? I'm building a collaborative environment for students. Currently we have a great simple chat implementation with history (Postgres) and presence (an agent), and even an admin panel where the teacher can follow the different groups (all rooms also broadcast to an admin channel). For collaborative editing, we embed an Etherpad in an iframe (later we'll also embed a wiki for them to work on their final project). 

However, Etherpad is quite heavy (slow to load, heavy on the server), and it's also messy to have a bunch of prompts and then trying to extract certain responses etc. I would much rather have a bunch of text boxes, one per prompt, that they can edit collaboratively. I have read a bit about operational transforms, and it seems that it would be an ideal fit for Elixir and Phoenix Channels.

I was looking at what currently exists - share.js looks like it's the most reliable and well-used option. There's also another library called gulf. All these are written in node, with javascript front-ends. There are some implementations of OT in Erlang, but they seem quite "toy", for educational purposes -- and anyway, implementing a good javascript client is key. So I am thinking about trying to port one of the backends (share.js, gulf or another one) to Elixir. 

It seems to me that removing the logic for handling channels, subscriptions etc, the actual meat of the matter is only around 400 lines of js, probably less in Elixir. Porting a backend would mean we could use an existing, tested front-end (and my Elixir is much stronger than my JS skills!).

Initially I'm mostly interested in only pure-text, small text pieces (not long documents), and few users - the simplest use case. However, if we could get that to work, I'd love to expand of course.

Reasons for not using existing node backends:

- would be fun if Elixir was faster/more scaleable

- integrates better with existing apps (ie. I want to be able to push/read text using Phoenix, react to events etc)

- no need to run additional servers

I'd love to hear if anyone has ideas, know about other projects/code that can be used, are interested in collaborating on something like this, etc. If we made something solid (even simple) it could be another killer feature for Elixir/Phoenix. (Imagine a simple OTP app that you can just add to your Phoenix dependencies, and you've got collaborative editing of any component you want). 

Stian

--

http://reganmian.net/blog -- Random Stuff that Matters

[pj.beta...@gmail.com]

I'm. Very! :)

This is my second day with Elixir/Phoenix coming from Meteor (javascript real-time framework) and your idea has the perfect timing for other people like me that are using nodes stack but evaluating Elixir/Phoenix.

Let me say that like some other features that I though were unique from Meteor as hot code reloading I'm sure that Elixir/Phoenix will shine in OT and collaborative editing (I was really surprised to see that hot code reloading here is immediate and in meteor it takes a few seconds).

Unfortunately I cannot offer any help as I'm still trying to learn Elixir...

[gilber...@gmail.com]

Hi Stian, have you made any progress on this front? I am also interested. If you've got a repo somewhere, perhaps I can be of help.

[Justin Workman]

This is something I'm definitely interested in for future projects. It seems Phoenix would be a perfect fit, and I like your idea of having the collaborative editing plugin handle the OT logic and leave the rest up to Phoenix channels and a JS client.

[Stian Håklev]

Hi guys, fun to get some interest in this long after the initial messages was posted.

I did indeed spend two afternoons porting javascript code to Elixir. It was an interesting challenge in itself, much of it could be rewritten almost word for word, but there were some bigger functions which used functions as closure to keep state which I had to rewrite quite a bit. Part of the problem is that I don't understand all the details of what the code is doing (I get many of the parts, but not all the details). 

I began with ot_text, since it seems like most of the logic for handling pure text operations was contained here (https://github.com/ottypes/text/blob/master/lib/text.js). Much of the other code is handling connection with the client, persistence etc, which it would probably make sense to completely replace with Phoenix channels etc.

Luckily ot_text has an extensive test suite and even a fuzzer (generative testing). However, of course, in JS... I looked at translating the test suite to Elixir too, but it seemed like a lot of work, and of course risking that I would make errors in the translation (now I need a test suite for my test suite...). Instead, I played around a bunch with whether it would be possible to call Elixir functions from JS. A big advantage is that all the individual functions are pure (although often implemented in a stateful way), taking very simple inputs that can be represented in json, and returning the same. 

My idea was to use RPC to have a simple JSON wrapper that would simply make calls to an Elixir server whenever a function got called, and return the result of that call. My problem was that almost all Node IO operations are asynchronous - but I wanted to fool the test suite into thinking these are synchronous operations. I eventually found a sync-http module, which actually works - however it has to do a lot of trickery to make http sync, and as a result, is extremely slow. 

It works though, and it found a bunch of bugs. It's a bit painstaking to fix these, as it's requires kind of tracing through the Elixir loops and reduces and seeing what goes wrong (often simple things that I missed when translating). There are still some functions that fail tests. 

However I also wanted to explore improving the speed of the test suite, perhaps using something like this https://github.com/chernser/erlnode. I think there are many JS libraries which have a backend and a frontend, where it might make sense to rewrite the backend in Elixir, so having a generic way of running JS test suites against Elixir code might actually be useful.

Anyway, I've pushed the code I have so far to Github, the translated module is here: https://github.com/houshuang/ot_text/blob/master/lib/text.ex

I'd love to push this forwards. Haven't had much time to work on it lately, but let me know if you'd like to help.

I'm also interested in looking at ProseMirrors backend, they are taking a different approach than operational transforms (http://marijnhaverbeke.nl/blog/collaborative-editing.html), and was just made open source, but I guess the API will be very much in flux the next six months.

Stian

--
You received this message because you are subscribed to the Google Groups "phoenix-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to phoenix-talk...@googlegroups.com.
To post to this group, send email to phoeni...@googlegroups.com.
Visit this group at http://groups.google.com/group/phoenix-talk.
To view this discussion on the web visit https://groups.google.com/d/msgid/phoenix-talk/03608fa1-8b50-4c9f-96a1-7a78eb81334a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[gilber...@gmail.com]

What a coincidence, I also plan to use Pheonix with ProseMirror! In fact this is the main reason I posted to this thread :)

Upon further investigation, it seems that ProseMirror takes the same approach as FirePad, which is merging on the client side instead of the server side. Knowing that, I would imagine the only missing piece is a central server that accepts/rejects user input.

Thoughts?

[Stian Håklev]

Not sure how much work happens server-side, couldn't even find a server component on the github repo, and it doesn't seem like collaborative is working yet (the code in collab/ is very minimal). I think we should probably just keep an eye on it for now. 

Stian

[gilber...@gmail.com]

It's not a server component, but I did find a "Dummy Server" in a test file :) I imagine a "real" server would have to implement something similar:

https://github.com/ProseMirror/prosemirror/blob/master/test/browser/test-collab.js

As I mentioned in my previous post, FirePad also takes a client-merging approach. This is promising, considering FireBase is not "OT-aware". I imagine we can build a library that partially mimics firebase for phoenix / ProseMirror, which is probably easier than building a full-fledged OT library.

[Kevin Montuori]

>>>>> "sh" == Stian Håklev <sha...@gmail.com> writes:

sh> I wonder if anyone here is interested in live collaborative
sh> editing? I'm building a collaborative environment for students.

I do some light duty (small chunks, ~3 users/each) real-time markdown
editing and preview with Nitrogen (http://nitrogenproject.com) and
mnesia. The amount of custom JS required is minimal: Nitrogen provides
well tested actions that are easy to wire in.

I don't see any particular reason why Nitrogen and Elixir couldn't be
compatible (but I also don't see any reason to use Elixir there when
Erlang fits the bill so well).

k.


--
Kevin Montuori
mont...@gmail.com

[Stian Håklev]

Very cool - do you have links to any source code? The nice thing about Elixir and Erlang is that you don't have to choose - perfectly easy to reuse code from either in either... 

[Kevin Montuori]

>>>>> "sh" == Stian Håklev <sha...@gmail.com> writes:

sh> Very cool - do you have links to any source code?

Unfortunatley the work I've done around this is proprietary. If you
take a gander at the demos though it should be obvious how to piece a
PoC together in short order.

[kdah...@gmail.com]

I would be very interested in porting etherpad to a phoenix/elixir backend too, I found one project but have not tested it yet https://github.com/dresden-weekly/alkyl