I'm still alive! Lets talk about ShareJS going forward

[Joseph Gentle]

I was just resting.

Sorry for ignoring everyone's emails for the past few months. I burned
myself out at lever and I needed some space away from ShareJS to
figure out what I actually want to be spending my time on. Its sort of
startling seeing the data in github's tracker - I wrote almost no code
for 6 months.

I'm hoping to (slowly) go through the backlog of sharejs related
emails and look at some of the issues. Feel free to email me again if
its important.

Thankyou to Nate for keeping everything running in the past few months
and everyone who's contributed patches to fix long running issues.

I know Friedel offered to take over maintaining ShareJS. Its a really
generous offer and I'm really sorry I haven't gotten back to you. I
appreciate it. We should chat at some point over skype about it all.


I think ShareJS has become two things:

First, it was a simple library for doing text based OT in nodejs. The
still not updated website still shows this. The examples from 0.6 are
really small and simple.

When I started at lever, we saw the opportunity to adapt sharejs into
something more - a distributed OT-based database. The database is
powerful and has lots of features for projections and queries and
stuff. But I think its a different use case. I like the code sharing
(and there's a lot of it), but I think the two libraries have
different use cases and different audiences.

Going forward, I think I want to split the project back into those two parts.

The small simple collaborative text editing library will be loosely
based on an updated version of the old sharejs 0.6 code. I'll make the
API a lot closer to 0.7, but keep the simplicity, merge in the cursor
code and add first class support for rich text. The cursor code was
held out of 0.7 because it doesn't handle multiple servers. The tiny
library for text editing won't support multiple servers anyway (maybe
you can shard based on document id or something). No more redis, no
more complicated nested options objects. It'll use browserify,
websockets, promises and dump coffeescript (with some ES6 maps and
sets). This will be my baby - I still want that light little library.
I'm kind of amazed that nobody has built that yet.

The second project will be the sharded OT database that sharejs 0.7
has become. We're talking about renaming it to sharedb. (I own
sharedb.org). Every document is a JSON document, with embeddable
subdocuments for text editing. We keep support for multiple servers,
live queries, projections, all the good stuff. Nate will be benevolent
dictator of that project, which will make it easier to keep everything
working smoothly with DerbyJS.

This isn't an overnight thing. We're all really busy shipping code.
I've gone back to teaching a few days a week and I'm writing an ios
game on the side. But I don't want to leave sharejs where it is at the
moment. ShareJS is too useful a tool to leave unfinished.

I'll hop in office hours at some point and we can talk about it in
person. (We need to schedule one which isn't at 4am Sydney time.)

-J

[Devon Govett]

Hey Joseph,

Glad to have you back, and happy to read a bit about the future direction of the project!

I think, if there are going to be two separate projects going forward, it would be best for everyone involved if as much code as possible could be shared between the projects. For example, the direction of moving the OT types out of ShareJS into their own modules is a good start. If the two projects could share that code, it would be awesome. Additionally, the networking stuff (client-server protocol) seems like it could be shared as well. The backend for transforming ops and saving them to a database could also be common.

Come to think of it, I actually think it may be better to start from the current ShareJS and livedb and make them more modular rather than splitting the project entirely in two. Move the redis and in process driver stuff from livedb into separate modules that can be plugged in. Move the memory store into a separate module, like livedb-mongo is. Move the query stuff into a separate plugin as well, for both ShareJS and livedb. Move the rest server out of ShareJS and into a separate module. Same for projections. Cursor support could also be added as a plugin, configurable to support whatever OT type you’re using (e.g. text or JSON), or even separate modules for each.

ShareJS would end up being just the client and server of the basic network protocol, only supporting fetching and subscribing to documents, and submitting ops via livedb. The protocol could be extendible so that other things could be added, such as queries, projections, and cursors. Users could use browserify to build the ShareJS client and each plugin they need in their application. Livedb would become just the basis of an OT backend, with support for multiple plugins for databases and drivers (already almost there). Out of the box it couldn’t do anything by itself - you’d need to add a backend and a driver at the minimum. It could be made just a bit more extendible to support things like queries and projections as well.

This looks like a lot when I write it out, but I think we’re already pretty close to what I described, at least on the livedb side (drivers and backends are already basically plugins). If we went with this modular design, and you wanted to build your small text editing library, you’d just require livedb with an in process driver and whatever backend you wanted, the ShareJS client/server, a plain text or rich text OT type, and cursor support. It would end up lighter weight than currently since you wouldn’t require anything you don’t need (e.g. JSON OT, redis, queries, and projections). You could even publish this as a module for others to use with zero configuration (and also a good example for people who want to build their own config).

I think this could be a lot better overall for everyone, since each project could build their own combination of only the pieces they need. I’m happy to help with this if you would like to go in this direction. What do you think?

Devon

> --
> You received this message because you are subscribed to the Google Groups "ShareJS" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sharejs+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Joseph Gentle]

I really like that idea... I guess I'm just a little nervous about
putting in all the plugin extension APIs we'll need. When I
implemented stuff, lots of features actually needed code changes in
lots of places. We could probably go down the list of features and
make exactly the set of extension APIs we'll need, but I don't know
how general the result would be.

For example queries. Queries change the client's working document set
in a way that the client can't predict when it fires the query.
Originally a client could subscribe to a query, then unsubscribe from
one of the documents in the query. Then ... what? When the client
reconnects what should happen? That simple fact resulted in tons of
changes & reworking. (Now iirc subscribing to a query just gives you
access to a live-updating list of query results and its up to the
client to subscribe / unsubscribe from them).

Projections were pretty easy because of how I implemented them, but
Nate is keen to move away from the current implementation. Moving away
from the named pseudo-collections would require the client to know
which projections its looking at, and we'll need to have some merged
view of documents where you have multiple projections. (And the
projections could overlap).

I like the plan in principle but I'm worried that making features like
projections and queries optional is probably a lot harder than just
having two versions the code. I definitely want an upgrade path of
sorts from the text / richtext OT to the full sharedb database, but...
yeah... :/

If you're keen to help out, maybe pull up sharejs + livedb, pull out
one of the features and have a play with what that extension API could
look like. Its been awhile since I looked at the code but there might
be some decent answers sneaking around somewhere in there.

-J

[Devon Govett]

Yeah I’m not completely sure what the requirements for an extension API would need to be yet either, but I think it should be possible. I’ll have a lot more time in a few weeks, so I’ll play around with this and see what I come up with then.

Devon

[Joseph Gentle]

Awesome, looking forward to it.

[Domagoj Kriskovic]

Hello,

I have one question... I want to use sharejs as "simple collaborative text editing library" but I see that current development is more oriented for creating OT based database... 

So can you advise me which version should I use? 

Should I make it work with newest version on master branch or 0.6.X would be more stable for my use-case?

Thanks,

Domagoj

[Joseph Gentle]

0.6.X is probably still more stable and better documented. I'll have a
better story around this use case soon - but for now 0.6.x is a good
choice.

-J

[James Healy]

We (theconversation.com) are using 0.6 (with a few small tweaks in our fork) for a simple collaborative text editor with a few hundred users and it's rock solid.

We haven't tried 0.7 and are in no hurry to while it's in a state of flux and 0.6 continues to serve our needs.

James

--

[Domagoj]

Ok, great, thank you!

[Marcel Klehr]

Joseph Gentle wrote:

I'm kind of amazed that nobody has built that yet.

 

I know I'm a bit late to the party -- I've been away, too. Anyway, I am building something like that. See https://github.com/marcelklehr/gulf

I'm looking forward to see in which direction shareJS is moving and what the future holds.

OT FTW!

Marcel

[jiten shah]

James - You don't see performance issues w.r.t. long-polling ? ShareJS 0.6 does not support websockets right.  

[James Healy]

On 21 Jul 2016 18:07, "jiten shah" <jiten...@gmail.com> wrote:
>
> ShareJS 0.6 does not support websockets right

It might, but we're using browser channel ( which is sort of long polling).

> You don't see performance issues w.r.t. long-polling ?

No, although we have a peak of only 30-40 concurrent users.

If anything, I feel like browser channel gives us fewer headaches than a websocket based approach. It works on all browsers, and has no issues with the load balancers that sit between users and our server.

James

[Joseph Gentle]

At lever we had some problems with how browserchannel interacted with
AWS's elastic load balancer. In any case it should be pretty easy to
get sharejs 0.6 working with websockets. Browserchannel emulates the
websocket API after all.

-J

[jiten shah]

Thank you James and Joseph.