Hello Derby

[Curran Kelleher]

Hello,

I'm getting some traction with Derby and building out an app. I have a bunch of questions I'd like to share.

CDN Content

Is it reasonable to load CSS and JS libraries from CDNs with Derby, or does this go against its core design? For example, I'm including Bootstrap from a CDN. I noticed there is an old project d-bootstrap that loads styles with app.loadStyles(__dirname + '/node_modules/bootstrap/dist/css/bootstrap.min'), but wouldn't this slow down the bundling process and bloat the bundle as compared to using a CDN? Loading the styles from a CDN works, however I can't seem to get the styles from my app to load after the CDN-loaded Bootstrap. I've added detailed comments to derby/issues/223 . Has anyone encountered this?

Dynamic Document References

Is there a way to have a document refer to another document? For example, one document (A) contains some text, and another document (B) contains text that can be interpreted as Markdown. I'd like to implement the ability to embed the text of A inside B by adding some special content to B that refers to A, maybe something like {{{embed(A)}}}. Has anyone done something like this?

Historical Versions

I'd like to add the ability for users to "release" document versions, much like Git tags but for single documents. Then, I'd like to have a list of releases for each document, with the ability to view the document content at each release version. I came across the following resources for this: (mailing list) accessing version history, (ShareDB issue) API for history management. From what I can find, this implementation of "getSnapshotAtRevision" seems to be the closest thing out there to what I am looking for. Is this the best solution right now, or does anyone know of a more up-to-date way to implement historical versions?

Authentication

Are folks using derby-login in production, or rolling your own solution? I would like to use it, but I have encountered issues when trying it out, and I'm quite intimidated by it. The fact that the de-facto example code is over 2 years old makes me wonder if I should really be writing my app based on that code, or if best practices with Derby have changed since then. Can anyone here vouch for derby-login as a good path for auth?

Access Control

I'd like to use sharedb-access for access control. This seems like the best way forward, but I'm a but frustrated that there is no complete app example that uses it. For example, where would "session.isAdmin" be defined? I don't see any mention of "isAdmin" in the derby-login code. I've never implemented authentication or access control before, and I'm a bit confused about how to put all the pieces together. Does anyone know of a full application example that demonstrates how to use sharedb-access?

React on ShareDB

While encountering the oldness of Derby, I've wondered if it might make sense to implement a similar full stack framework on React and ShareDB/Racer. Amelisa looks like a really cool project, but I'm not convinced it makes sense to re-implement the real-time with CRDT. ShareDB seems very mature and reliable as a synchronization engine. Droganov's racer-react looks really cool, but it also looks like a one-off thing that was made quickly, as a proof-of-concept that it is possible. Then there is droganov/rkta, which seems to aspire to be a Derby-like framework, but it has an insane amount of dependencies and hardly any documentation, so I'm not really sure what it is. I guess I'll have to try it out.

Thank you all for working on Derby, it's quite an interesting piece of technology. It seems like the only mature full stack framework that provides proper real-time collaborative features. I'm ready to struggle through the poor documentation because of the amazing features Derby has to offer, and I'd like to contribute back documentation enhancements when I can.

Best regards,

Curran

[Carl-Johan Blomqvist]

Have some answers inline below.

On Thursday, April 28, 2016 at 2:11:26 PM UTC+2, Curran Kelleher wrote:

Hello,

I'm getting some traction with Derby and building out an app. I have a bunch of questions I'd like to share.

CDN Content

Is it reasonable to load CSS and JS libraries from CDNs with Derby, or does this go against its core design? For example, I'm including Bootstrap from a CDN. I noticed there is an old project d-bootstrap that loads styles with app.loadStyles(__dirname + '/node_modules/bootstrap/dist/css/bootstrap.min'), but wouldn't this slow down the bundling process and bloat the bundle as compared to using a CDN? Loading the styles from a CDN works, however I can't seem to get the styles from my app to load after the CDN-loaded Bootstrap. I've added detailed comments to derby/issues/223 . Has anyone encountered this?

Everything is generated upon requests and put into the public folder. Possibly one can use this folder and put it up upon a CDN. Some tweaking might be needed though to make this work properly.

 

Dynamic Document References

Is there a way to have a document refer to another document? For example, one document (A) contains some text, and another document (B) contains text that can be interpreted as Markdown. I'd like to implement the ability to embed the text of A inside B by adding some special content to B that refers to A, maybe something like {{{embed(A)}}}. Has anyone done something like this?

I haven't heard of anybody doing anything similar (I believe I have been one of the earliest adopters of Derby so I should probably have heard something about it, but of course I'm not sure).

 

Historical Versions

I'd like to add the ability for users to "release" document versions, much like Git tags but for single documents. Then, I'd like to have a list of releases for each document, with the ability to view the document content at each release version. I came across the following resources for this: (mailing list) accessing version history, (ShareDB issue) API for history management. From what I can find, this implementation of "getSnapshotAtRevision" seems to be the closest thing out there to what I am looking for. Is this the best solution right now, or does anyone know of a more up-to-date way to implement historical versions?

As posted elsewhere by Nate (I believe) there's nothing built in but it should be "relatively" simple to implement inverse operations and thus be able to rewind pretty easily. Of course, some nice front-facing interface and a lot of glue code would be needed.

 

Authentication

Are folks using derby-login in production, or rolling your own solution? I would like to use it, but I have encountered issues when trying it out, and I'm quite intimidated by it. The fact that the de-facto example code is over 2 years old makes me wonder if I should really be writing my app based on that code, or if best practices with Derby have changed since then. Can anyone here vouch for derby-login as a good path for auth?

We use derby-login in production. The issue is solved by my comment (i.e. reshuffling the order of using express middlewares, AFAIK). About the old example code, see my reply regarding the status and activity in the community in the bottom.

Note that derby-login only solves auth, but not access control.

 

Access Control

I'd like to use sharedb-access for access control. This seems like the best way forward, but I'm a but frustrated that there is no complete app example that uses it. For example, where would "session.isAdmin" be defined? I don't see any mention of "isAdmin" in the derby-login code. I've never implemented authentication or access control before, and I'm a bit confused about how to put all the pieces together. Does anyone know of a full application example that demonstrates how to use sharedb-access?

In the sharedb-access example, session.isAdmin would be something that you setup by your authentication system - e.g. derby-login. I think you pain point is with derby-login - if you sort that out sharedb-access is pretty straight forward for most scenarios. It's easier to help out with derby-login if you ask specific questions about it and post code, then I'll be able to help out much easier. I'm typically online on gitter so that might be a good place to chat.

 

React on ShareDB

While encountering the oldness of Derby, I've wondered if it might make sense to implement a similar full stack framework on React and ShareDB/Racer. Amelisa looks like a really cool project, but I'm not convinced it makes sense to re-implement the real-time with CRDT. ShareDB seems very mature and reliable as a synchronization engine. Droganov's racer-react looks really cool, but it also looks like a one-off thing that was made quickly, as a proof-of-concept that it is possible. Then there is droganov/rkta, which seems to aspire to be a Derby-like framework, but it has an insane amount of dependencies and hardly any documentation, so I'm not really sure what it is. I guess I'll have to try it out.

As for this question, this makes for a much bigger discussion than your other thoughts/questions/comments. First of all, let's note that I'm not officially speaking for DerbyJS or the Lever organization in any way. I've been following DerbyJS closely though since it's public inception a couple years ago.

So, for your question, let's divide it into some separate topics:

1) What frameworks and libraries should I use for my project?

Obviously, this depends on your project and the needs of your project. Let's not worry about that for a moment, that's after all something I don't know anything about. If we don't worry about that, let's first consider two time horizons; Current status and how future proof your choice is. For each horizon, we have our needs met by the amount of features we're given by the library/framework, the stability/quality of the code, how accessible it is to learn and use the code (i.e. through documentation, tutorials in the community, etc.), how easy it is to extend the code yourself for those parts that are missing, and how much third party plugins you can find. All of these factors comes from the quality of the core development team, the community, and how they interact, so let's go through each part and take a look.

As for the core team, the vast majority of effort made to Derby is now in the hands of Nate from Lever (one of the original creators). My very personal opinion is that he has a lot of history that points to him making smart decisions about the direction of the framework. The downside is that he is only one man, and that his time (that he puts into Derby) seems to be quite limited (which is not a big surprise given he's the CTO of Lever and probably has a lot of other things to constantly worry about). At times, some additions have been made by others, most often part of the Lever team (Brian originally then Joseph, and so on). No major contributor seems to be around no more except for Nate though. In addition to Nate, it seems people from the Lever team occasionally makes contributions. Zeus has been a little bit active and done some minor parts lately for example. All in all, Nate controls the project more or less with an iron fist (if I may use that expression), which means he control everything that goes into the Derby-stack (i.e. Derby and all it's related modules such as Racer, Share, Tracks, and others). Speaking purely based upon my historic experience, there are very few PRs getting pulled in. Not even PRs made from people on the Lever team seems to make it in any kind of timely fashion (e.g. https://github.com/derbyjs/derby/pull/521 ). I've tried a couple of times myself with no success (one PR was so old when it was reviewed that it had become so difficult to review that it was closed instead...). This is a little bit worrying, because that limits the progress of Derby to basically Nate. My, again very personal, opinion is that Derby is too big of a project for one man. I'd argue this is also obvious from how new, cool and innovative Derby was when it was first released. Since, development has been too slow so now other frameworks have caught up for most parts. For example, server-side rendering (i.e. universal rendering) is more or less a part of every respectable framework out there. Derby had this even before React (at least publicly) existed!

For the community, it is my take that the above hinder has limited the growth of the community, and I fear we've passed the peak level of interest in Derby. In particular the frontend part of Derby is becoming more and more irrelevant with React (and it's accompanying ecosystem), Angular 2, Embers modernization, etcetera. For features, Share, and also Racer, is still rather innovative compared to what's out there. The only framework which is on a similar level when it comes to dealing with the Model-part of your app is in my opinion Meteor. Other libraries that takes care of some stuff exists, like Swarm, PouchDB - but nothing that takes such a complete responsibility for your data (some services also exist such as Firebase [by Google] and recently shut down Parse [by FB]). There are still though quite some features missing in the Racer/Share stack to make them complete (again, in my opinion). The ease of use is not optimal, such as your issues with derby-login/sharedb-access - I believe such significant pieces should be easier to hook on to your stack - or such as how to do validation and easily build a larger model layer. I tried to tackle some of those issues by creating a couple of libraries, e.g. derby-validator, derby-ar and derby-view, but far more can be done in my opinion.

Then, there's also some "bugs" (or minor unexpected features missing - whichever way you see it) which considering the above, will probably not get handled any time soon. Furthermore, tooling around Derby is limited, and will probably remain limited, also consider the above.

So, to summerize my very personal thoughts and opinions;

- Derby is a too project for such governance as is currently.

- I see no change in governance happening anytime soon - thus, I believe Derby will eventually die off.

- The pieces which have most value today is Racer/ShareDB. These are the pieces I'd look at if I wanted to use anything that I'd want to be able to maintain something for a while. These pieces also seem to be where Nate is putting most attention (has been the last 1-2 years, if not more).

Regarding Racer/ShareDB vs alternatives. ShareDB seems to be the major realtime Operational Transformation library out there currently. Racer is a nice add-on, but the value of Racer without Share is limited. If you need OT then Share seems to be a good option, considering there are no better alternatives and then any help you'd get from Nate/Lever/community is welcome since the alternative is to maintain something completely yourself anyway. One major drawback of Share/Racer is the non-existing offline support. If you need that feature, you'd have to develop it yourself.

Which basically is what the guys behind Amelisa did (there are more people interested in Amelisa than meets the eye). They've probably been the most active bunch of people in the Derby community throughout it's history. The reason why Amelisa went with CRDT is because of offline - it's (from what I understand/heard) much easier to do offline with it.

If you use any of these libraries (Amelisa or Racer/ShareDB) without Derby you'd need to use it with some other frontend framework. The only reason this is not super simple is because you'd want to somehow be able to load all data from your backend service before rendering server side (and arguably also between page renderings client side). Most frameworks today aren't built to support that (e.g. React/Angular2). You'd also need to write some glue code to get everything up and running (like what droganov did with React) - it's a little bit cumbersome but probably more manageable. The "load-all-data-before-you-do-actual-rendering"-problem is a relatively known problem and I know some people are thinking of how to solve it, but it's still quite unsolved, and I've yet to see any attempt which looks like it will be the way that gets established as the main way to solve that problem.

In my personal opinion, I really like what the Angular team is doing with Angular2. Some people prefer React. Both have huge communities and will evolve and so forth - both seems to be safe bets. That doesn't cover the Model part of you application though. I personally like what the Amelisa guys are doing (and I prefer the responsiveness and mentality of those guys - they seem much more open to contributions for example). Both Racer/Share and Amelisa is though very small communities, which makes them rather unreliable. Meteor has gathered a much larger community and seems be a much more reliable project. It does lack any advanced conflict resolution though, which makes it questionable for offline/sync-later stuff and it doesn't seem like this is a priority for them. They also have some quirks like their own package manager, questionable motives (after all, they want to make money on people using their framework), etcetera. Same goes with services such as Firebase - there's no advanced conflict resolution AFAIK.

So, what should you do? If you want to dig in to using ShareDB with React, you should probably be aware of that you'd most likely have to do quite a lot of the work to make it work yourself. Thus, I'd not expect any significant support from the community (again, consider the above stagnancy of the Derby community as well as project). Then again, it all depends on your needs for your specific project. If you want something fast up and running which you don't think you'll have to update later on, Derby works fine (at least AFAIK and to my experience with it). If you need something future-proof and you don't really need any fancy conflict resolution algoritm (or offline support), then why not use Meteor? If you need offline support, then I'd look into Amelisa or PouchDB (or Mongo). If you need OT, then I'd look into ShareDB/Racer with React/Angular/something other for the frontend - if you have the resources to support maintaining the glue code.

Some rambling from me. This type of question occasionally pops up so I thought it be good to answer it properly so that one can point people here.

Again, all of this is just my own opinion. I love the work that Nate and others have put into Derby throughout the years - it's been a fun ride! I hope I wasn't too negative about the outlook of Derby!

Cheers,

[Curran Kelleher]

Hi Carl-Johan,

Thank you for crafting such a detailed reply. I really appreciate your spending time on this.

Your reflections at the end are quite useful for me.

Cheers,

Curran

[Nate Smith]

As a quick follow up, I think this is pretty accurate and I appreciated the time put in.

Most recently, I indeed have been focusing my efforts on the backend side of things more than the DerbyJS frontend codebase. I agree ShareDB continues to be the most differentiated part of the stack and the most important one to develop as the foundation for everything else. I've always viewed these projects as fundamentally related but separate and important in their own ways.

100% correct that I can't do all of these projects by myself. In fact, I'm spending a lot of my time hiring a platform engineering team at Lever. We of course have supporting the users of our product as our first priority, but this is the first time we've started to build a dedicated team separate from our product engineering team.

I am planning on continuing to develop Derby and I think there is value in it still. Of course we have a large codebase written on top of Derby and it is working well for us at Lever, so we'll continue to maintain it for some time to come.

But beyond our existing code, there are some really interesting differences technically between the way that Derby renders compared to React, Angular, Angular 2, Ember, et al. I'm curious to investigate the performance of the different approaches on a few levels, though Derby hasn't been optimized much yet. Personally, I also think the template syntax is nice in certain ways compared to the others. It has always been designed from the ground up to do Universal rendering and being the first framework with that focus out of these peers, I've learned a lot. ;-)

There is a long way to go, and I'm excited to see people use ShareDB with Angular 2 or React as well as Derby. We're continuing to support Derby at Lever, and we are beginning to build our platform team in earnest.

--
You received this message because you are subscribed to the Google Groups "Derby" group.
To unsubscribe from this group and stop receiving emails from it, send an email to derbyjs+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.