Meteor and the future of SocketStream

[Owen B]

As many of you know, Meteor (www.meteor.com), a new realtime web framework was released yesterday. I am taking the time to comment about it here as I believe it is the first real 'competitor' to SocketStream.

First up I would like to congratulate Matt, Geoff and the team for the immense amount of hard work that's gone into their initial release. I am really impressed, not just with the sheer audaciousness of Meteor, but also the launch, website and screencast.

So how does Meteor stack up to SocketStream?

Let me start by addressing the subject of Models. Models are currently the greatest reason to use Meteor instead of SocketStream. Right now if your app needs Models and reactive templating today, you should be looking at Meteor as there is currently no equivalent in SocketStream.

So what is the future for Models in SocketStream? 

After much thought I have decided not to integrate any Model/ORM layer into the SocketStream core. There are several reasons for this:

1. Many apps (especially games) do not need Models. Light-weight function calls combined with the power of websocket middleware are often better in many cases.
   
2. I think it's far too early to declare *one way* of doing Models. Security is a *major* concern and has yet to be addressed at all in Meteor
   
3. I am more interested in exploring ways to leverage existing technologies which already include support for client-side models (e.g. Backbone.js and Ember.js) rather than reimplementing everything from scratch. I must stress than any future Backbone/Ember + SocketStream combination will only ever be implemented as optional modules, we will not be bloating the core with such adventures.
   

As may of you know, SocketStream 0.3 allows us to support different Request Responders (websocket message types). The 'event' and 'rpc' responders are the first two responders we've created and are bundled by default.

Now I am pretty much happy with the API, there will be new documentation by the end of the week allowing people to create their own Request Responders which can be written and distributed as optional modules. Thus I'm hoping others will start experimenting with different ways to approach Models whilst I focus more on developing www.socketstream.org.

So putting Models to one side, what about everything else...

Well I could spend time talking about the difference in features (there are many), or the license (SocketStream is MIT, Meteor is GPL), but these things could easily change in the future.

Instead I want to focus on what I believe to be the fundamental difference between the two: Meteor's decision to make a fullstack framework vs SocketStream's module approach.

As Apple have proved, there is something very beautiful about end-to-end integration done right.

In many ways, Meteor was the framework I *originally* wanted SocketStream to be - an end-to-end ecosystem where everything works seamlessly together, allowing developers to add new modules with commands such as 'socketstream add mongodb' and start the app up with a command such as 'socketstream start' (as we used to do in SocketStream 0.1 and 0.2).

SocketStream 0.2 was my attempt to build this idyllic fullstack solution, complete with in-built scalability, authentication, HTTP APIs and the ability to integrate with other services over ZeroMQ.

Though the feature list sounded great at first, towards the end of last year I began to realise I was heading in the wrong direction.

No matter how much functionality I added, people always wanted more. As 'more' often meant breaking existing behaviour, this led to adding more configuration options, and ultimately more complexity.

Worse still, I always knew there were other modules out there on NPM that not only did a better job, but had the passion of a dedicated person (or community) to document them and maintain them.

Consequently this led to a radical change of direction and a complete re-write of SocketStream to what we have today.

SocketStream 0.3

The result is that SocketStream 0.3 is an altogether different beast:

1. Instead of a proprietary start command SocketStream now behaves like any other Node.js module, so you start your app with 'node app.js'. It may not feel as sexy as 'socketstream start', but it sends a clear message that SocketStream is just another module in your stack and is treated no different to any other framework or library you may choose to use at the same time.
   
   
2. Instead of having a proprietary package format we fully embrace NPM and go to great lengths to ensure SocketStream is compatible with virtually all modules on there. I know many of you are already using SocketStream with Express, Everyauth, Mongo, Redis and lots more. You can even share session data between Express thanks to our use of Connect Session Store.
   
   
3. A great deal of effort has gone into making the SocketStream core as lean as possible by (pushing all opinionated choices into external modules) and also as Node-like as possible.
   

The temptation to implement some sort of 'smart module' system has always been there; but I've resisted the urge. Requiring modules from NPM may not be as slick as 'meteor add coffeescript', but it makes it much easier to debug your app, swap out one module with another, handle conflicts, and experiment with different versions if things go wrong.

What's next for SocketStream

Now development on 0.3 has just about finished (the next release will be 0.3.0 and pushed to NPM), I will focus on improving the areas in which SocketStream is currently horribly deficient: lack of example apps and integration code, more documentation and testing, tutorials and screencasts.

Whilst the Meteor team will be working on satisfying users insatiable appetite for new features; the SocketStream core is unlikely to change in size and features from this point onwards. If anything I hope we get smaller and leaner by better design and refactoring (the sort of code contributions I value most).

Rather than try to develop every new feature or optional module myself, I want to be able to provide developers with the tools and APIs to be creative and experiment with new ideas; as well as providing a platform (via the website) to advertise their module for others to find, rate and comment on.

In conclusion

One thing I know we can all agree on is that realtime web apps are the future and the *only* way we will be developing web apps within a few years time.

This is a very exciting time for SocketStream, Meteor and a whole host of similar frameworks which are no doubt in stealth development right now. Time will tell which approach is better. I honestly believe there is value in both and I have no doubt Meteor will be successful in their own right - they deserve to be.

But having had the luxury of trying both approaches over the last 18 months, I feel more strongly then ever that SocketStream's modular, integrative approach, designed to leverage the best of the Node community (rather than abstracting it away), is the best way forward.

Owen

P.S. I will no doubt be debating this further on NodeUp (http://nodeup.com) this Sunday. Be sure to tune in :)

[Davis Ford]

excellent summary -- looking forward to 0.3 release

On Thursday, April 12, 2012 11:28:04 AM UTC-4, Owen B wrote:

[Carl Larson]

Thanks for the great explanation of the differences between Meteor and SocketStream. It does seem strange, maybe just ambitious, to go with such a monolithic approach as I thought the trend was very much away from that. Though maybe it's easier to create a business model around the tool when you control more of the stack. Very slick screencast, though.

You mentioned the lack of example apps. I learn by doing, but my imagination is sometimes lacking for what to work on. If you (Owen) or others threw out some ideas for smallish projects that would work well for examples then I think I'd enjoy working on some. It wouldn't be altruistic either, as I'm looking to build up a body of some work.

Great going, by the way.

~Carl

[Gilbert B Garza]

Great writeup, Owen. This is all exactly why SocketStream interests me so greatly. I can't wait until I get a chance to use it.

Gilbert

[Carl-Johan B]

To anyone wanting to know the big, although not fundamental, differences of the two projects, read below (feel free to add replies below of anything I missed, worth mentioning):

Licensing

SocketStream uses the MIT license whereas Meteor uses the GPL license. The big difference here is that it's possible to completely freely use SocketStream in any application, commercial or not, without giving out your code. GPL enforces you to. Worth mentioning is that the Meteor team has said they happily will look into providing another commercial license (which basically means, you'd have to pay them to use Meteor). Conclusion: SocketStream can be used to make money, without paying. Meteor not so much.

Security

Because SocketStream are easier to use with existing libraries for authentication (because SocketStream is more a library than a framework in some sense) than Meteor, it can provide security. Meteor does currently not have any options for authentication, which in reality means that all your clients gains access to your complete database. However, the team behind Meteor have explicitly said this is one thing they are going to fix for upcoming releases. Worth mentioning though is that there are some people skeptical of how secure this authentication model will really be.

Other than that, I think Owen put it pretty well in the original post here - SocketStream does only so much of what Meteor does. Meteor is a full-stack solution whereas SocketStream has turned into just one part of what Meteor provides and instead focuses more on making it easy to integrate other tools/projects with it.

[Carl-Johan B]

I'm aware the two points I raised really seem aimed at promoting SocketStream and demoting Meteor - but really that's not my goal. Just, they are simply the things I would worry about if I were to choose between the two for a project that might make some kind of profit. They both have their goods and bads. It's all depending on what you are looking for, and the characteristics around your project, etcetera.

[Davis Ford]

On Thu, Apr 12, 2012 at 8:10 PM, Carl-Johan B <carljohan...@gmail.com> wrote:

To anyone wanting to know the big, although not fundamental, differences of the two projects, read below (feel free to add replies below of anything I missed, worth mentioning):

A couple other interesting items, from their docs http://docs.meteor.com/#structuringyourapp - they run a fiber with thread/request

Meteor gathers all your JavaScript files, excluding anything under the client and public subdirectories, and loads them into a Node.js server instance inside a fiber. In Meteor, your server code runs in a single thread per request, not in the asynchronous callback style typical of Node. 

 Also, it doesn't appear that they take advantage of npm, but built their own packaging / deploy mechanism.

[David Rosen]

It's an odd decision of the Meteor team to leave security as an afterthought. Having your whole database accessible from the client by default is convenient but unwise.

One of the nice things about SocketStream is that it didn't enforce any particular data model. Before I found SocketStream, I had already decided I like the MVVM of Knockout (www.knockoutjs.com) and also wanted to use MongoDB (www.mongodb.org). SocketStream gives me a pretty flexible framework for building a real-time app using my own choice of models.

I completely agree that Node, NPM, and everything else should not be abstracted away, as it allows me to use almost any new technology that comes out.

Dave

[Stephen Handley]

I'm currently using Meteor to prototype an app (http://coachella.hello10.com). I've found that the issues with not using NPM, their wrapper around mongodb-native, and the mishmash of sync and async for RPC (Meteor.methods/call) makes it pretty obnoxious to use at times. 

That said, their reactive ui stuff is impressive... binding data providers to templates has been a huge timesaver for me.

Meteor is the kind of framework that makes for an amazing screencast but when you get into the details, the magic that made that screencast so cool gets in the way more than anything. Security is clearly their biggest oversight and I've ended up not using any of the client-side db write stuff...and no publish / subscribe. 

I'm interested to see what they do with their auth branch, but I suspect Meteor will be primarily useful for prototyping and toy apps. 

I appreciate the minimalist / plays well with others / library approach you're taking with SocketStream, I think that's the right move for developing something people will actually use in production.  

On Thursday, April 12, 2012 8:28:04 AM UTC-7, Owen B wrote:

[Cory Armbrecht]

Agreed on not abstracting away. I have been planning out my development based on socketstream used in conjunction with other modules. I saw Meteor, and needed this exact explanation. Thank you!

[Owen B]

Hi guys

Thanks for all the great comments.

Just to let you know, I've decided to improve the Request Responder API a little before documenting it and releasing an example module. This will all be available on Github by the end of this week (22nd April).

Over the weekend I got an awesome model helicopter which uses an iPhone app as the controller (http://www.ihelicopters.net/shop/ihelicopters/ihelicopter-iphone-controlled-rc-helicopter-777-170/). 

It really made me think... No one at Apple could have possibly envisaged the many amazing and varied apps developers would build on top of their platform.

In a similar (albeit much smaller) way, I believe the Request Responder API will lead to future innovations around models, model synching, low-level gaming protocols, user presence, and all sorts of other crazy stuff I can't even imagine.

If this idea works as well as I hope, the coming months should be very exciting.

Owen

[Robert]

Just some quick thoughts on Derby.js -- which has a similar mission/vision as Meteor.js, built by a couple Google search guys.

Over the weekend I took Derby.js for a spin and immediately realized some pretty obnoxious inflexibility with regards to front end UI control. I opened a bunch of tickets over there in hopes of getting some things fixed, and I still like Derby (probably more than Meteor) as an idea, and possibly, if they fix some things, in the future as a solution.

But right now I'm definitely leaning more towards SocketStream *precisely* because you are not trying to do and be everything at once. This seems the right attitude.