SocketStream Update

[Owen B]

Hi all

I just wanted to give you a quick update on what going on at the moment.

As many of you know, there are several great Javascript/Node conferences going on back-to-back at the moment. I spoke at LXJS a few weeks ago (video here http://www.youtube.com/watch?v=LOS1lpWXphs) and will be flying to Dublin tomorrow to speak at NodeDublin.com. After that I fly back to London on Saturday for only 4 hours before flying to Portland to speak at RealtimeConf.com. Phew!

Needless to say preparing for all these talks has kept me pretty busy these last few weeks. All this coming at exactly the same time I chose to quit my day job at AOL (see here https://github.com/socketstream/socketstream/commit/859fff932ca9b324e1db60dbccf9c3f06480727e) and move house :)

So please accept my apologies - I know I haven't been around to answer emails/issues as quickly as usual. Rest assured, normal service will resume within a few weeks. In the meantime, if any more experienced users can assist newer users, it would be very much appreciated.

The good news is I'm ready to demo the first bit of 0.4 code in Dublin on Thursday and it *should* hit github this week. It doesn't do much at the moment, but that's because this is actually the *second* full re-write of 0.3.

The first was a refactor of what we have right now. The second was inspired by SubStack's awesome talk on Streams (http://www.youtube.com/watch?v=lQAV3bPOYHo) which I encourage everyone to watch.

Some ideas are so powerful they can't be ignored. As soon as I saw his presentation and started speaking to Dominic Tarr, I knew it would be worth delaying SocketStream 0.4 a little longer to ensure we have something that's truly way ahead of the curve and as compatible with the rest of the Node.js ecosystem as possible.

Stay turned for more updates soon.

Owen

PS: If you're attending NodeDublin or RealtimeConf, be sure to say hello!

[Gilbert B Garza]

Congrats on pursuing your dreams, Owen! You're one of us now ;)

I'm happy to see that you're doing the heavy refactoring early on. You're really building something great; I'm eager to see the moment it catches on widespread.

Gilbert

[Anephenix]

I'd like to add a few words.

Well done to Owen for deciding to make a break for it, it's a bold and gutsy decision.

I'm glad that SocketStream is now free from AOL.

Having built Dashku and seen companies put it to use, I think that there is a great potential to build an ecosystem around SocketStream. That's something which I will be working towards with Anephenix.

Regards,

Paul Jensen

[Robert]

Like.

[Matthew de Detrich]

Congratulations!

[Owen B]

Thanks guys. Appreciated.

I spoke at NodeDublin a few days ago and demoed the *very* early work I've been doing on a brand new experimental version of SocketStream 0.4 that embraces the Node Stream API.

You can find the code I demoed here: https://github.com/socketstream/socketstream-0.4  Note the websocket part isn't implemented yet but will be soon.

It's too early to say yet if the Streams approach used here is going to be viable, but I'm determined to give it a good go and help others in the Node community by providing some 'real world' testing and scenarios. If I need to back peddle and go back to using middleware I will, but that's the purpose of the experiment.

Before I decided to start from scratch with Streams in mind, I did a fair amount of work on other areas of 0.4 which I will be merging in / reimplementing in the coming weeks. Once I'm happy with the overall design and API I'll move the code to the master branch of the main SocketStream project.

Comments and pull requests appreciated as always.

Next stop: RealTime conf in Portland!

Owen

[Nick H]

Hi Owen,

Very curious about the integration of streams.
Can you give us a very quick overview of what sort of real-world functionality/advantages will come with using streams in the socketstream core?

Thanks,

Nick

[Owen B]

Hi Nick

Good question. The benefit of Streams are two fold:

1. The ability to transmit incremental amounts of data (useful for sending large files)

2. A standard API blessed by the brightest minds in the Node community

The case for using Streams to transmit and receive data over the websocket is a no brainer for me as both criteria apply. This is definitely going to happen in 0.4.

As for streaming asset files, I'm still not 100% sure about this yet. That's the purpose of the experiment.

When sending small HTML/CSS files there is little need for 1, however 2 could potentially allow us to get rid of abstraction layers such as 'ss-stylus' etc - a major win.

In addition, by teaching users that SocketStream is simply a collection of modules that pipe data into each other (as I have attempted to do in the new Tutorial), I'm hoping to make SocketStream less 'black boxy' and more of a toolkit of great modules designed to work well together.

Owen

[Robert]

Awesome work here, Owen!

[Owen B]

Hi all

I know I've got a tonne of emails and posts to reply to and I'll try to get to them all, but I just wanted to share with you some good news.

My five (almost) back-to-back talks in Oslo, Lisbon, Dublin, Portland and SF are finally over - phew! For the most part they all went well and a lot more people now know about SocketStream than a few months ago. I will tweet links to the videos when they get posted and update the README.

Now all that's finally over, I can focus fully on finishing 0.4. Luckily I had a major break through last week. I realised that Live Reload and PubSub no longer need to be hard-wired into the framework - they are just modules that communicate over the websocket like everything else.

Hence I know now that the 0.4 core is going to be a fraction of the size of 0.3 and will basically just do two things really well:

1. Allow you to define Single Page Clients (and look after the serving/packing/etc using the new Streams interface)

2. Multiplex Streams over the WebSocket via a new API (likely to be called 'Services', but basically the old Request Responder idea) and deliver appropriate code to the client.

This means we'll still be able to support everything we've done in the past, but with a much leaner core and better architecture. Take a look at what I mean here: https://github.com/socketstream/socketstream-0.4/blob/master/example_app/app.js#L30-32

If we get this right you won't even need to use our RPC module, you'll be able to use DNode. Plus everything is 100% compatible with the Node Core Stream API. So you can take any stream and pipe() the data over the WebSocket. This is truly exciting stuff - especially as you'll be able to use Streams in the browsers too and pipe geolocation data or any other info back to the server.

Lots still to figure out and performance tune, but I'm really confident we're now heading in the right direction. Please feel free to download the 0.4 code and try out the example app. Just remember everything is still subject to change.

My next task is to reduce the amount of over-the-wire traffic (already underway) and increase performance. After that I'm going to look at Sessions and Auth. If we can crack that, we're almost done.

Let me know what you think!

Owen

[Robert]

Fantastic news!

[Leandro Ostera Villalva]

Kudos! This is looking great!

--

Regards,

Leandro Ostera Villalva

[Michael Karliner]

Owen,
    this is turning into a thing of beauty.

    As an educational exercise, I tried making a new
    service by copying with the pub/sub service and
    mangling it. That showed me how a network
    objects (nobjects?) might work. Lovely.

    Could you possibly consider putting a commented, minimal
    service with read/write streams as a template into the next iteration of 0.4?
       

    Mike

[john]

Hi Owen,

For the stream changes have you looked at using (or offering the option) rxjs (Microsoft recently open sourced it) http://www.youtube.com/watch?v=FqBq4uoiG0M

John

[Mike Gleason]

Hi Owen, 

Is there any update on 0.4?

It's been a couple months since there have been any updates... 

Thanks, Mike.

On Tuesday, October 16, 2012 7:50:15 AM UTC-7, Owen B wrote:

[Owen B]

Hey Mike

Yes indeed, it's been a while. There are two reasons:

1. I've just returned from a fantastic two week vacation in Florida feeling very much relaxed and raring to go. Prior to that I was busy moving house, setting up my new company, and doing a very intensive contract in the City. Sadly I've not had much free time to work on SocketStream in the last few months, but that's about to change (see below).

2. I've got a very important decision to make that's currently blocking further development on 0.4. So far I've been using the Node Streams API throughout SocketStream; however I'm having major concerns over the performance of this approach. While I love the 'pipeability' of Streams, the last thing I want is for 0.4 to be significantly slower (and hence handle less clients-per-server) than 0.3. The only way to know this for sure is to benchmark it and then decide.

Hence the next step is to create a new 'non-streaming' branch of 0.4 (which will multiplex messages in the same way as 0.3) so I can run a benchmark which will determine whether or not to go down the Streams (or Streams2) route. If anyone would like to help with this, or writing the benchmarking took, it would be greatly appreciated. Just let me know so we don't duplicate effort.

The good news is I'm finally back home in London with no plans to go anywhere for a while. I have the next week or so free to work on SocketStream and related projects, so you'll see some movement on 0.4 and replies to a lot of outstanding 0.3 issues and emails shortly.

I'm also clearer now than ever about what I want SocketStream 0.4 and beyond to be. I'll post more about this when I get a moment.

Thanks to everyone (especially Paul) for helping others on here whilst I was away. I'm happy to see we now have a great community of devs here and I'm going to work hard to build this up further in 2013.

Owen

[Robert Hall]

Welcome back! (Ready to help in any way I know how.)

[Owen B]

Hi everyone

I've just pushed a major update to SocketStream 0.4 which takes us a big step closer towards a final design:

https://github.com/socketstream/socketstream-0.4/blob/master/HISTORY.md

Comments and suggestions welcome.

Owen

[Davis Ford]

+1 for JavaScript over CoffeeScript -- I will give it a spin very soon.  Thanks for your hard work, Owen

[Robert Hall]

Fantastic news!

[Robert Hall]

Owen -- one thing I think nobody's really mentioned is the idea on starting and stopping sockets at will. Right now it auto fires up when the app first loads, but what if you wanted that event delayed for any reason? I talked w/ Mike Pennisi at this Boston Meetup http://www.meetup.com/Node-js-Boston/events/103683742/ If I understood him correctly, he figured out a way to do that (w/ socket.io)

On Wednesday, March 13, 2013 7:36:31 AM UTC-4, Owen B wrote:

[Owen B]

Hi Robert

This is already in 0.4. When you auto-generate a starter app it will connect right away, but the connect statement is now a separate command - so you won't need to connect to the server at all if you don't want to. Not sure there's much of a use case for arbitrarily disconnecting then reconnecting. Can you think of one?

Owen

[Robert Hall]

That's awesome! As always, one step ahead. :)

As for use case for disconnecting the socket, absolutely. Here's a very simple one:

"offline mode" which is becoming increasingly popular in HTML5 apps.

Chew on it. :)

[Owen B]

Hi all

Another major update today: https://github.com/socketstream/socketstream-0.4/blob/master/HISTORY.md

Also please read: https://github.com/socketstream/socketstream-0.4/tree/master/mods/realtime-service

This is the first implementation of a big idea I've been working on for a while: Creating reusable, testable services (similar to but much better than Request Responders in 0.3) which are not dependent on any framework.

Thus RPC, LiveReload, PubSub etc are now all independent modules which can be loaded in using the 'realtime-service' module.

While Realtime Services will form the heart of SocketStream 0.4, it is now possible for other framework creators to implement them independently of SocketStream. Thus, the investment you make in your application code (all those RPC files you've created) is further protected.

There's still much work to do in this idea, but I'm very confident this is the way forward. I'll be speaking about Realtime Services (and how this idea compares to proprietary addon modules used in Meteor) at RealtimeConf EU in April.

Robert: The good news is other developers will soon be free to implement 'offline support' by creating a new Realtime Service which supports this :)

Owen

[Will]

Great to see things in motion!

Will

[Robert Hall]

These sound like good ideas.

[Owen B]

Hi all

Pushed another big bunch of changes to 0.4 today.

Highlights

Sessions are now back (still fully compatible with Express.js), and SockJS support is now working (and reconnecting properly for the first time ever!). In addition, you can now connect to a SocketStream server from an external Node process (as well as a browser), doing away with the need for ss-console entirely.

Take a look at the change log here: https://github.com/socketstream/socketstream-0.4/blob/master/HISTORY.md

There are two parts to SocketStream: The Realtime/Websocket server part and the client asset building part. I'm now feeling happy with the overal design of the first and hope to have this ready in a few weeks, in time for Realtime Conf EU. The second half of things (client asset building etc) will follow shortly after.

By far the biggest change in 0.4 is the new module approach. I've been listening to Substack talk and write extensively about this, and I believe he's spot on. Everything that can be a standalone module now is, or shortly will be.

This makes SocketStream far more attractive to those who, for example, just want the realtime server but want to use their own asset build process.

The example_app is working well within 0.4, so feel free to give it a go.

Owen

[Gilbert B Garza]

Incredible stuff. I'm really excited for SocketStream's future.

Gilbert

Owen

--
You received this message because you are subscribed to the Google Groups "SocketStream" group.
To unsubscribe from this group and stop receiving emails from it, send an email to socketstream...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Owen B]

Thanks Gilbert

Just pushed another big update: https://github.com/socketstream/socketstream-0.4/blob/master/HISTORY.md

Also updated the README: https://github.com/socketstream/socketstream-0.4

The Realtime Server part is now feature complete. I still need to do a bit more to restore 0.3 functionality such as subscribing to channels, but that shouldn't take too long.

I've also begun work on the Client Asset Serving/Building part. Once that's done I'll be able to get the first 0.4 alpha out. 

Although a few minor changes will be required, it should be relatively easy to move a 0.3 project to 0.4 once the first alpha's out.

Cheers,

Owen

[Mike Gleason]

Hi Owen,

I just wanted to give some feedback on SocketStream 0.4.  I'm super-exited about the direction it's headed and I only wish it was released now!  Making it production-ready and opinionated with the freedom to change any part of the stack later down the line is a beautiful thing!

FYI - I thought http://nodeup.com/fortytwo - "a scaling websockets show" was really useful.

In-line with being opinionated, and given that SSL is so important for ensuring users can connect via websockets, I'm wondering:

• Does it make sense to offer a config option to enforce a "site-wide, SSL/TSL-only" policy?  - possibly even HSTS (HTTPS Strict Transport Security).  This way, you could turn the option off to develop without SSL, but turn it on for production.

When I first read the prism readme, I mis-understood what the in-memory session cache was and thought it was a substitute for Redis - rather than a compliment.  But I still had a couple questions:

• Does SocketStream/Prism authenticate every request (each RPC request, PubSub, etc)?  Or does it authenticate only once - when the Websocket is established?
• I wonder if there's a way to add the step for Prism building a custom client to the Asset Server (or a bower or npm build step) rather than including it in index.js?

Thanks for all your hard work.  I can't wait to for the rest of the pieces to come together.

[Davis Ford]

Owen, what are the jade-stream and stylus-stream dependencies you have in the Express example server?

https://github.com/socketstream/socketstream-0.4/blob/master/example_app/scripts/express_example.js#L11

I cannot find any references to them in your github repo, or in npm, or in any package.json file.

Owen

--

You received this message because you are subscribed to the Google Groups "SocketStream" group.
To unsubscribe from this group and stop receiving emails from it, send an email to socketstream...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

http://daisyworks.com

@zenocon