State of redis store and horizontal scaling in general?

[cendrizzi]

Just listened to the nodeup podcast on websocket scaling (nice work for those of you producing that, some of which I've seen on this group). 

Anyway the redis store is the supported way to scale socket.io across processes, however, I keep hearing a lot of real world cautionary tales that stem from it. Basically many people saying it simple has not worked for them when it comes to scaling.

I'm in startup mode right now and have socket.io working very well (using SSL) in terms of code stability. However, I am just starting to look more closely at scaling the infrastructure further. I need to support at least ten thousand concurrent connections initially, using a lot of rooms, however it can be anticipated that there will fairly light talk going over the wire most of the time (this is not a chat app that could literally see tons of chatter across the wire). Obviously the better things go the more concurrent connections will exist in the future. My main interest in websockets has always been the light overhead and most importantly the very quick, real time updates.

Anyway, is the redis store even usable in it's current form for scaling really big? At present I'm using redis.io so hopefully they can scale to meet my redis store demands but I'm pretty concerned when it seems like so many have to fork their own socket.io implementation just to scale it property.

Any help/advise/code etc to help would be really, really nice. I would also love to really understand why the redis backend has been problematic for others. I would switch to engine.io if I didn't want the reconnection and room support (while I don't have rooms in the real sense I do need to partition the communication which rooms provide). 

I would REALLY appreciate any help from any of you who have already been down this path (and might even be interested in some consultation work if interested).

Lastly, I should mention that I've been playing with nodejitsu and modulus as my hosting service. At this juncture I have no interest in setting up my own infrastructure as I would rather rely on others expertise.

[cendrizzi]

No one can chime in on this? I hear all these bad stories but yet never hear anything on what to do about it.

[Filipe]

Today we finished implementing Socket.io + Redis + Node Clusters + Authentication (share express session). Everything went fine.

Now we will implement load balancer with stick sessions.

[Duncan Wilkie]

Hey,

Are you going to write a blog post about your setup and experiences? I'd be interested in reading it, I'm in the process of setting up a fairly small scale - internal only websocket "real-time" web application - and I'm interested in other peoples setup, though I'm going for Sockjs rather than socket.io

- mrdnk

--
You received this message because you are subscribed to the Google Groups "Socket.IO" group.
To unsubscribe from this group and stop receiving emails from it, send an email to socket_io+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[DAVIES, Huw 4]

+1 for a blog post on your experiences and how you did it.

****** This email is sent for and on behalf of Admiral Group plc ******

Admiral Group plc is registered in England and Wales at Capital Tower, Greyfriars Road, Cardiff CF10 3AZ (Reg. No. 03849958). EUI Limited (Reg No: 02686904) is a subsidiary of Admiral Group plc and is authorised and regulated by the Financial Services Authority (Firm reference number: 309378). These details can be confirmed by visiting the Financial Services Authority's Register, www.fsa.gov.uk/register/. EUI Limited acts for, and on behalf of, other regulated insurance companies. Further details may be provided on request.

Any opinions expressed in this email are those of the individual and not necessarily the company. This email and any files transmitted with it, including replies and forwarded copies (which may contain alterations) subsequently transmitted from the Company, are confidential and solely for the use of the intended recipient. It may contain material protected by attorney-client privilege. If you are not the intended recipient or the person responsible for delivering to the intended recipient, be advised that you have received this email in error and that any use is strictly prohibited.

If you have received this email in error please notify our customer services department by telephone on +44 (0)871 882 0000. Please then delete this email and any copies of it.
Security Warning: Please note that this email has been created in the knowledge that Internet email is not a 100% secure communications medium. We advise that you understand and accept this lack of security when emailing us.
Viruses: Although we have taken steps to ensure that this email and any attachments are free from any virus, we advise that in keeping with good computing practice the recipient should ensure they are actually virus free.
We may monitor the content of emails sent and received via our network for viruses or unauthorised use and for other lawful business purposes.

[Alejandro Galvis]

Hey, 

Probably you know better than I, who is also more or less newbie in town and have to go snapped through the floor with the impressive socket.io and redis, anyway have read that redis is the way to go as a "temporary database" for scaling and some people having problems with redis: I haven't until now. Anyway, as any help is appreciated...., just made a one-one multiplayer game with socket.io only (no redis), and what I can say is that you always have to make your own socket.io functions (you wont find all the code available so push it up and show us zen master what you can do) and from these functions the overall performance is achieved. following the example I'm sending light info every 16ms from user to user, so the "realtime" is "trunked" to each 16ms per user, but even at such infinitely low throughput (one-one multiplayer),  I've seen some weird socket.io behavior especially when automatic reload events happen from the client (one browser and 2 ids for instance). Also disconnect functions are required to be made both when normally closing the session as well as inter-session (e.g. if one of the members of the game closes the window the server goes down as the client session is over but another gamer still wants to connect to the disconnected user causes failure if the functions you write don't handle that). Probably when scaling to thousand or millions of users you will have to implement some sort of TDMA schema (dont know if other comms sachems like CDMA or so could actually be implemented here) and do some queue design to optimize network performance for your app and how the packets are served, something like load and user balancing, but you will have to create it and to optimize it for your app/system. 

I think nothing is really written here, it's a new world as node.js and socket.io is a relatively new technology that is beginning to be adopted and credited (and they are awesome!!!). In the game I had also to put the username tied to the client.id so to send by username or id indistinctly allowing the "rooms" making easier, guess this would also allow easy user and load balancing somehow when scaling. Finally, on such critical system, infrastructure relying seems like the best approach initially, probably if the startup goes great then you will have to invest lot of money in servers and a VMware like system, some reliable hundreds TB or PetaB storage, electrical wiring, etc,etc,etc, which can be some interesting tens of millions of dollars (let's be crazy and say between 30 and 50Million dollars without software to begin with :D)  at present value. 

HOH, Last but not least :D, if you are storing data for your users, e.g. like private messages or whatever, when scaling try to test your damned system to full load as you don't want that after 2 or 3 years the system goes down for a week because your system is full of data and your purging system didn't work for whatever reason and the database gets screwed up. Hope this helps. 

[Filipe]

Guys, unfortunately, false positive. Forcing some workers to have an uncaught exception and exit is when things get really messy :(

[cendrizzi]

What does this mean? Does this mean your setup isn't working?

[Filipe]

Right, it isn't working.

[Clark Endrizzi]

So you are seeing a problem with socket.io as it is or is it because you are trying to do something specific to your setup and seeing problems as a result?

Also is the problem you are seeing because of scaling or are you seeing it regardless. Right now my stuff is not being hit very hard, but I’m certainly not seeing any strange exceptions. 

Sorry for all the questions. Just interested in any potential pitfalls I can avoid.

You received this message because you are subscribed to a topic in the Google Groups "Socket.IO" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/socket_io/iSXqmkpgc14/unsubscribe.
To unsubscribe from this group and all its topics, send an email to socket_io+...@googlegroups.com.