Amazon EC2 advice

[Robert Hall]

I'm about to launch an SS app that could see some major traffic initially, in the first two months (we're being featured by some heavy-hitting news outlets), and I want to make sure I've got my bases covered. We've chosen to go the EC2 route. I've read Paul and Owen's advice on hosting in production based on Dashku, but I've been DYI on the IT side, and I need some advice. Hoping to elicit some help on the way while I google around at the same time. Maybe this is too general, but if anyone feels like chiming in (even just asking questions to help me figure out what I need to ask) that would be appreciated.

Currently, (just for development) we are loaded w/ two micro tier instances, one for the app, another for MongoDB. I'm ready to up that to what's needed. We don't have anything elastic going on in terms of ELB or HAProxy, though I plan on setting that up for whatever traffic volume comes our way. All our assets are up and running on S3. We're a media-heavy site (video, audio, data visualizations), so debating the performance merits versus costs of throwing cloudfront in front of S3 (streaming is actually not our primary concern).

My main concern is about scaling according to our needs. Any advice or war stories would be appreciated. Thx.

-- Robert

[Robert Hall]

(p.s. I'll be hanging out in IRC as much as possible if anyone wants to ping me there: ArxPoetica)

Thx!

[Owen B]

Hey Robert

I'd recommend looking into Amazon Elastic Beanstalk which now supports Node.js. I deployed a web app the other day using Engine.io (websockets) and it worked well. Yet to try it with SocketStream, but I expect it will work fine.

Few things to remember:

1. Set the Load Balancer to TCP (not the default HTTP) for websockets to work

2. Turn off nginx and other proxies (you want everything to go to Node.js directly)

3. Make sure you're using the Redis Session Store and Redis Pub/sub. You can use something like Redis to Go on EC2.

4. Spin up two servers and test. Make sure you all the sessions and pub sub works as expected even with two clients connecting to two servers.

Once you have this you should have a pretty scalable system, able to grow or shrink with demand.

Ideally add something to monitor port 80 and restart the Node process if it goes down. I think Beanstalk can help you here too but I'm not certain.

Please bear in mind I still don't feel 0.3 is entirely ready for big-time production use (in that it can easily stand up to DOS, hacking attempts, or even tested under extremely high load). These are all things I'm bearing in mind whilst working on 0.4.

Good luck!

Owen

[Robert Hall]

Thx Owen -- this is good to get me started -- I'll return to it, I'm sure w/ more questions.

[Thomas White]

Robert,

I have interests in the same area and I am following this thread with curiosity. 

The following links are not directly related to your question but cover the socket scalability. I hope they will be of help. 

https://github.com/observing/balancerbattle

http://blog.mixu.net/2011/08/13/nginx-websockets-ssl-and-socket-io-deployment/

Thomas

------

--
You received this message because you are subscribed to the Google Groups "SocketStream" group.
To unsubscribe from this group and stop receiving emails from it, send an email to socketstream...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Robert Hall]

Oh, regarding restarting node process if it goes down, Owen, we've been using Forever w/ good success. Is this the kind of thing you're referring to?

[Paul Jensen]

Hi Robert,

A few extra pointers. If you're running the app across multiple processes/servers, you might need to configure the client-side assets so that they use the same filename as opposed to the default timestamp option.

Secondly, if you're using Ubuntu, I can recommend using upstart to manage booting your app. Here is an example configuration:

pre-start script

  # prepare environment

  mkdir -p /var/log/APP_NAME

end script

start on runlevel [2345]

stop on shutdown

respawn

respawn limit 10 5

chdir PATH_TO_APP

script

  SS_ENV=production PATH_TO_NODE_BIN app.js >> /var/log/APP_NAME/production.log 2>&1

end script

The advantages of using this over forever are that it uses less memory, and you can configure it to boot the app when the process dies, as well as when the server reboots.

Regards,

Paul Jensen

[Mauro Pompilio]

Hi Robert,

If you're going to follow Paul's advice of using Upstart (which I also recommend) there's a very useful Ruby gem called Foreman[1]. If you deployed something to Heroku you might be familiar with it. It enables you to manage the processes your app need to run. You can use it locally and when you're ready to deploy, you can export your Profile to Upstart scripts (you can do this with Capistrano during deployment for example).

Here's the getting started page: http://blog.daviddollar.org/2011/05/06/introducing-foreman.html

And here's an example of the Foreman export through Capistrano that I used for one of my projects: https://github.com/malditogeek/astor-dashboard/blob/master/config/deploy.rb.sample

Cheers!

Mauro.

--
Vi, Vi, Vi. The Editor of the Beast!
http://www.vim.org

[Robert Hall]

You guys are awesome -- thanks for all the pointers.

[Robert Hall]

Paul -- I don't think I'm running Ubuntu

When I type $ cat /etc/issue I get:

Amazon Linux AMI release 2012.09

Kernel \r on an \m

Being DIY, and less experienced, do you recommend a better AMI? Some that come prebundled w/ Node.js? I think I set it up myself, but maybe there's something preoptimized in the marketplace? Would something like this be good?

https://aws.amazon.com/marketplace/pp/B008ASK0ZE/ref=mkt_blg_jb_wir617

Thanks again for the tips. I'll be digging in over the next couple weeks, to optimize and test.

-- R

[Paul Jensen]

Hi Robert,

I've done some research, and upstart is available on Amazon Linux, which is a variant of Red Hat Enterprise Linux.

Some details I forgot to mention: Upstart's homepage: http://upstart.ubuntu.com/

That example script can be dropped into /etc/init/APP_NAME.conf

Then you can run it like this:

    service APP_NAME start

You'll probably want to run "which service" just to check that it is available, otherwise install via yum.

[Robert Hall]

ah...interesting! thx

[Paul Jensen]

If you're interested in some rudimentary load testing, I can recommend blitz.io. It can hammer custom http routes and give you an idea of how well the servers will cope with serving html (and other assets if they're not already on a CDN). It won't test websockets (as far as I know).

Regards,

Paul Jensen