Load Testing Meteor

[Adrian Lanning]

Started talking about load testing Meteor apps in another thread...

  HA Meteor on AWS

  https://groups.google.com/forum/?fromgroups=#!topic/meteor-talk/677AGqvfwR4

... but thought it would be useful to pull this out into its own topic.

Here's the discussion so far.  Please join in!

========================================

Adrian Lanning

  Load testing - Selenium/WebDriver tests seem to be the best option.  So far, Sauce Labs is my front-runner.  I get a weird vibe from the Neustar website (formerly BrowserMob).  Anyone used either before or have a better alternative?

========================================

Andrew Wilcox

A word of warning, Selenium tests are a huge huge time sink.   I spent weeks trying to get reliable tests working and ended up with a test suite that only mostly worked and that I abandoned.

Personally, I'd do my functional or load testing myself right from inside of my Meteor app.  All Selenium does is wait for certain conditions to be true on the web page and then take particular actions... which I can do far more easily and reliably using JavaScript and jQuery.

========================================

Adrian Lanning

@Andrew  Ugh, I hear you.  I had a similar experience with Selenium several years ago.  I'm certainly interested in finding a better solution.

For load testing we don't have to run through all functionality, just enough to simulate a reasonable load.  So maybe something like this?

  1. Make a slightly different version of our app which auto-logs in

  2. Spin up a bunch of micro instances

  3. Use PhantomJS to connect  (don't even need WebDriver, right?) 

  4. On Meteor.startup, app automatically does a form submission and subscribes to receive data

What do you think?

========================================

Andrew Wilcox

>> For load testing we don't have to run through all 

>> functionality, just enough to simulate a reasonable 

>> load.  So maybe something like this?

>>   1. Make a slightly different version of our app which auto-logs in

 

Yes, exactly.  You have your regular app, and then you have an extra "load-test.js" file that you add on the client to automatically run your tests.

>>  2. Spin up a bunch of micro instances

Maybe... you might find that you don't actually need more than one instance.  When load testing, your test code can hammer the server continuously, cycling through steps a lot faster than a person would clicking through the app.  And you can run many multiple copies of the client on one instance.

I wouldn't use micro instances though.  They will let you use CPU in bursts, but if you use too much then you'll get throttled way back.  Start with a small, move up to a medium if you need to.   (I wouldn't be surprised if you find that a single medium instance can hammer your server with as much load as you want).

>>  3. Use PhantomJS to connect  (don't even need WebDriver, right?) 

That's right, plain JavaScript will do it!  :)

And it makes it easy to develop, because you can write your tests against a real browser, and then run the finished test in PhantomJS.

There's only a few things that plain JavaScript can't do on its own... for example, if you're testing your Facebook login path, your app in the browser isn't going to be able to type into Facebook's popup login window.

 

>>  4. On Meteor.startup, app automatically does a form submission and subscribes to receive data

 You got it!

[Adrian Lanning]

Thanks Andrew!  Starting to come together for me.  

Regarding what to measure and how, I'm thinking:

  1. Client - response time via startTime/endTime comparisons

  2. Client - all errors received from server

  3. Client - number of records received from subscriptions

              (to match against an expected set)

  3. Server - CPU usage via sysstat/sar

  4. Server - memory usage via sysstat/sar

  5. Server - errors via logging

  6. Total number of test clients over time via 'started' entries

Client-side measurements would be written to a separate db; probably RDS.

Anything I'm missing?

Also need to look into how to practically handle the test MongoDB replica set.  Ideally we would run tests against a short-lived replica set without having to incur MongoHQ's $100 monthly cost.  Have to get in touch with MongoHQ to see what we can do.

[Andrew Wilcox]

Regarding what to measure and how, I'm thinking:

What is the goal of your testing?  Do you want to find out how many server instances you'd need to support a certain number of clients?

You might want to also measure network usage, to see that you aren't maxing out your network connection (I rather doubt you would be, but it's easy to check).

Also need to look into how to practically handle the test MongoDB replica set.  Ideally we would run tests against a short-lived replica set without having to incur MongoHQ's $100 monthly cost.  Have to get in touch with MongoHQ to see what we can do.

Are you using MongoHQ in production?  You could run your own MongoDB servers for testing... unless you also want to load test MongoHQ to be sure that it won't be a bottleneck either?

[Adrian Lanning]

>> What is the goal of your testing?  Do you want

>> to find out how many server instances you'd

>> need to support a certain number of clients?

Right, that's our core goal.

Additional goals are:

  * understand how performance changes along the 

    load curve

  * determine optimal instance size for our always

    running, baseline servers

  * determine optimal instance size for instances

    we spin up during known-active times

...where optimal means most efficient / cheapest.

We'll run these tests against different server sizes 

(medium, large, etc) and compare how they perform.

>> You might want to also measure network usage,

>> to see that you aren't maxing out your network

>> connection (I rather doubt you would be, but

>> it's easy to check).

Good suggestion, will do.  

FYI, I ran across this product which monitors CPU, memory, and network IO: 

http://www.manageengine.com/products/applications_manager/amazon-ec2-monitoring.html

 

(free version supports up to 5 monitored servers but I haven't tried it yet)

>> Are you using MongoHQ in production?  

Yes, we will be using MongoHQ.

>> You could run your own MongoDB servers for testing...

>> unless you also want to load test MongoHQ to be

>> sure that it won't be a bottleneck either?

Right, ideally we'd test a duplicate of prod.  Is it worth

$100 to do so?  Not at this point.  We'll see what they 

say regarding our use-case.  Maybe we can work

something out.  If not, self-hosting for test is a good

fall-back.

I'm planning on load testing one of the example Meteor apps

first to work out any kinks.  Any preference for which one

you'd like to see?  

How about suggestions for how to package this up so others

can easily reproduce?  Would be great if we had a meteor.com

app which aggregates load test stats like browserscope does.

[Sam Hatoum]

Hello

ON TESTING

I've used Selenium on multiple enterprise projects and Like Fritz (comment in other thread), I've been able to get an extreme level of reliability. Not with WebdriverJS (which I'm learning now) but with plain old Selenese on one project, and again on another project with WebDriver + Java. I think Selenium is the grand daddy of all browser testing tools for the following reasons:

1. It allows you to run your tests on multiple browsers, including mobile
2. Webdriver uses native browser commands to control browsers (IE, Chrome, Safari, Firefox, iOS, Android)
3. SeleniumServer supports mobile support at a framework level, thus allowing you to use Android/iOS as agents from your CI server
4. SeleniumServer gives you excellent reporting
5. You can also run Selenium using PhantomJS/GhostDriver

If you are using plain JavaScript directly and it's working for you, that's awesome. But be cognicent that Selenium 1.0 did and this and WebDriver (Selenium2.0) was invented to solve the problems of running JS to do DOM querying. I remember having  to debug the tests more than the code when trying to run on multiple browsers, not to mention the cross-domain limitations. I imagine in today's web apps, you'll need to login to another service, like google/facebook login (simple use) or create a document on google drive and see it show up in your app.

I'm currently having a nightmare of a time trying to get my head around deferred objects and promises and such, but feel like once I've got a handle on it, it'll be worth the effort for the reasons above.

If you'd like alternatives, I've heard of a few successes of people using CasperJS and ZombieJS, but these guys won't give you points 1, 2 and 3 above.

ON HA

RE the 10 second delay. I'd be careful with collisions. Does your app require transactional operations? If not you should be OK. If you do, I'd ensure you include that in your design as to ensure you don't get inconsistent data as two clients modify the data a transaction depends on in the 10 second window.

It's possible to scale websocket apps so long as they share the socket sessions. SocketIO does this with Redis store for instance. I can't see any support for this any time soon using SockJS, which is what meteor uses. This is causing me some anxiety as our app has a need to do this.

ON LOAD

I've used both BrowserMob and SauceLabs and have always opted to build my own using Amazon, because it's easy and cheaper to have agents that start and stop on demand, which TeamCity can do on Amazon out of the box. TeamCity is also free for small projects. The best free tool I've used for doing load testing in the past is jmeter running on Amazon. Jmeter allows you to do pretty complex stuff. So if you want a tried and tested solution, you can find quite a few scripts that do jmeter on ec2. There's also Bees with Machine Guns if you want to test for DoS attacks.

I will also be looking at load testing and personally, I'm trying to get a pure JavaScript solution end to end. So far I've got my eyes on Benchmark.js and JSLitmus . There's also a script builder for Benchmark.js called JSPerf  which could prove useful. 

ON MONITORING

I'm not sure you need to use the manage engine tool. Have you see Amazon's CloudWatch? You can setup alerts and get detailed reports using this out of the box.

You can also use Google Analytics to do user timings. This is useful as it shows you end user results over time, thus allowing you to know if your performance is increasing/decreasing due to code pushes.

I'm working on most of the things you're working on too and sharing my learnings here: https://github.com/xolvio/real-time-development-with-meteor. I'd like to include performance testing in the build chain so perhaps we can collaborate on this load testing example you're writing?

Sam

--
You received this message because you are subscribed to the Google Groups "meteor-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meteor-talk...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[matt debergalis]

Hi Sam.  I don't think there's anything to worry about with respect to SockJS.

Redis support in Socket.IO isn't relevant to multi-process scaling because we just use the transport layer as a reliable bidirectional stream.  Though reconnect support isn't yet finished, the idea is a client can lose its connection to the server and then resume the DDP session on a different low-level socket.  So the (bulk of the) server's per-client state can't live in the transport layer.

I also don't see how Socket.IO rooms would help.  We don't open multiple sockets between the client and the server.  You can imagine muxing multiple DDP sessions over a single socket in a server-to-server setting.  But you'd want raw TCP or websockets there, not a compatibility library.

[Sam Hatoum]

Hey Matt. Thanks for answering that, I was just writing the question in the meteor-core group. I see what you're saying about the DDP reconnect, that would do the trick.  I was showing the rooms file for this line (26), which shows how multi-process scaling was done for peoplerooms:

    io.set('store', new RedisStore({redisPub:pub, redisSub:sub, redisClient:store}));

My concern is that I'm building an app with extremely high chatter. I basically keep the view state in sync on all connected clients. Think the user dragging an object and every pixel move is distributed to other connected users. With sticky sessions, I would be forced into sharding for scaling and fail-over approach for HA, which I could do, but it more involved than distributing load in a round-robin fashion across an auto-scaling cluster.

I'm looking forward to the reconnect support.

Sam

[Andrew Wilcox]

On Sunday, April 7, 2013 11:32:43 PM UTC-4, Sam Hatoum wrote:

 

I've used Selenium on multiple enterprise projects and Like Fritz (comment in other thread), I've been able to get an extreme level of reliability. Not with WebdriverJS (which I'm learning now) 

Well, I was using Webdriver (Selenium 2), and my tests wouldn't run on Opera because the Opera webdriver was buggy (it couldn't set the window focus).  Now I'm not a Selenium expert and someone who knows more than I do may have gotten it to work with further investigation / using a different driver version / a different approach / etc.   I also had trouble installing web drivers for browsers aside from Firefox, and didn't have much luck with Saucelabs.  Which again, might be fixable with sufficient effort, but all I can report on is my personal experience that Selenium 2 / Webdriver sounds great, but in practice I wasn't able to get it to work for my tests given the amount of time I was willing to put in.

But that's not very interesting.  If someone is testing Meteor with Selenium, it'd be super for them to post the details of their working configuration.

[Sam Hatoum]

Hey Andrew

Those are indeed horrible pain points (I read your other thread too).

I've manage to cross some major hurdles with webdriverjs and I'm doing a write up and sharing the code in the next few of days. It currently runs on my machine with Chrome, Firefox, Safari and PhantomJS. I've not tried Opera, but there is a native driver (vendor made) so it shouldn't be much trouble.

You can even debug the tests line by line and see the interactions happening in a browser with browser debugging tools too, which is a HUGE advantage over Casper and other PhantomJS only solutions, although promises, deferred objects and flows will play twister with your mind!

Sam

[Adrian Lanning]

Hi Sam, 

Thanks for all the advice in your earlier post.  Would love to collaborate on this. :-)

How're these for long-term goals?

  * Space Bees with Machine Guns - command-line tool to spin up instances,or use existing instances, and run phantomjs scripts (like Bees with Machine Guns)

  * Meteorscope - meteor.com site which aggregates benchmarks for different deployment configurations running publicly-available Meteor apps/examples (like Browserscope)

This looked exciting, but pricing made it a dead end:

  http://ariya.ofilabs.com/2012/07/cloud-phantomjs-with-ironworker.html

Found a couple of projects that may be useful starting points for the phantomjs script that would run and collect basic stats:

  * http://loadreport.wesleyhales.com/

  * https://github.com/macbre/phantomas

I'm thinking the test script can write custom data to console.log and the test runner can report that.  I did something similar with phantomjs when I was debugging the spiderable package.

That would be useful so we could get timing information on how long it takes for a client to receive some expected result, say a table of published data, and also for checking that it received all the data it was supposed to.  Although that's starting to blur the lines between load testing and functional testing.

What do you think?

On Sunday, April 7, 2013 11:32:43 PM UTC-4, Sam Hatoum wrote:

[Sam Hatoum]

Hey Adrian

The long term goals look pretty cool. I think you're right to blur the line between functional and load testing as you're ensuring data integrity by doing so. 

My first thoughts are, what are the outcomes we'd like from these reports? The tools you've mentioned are great diagnostic tools to help solve the overall problem of "is my site running at the expected thresholds".

Ultimately it's all about feedback and getting that as quickly as possible. So now we can split into two types of feedback when it comes to a build chain:

1. Predictive — Anything we can detect and act on before a release
2. Latent — Anything we can detect and act on after a release

So now my question becomes, what performance feedback do we need to a) make a better product and b) develop quicker?

Here are some performance related feedback items that I've used/seen in the past: (Please feel elaborate as you see fit)

1. Predictive: Load tests — preferably on a mirror of live, to aid capacity planning
2. Predictive: Stress tests —preferably on a mirror of live, to find data integrity problems
   
3. Predictive: Soak — preferably on a mirror of live, to find leaks
   
4. Predictive & Latent: Transaction time trends — the time it takes to do a set of transactions over time

I've previously used JMeter to do the first 3 items.  As new features are developed, they'd be added to performance testing suite and this would run overnight on a regular basis. I wouldn't use this to block the build pipeline and a nightly cycle (or less depending on your code pushes) is good enough. Perhaps we can do something similar here with Benchmark.js + Webdriver + PhantomJS for a pure JS solution? What would be really interesting, is having some shared functionality between the end-to-end tests and the performance tests, therefore reducing the effort required to write and maintain scripts.

For the 4th point, I've previously used Selenium to record the trend. That is, how long each test took, and built a map locally and on the build server. Even though these tests never ran against a full environment stack, they were still useful as they were relative to the environment they were running on, and the trend was the useful metric, not the timings themselves. 

Now I've been thinking a lot about using Google Analytics to do this. It's the place that I will be going to see my visitor flow and the querying abilities there are fantastic. If you look at this code here, you'll see the TrackTiming object, which has this line:

  window._gaq.push(['_trackTiming', this.category, this.variable, timeSpent, this.label]);

What we could do at runtime for dev and build server environments, is stub the window._gaq method so that it sends the results of timings to a collector. And on qa/production environments it can be sent to Google Analytics. 

Another good thing about this approach is that it measures the performance on various browsers, and around particular transactions. For instance, we could start timing when the user takes an action, and the stop timing when the view for that action has been rendered/updated. You should log in to google analytics and look at the Content -> Site Speed -> User Timings section. You can run queries like "

The other idea I've been thinking about, is reporting thresholds in observatoryjs, so they show up in the logs as warnings etc.

I'm all about minimizing dev effort and filtering out the noise to expose the useful info. I actually have a feedback fetish and the project I'm working  is attempting to reduce the amount of time it takes to get feedback. I've coined the approach real time development so have a look if you get a chance to see where my head's at.

So I really like your idea of Space Bees. We could easily do that once we've worked out the "attack" method, by spinning up a load of on EC2 micro instances. It's super easy to do too with CloudFormations. Maybe we could call it GhostRider ;)

Sam

[Adrian Lanning]

After coming up short searching for an existing phantomjs load testing tool and then reviewing a wide range of regular load-testers (ab, Jmeter, httperf, wrk, tsung) I'm re-thinking what Space Bees should be.

Key thing is that using phantomjs doesn't appear to be necessary so I think using one of the more established load testers that you're familiar with should work.

I'm thinking a more classic approach:

  1. Record http traffic of user interaction with website

  2. Apply variables where appropriate to scale reqs

  3. Play back reqs using permuted variables

I know you mentioned you've used Jmeter before.  Have you tried Tsung?  It's getting a lot of love from the articles I found online and Tsung seems to be able to both record/playback (in a regular browser via proxy) and permute play back timing.  Not sure about actually permuting values in play back though.

This way may be even simpler:

  1. adjust Meteor code to automatically subscribe/pub every request

  2. use auto-publish to send data to every connection

  3. testing tool only needs to keep the Meteor session alive for a bit

Although I don't know enough about how Meteor sessions work to know which one is easier.  

I found a fork of Bees with Machine Guns that seems to integrate Tsung rather than ab:

  https://github.com/nfarina/beeswithmachineguns/commit/e312f8bfa79d8f86267230443b331e98c040eb83

Good references:

  How to generate a million HTTP requests/sec

    http://dak1n1.com/blog/14-http-load-generate

  Tsung

    https://github.com/processone/tsung

  Httperf

    http://www.igvita.com/2008/09/30/load-testing-with-log-replay/

  Very simple XhrPolling test script

    https://github.com/sockjs/sockjs-protocol/blob/master/smoke-test.coffee

[Andrew Wilcox]

I'm thinking a more classic approach:

  1. Record http traffic of user interaction with website

  2. Apply variables where appropriate to scale reqs

  3. Play back reqs using permuted variables

This sounds like it would be hard to me.  First, you'd need a load tester that talks SockJS.  Then, the Meteor server will return data to the client as it becomes available from the database, so the order of messages that you get from the server can be different each time you connect.  The Meteor client knows the meaning of the various messages, and so knows when to send the next batch of client messages (e.g. after a subscription is successful, after the user logs in, after a document has changed), but the load tester isn't going to know that, so it won't know when to send the recorded client messages.

On the other hand, back to my original suggestion, if you write your load tester as a Meteor app then all these low level details are taken care of for you.

[Mina Mikhail]

Really looking forward to the results of the investigations in this thread. I don't have any experience with doing load testing but I just spent the last week experimenting and deploying a few different scalable server setups for Pegleg and now am at the point where it would be awesome if there were a tool I could just point at them and figure out which one is the best. My approach so far has been completely anecdotal and unscientific.

I plan to put together notes on my experiences trying to scale a simple Meteor app in different environments over the next week, which will hopefully spur some discussion around optimal approaches to scaling and deployment plus how to test performance.

[Tom Coleman]

I'd chime in here and say, rather than 

a) trying to replicate a meteor app by replaying HTTP requests (or WS messages?), which as Andrew said sounds a bit difficult OR

b) using a Meteor client which requires a headless browser and thus a relatively large amount of resources

a third option to consider is:

c) using a ddp client library (for instance https://github.com/oortcloud/node-ddp-client) to simulate the DDP messages sent to a server by a Meteor client.

In the longer term I think c) is the correct approach, alongside some kind of DDP message recorder / replayer.

Tom

[Sam Hatoum]

Hello all

@Adrian

I've not tried tsung. It looks pretty powerful, but as others are pointing out the problem with traditional testing tools (jMeter/tsung) is they don't fire up a browser and meteor doesn't behave like a traditional request/response. I agree with Andrew that it would be very hard to do with those tools as the order of things coming back is going to be unpredictable and these tools don't support this so we'd have to build some hefty intelligence into the tools to handle it. 

@Andrew

Writing a performance driver as a meteor app sounds interesting. How would you go about scaling that to increase requests? I'm assuming an array of headless browser but thought I'd better ask.

@Tom

Very nice indeed. This draws the boundary at exactly the right place.

What would be interesting, is if the recorder/player could create these scenarios from end-to-end tests. Perhaps an intelligent recorder could do something like this:

1. Hooks in to the DDP at runtime and record all the message from a test suite (one feature-level test at time)
2. Post-analyses the recording and sees where the same messages have varied data . Something like: [x, y, z were sent to foo], [a b c was received from bar]
3. Creates a set of scenarios with externalized data with all the permutations

I'm thinking an agent-based test rig is also needed, that can:

1. Runners to execute scenarios and collect metrics (perhaps using http://benchmarkjs.com)
2. Scale up runners (perhaps using node-threads-a-gogo to utilize all the CPU cores of an agent)
3. Scale up agents (using cloudformation templates)
4. Aggregate data from all the agents (a meteor app maybe?)

Sam

[Sam Hatoum]

Step 4 should ready: 

Aggregate and visualize data as it arrives from all the agents (a meteor app with d3 maybe?)

:)

[Adrian Lanning]

Hey everyone, 

Got something started here:

  https://github.com/alanning/meteor-load-test

I was hoping to use one of the more established tools to take advantage of the existing reporting / graphing features and distribution methods. But as everyone pointed out, its a real pain!  :-)

Tried Tsung first but its proxy recorder has problems recording interactions with Meteor apps for some reason. 

Next up was Grinder which recorded fine, but I got stuck at the sockjs interactions.  I couldn't find a simple sockjs client for Java/Clojure; vertx supports sockjs but that looked like a lot of work to integrate and my java is rusty (and my Clojure is much worse).

So went back to phantomjs.  I used the spiderable package as a reference and have a simple test script 'loadmeteor.js' which loads a Meteor app and triggers a simple server interaction.

The example app itself has some customizable settings to adjust what data is published to clients.  Check out settings.json.

Next steps are:

 * integrating scripts from Bees with Machine Guns to scale

 * figure out how to get phantomjs to receive updated data as clients send in new values

Lemme know what you think!

[Andrew Wilcox]

If you wanted to write this without the phantomjs-specific code in loadmeteor.js, you could do the following:

- Factor out the code which runs in response to the button press into its own function ("insertTwo").  Now you have a separation of concerns between the UI (which triggers actions), and then actions themselves (which you want to be able to trigger in other situations, such as load testing).

- Add an additional Meteor JavaScript file to the Meteor app, used only for load testing, which on Meteor startup calls the "insertTwo" function.

Now you have a load tester which will run in any browser environment, not just phantomjs:

- It's easy to use phantomjs when you want to, but it's also easy to use a real browser if you want to (if, for example, a real browser is more convenient for development).

- If you run into something strange, like it looks like the load is different in a particular browser for some weird reason, that's easy to investigate.

- If you decide to use some other headless browser instead of phantomjs for whatever reason, you can plug in your load testing app without having to write new injection code specific to that new environment.

[Sam Hatoum]

@Adrian

It's always easier to start and get results. I'll try it out soon.

@Andrew

If I'm understanding correctly, this would be like building a DDP proxy into the app. That's an cool idea and could work, though I'd be worried about the proxy layer skewing the load results. Would be interesting to see it work

[Andrew Wilcox]

@Andrew

If I'm understanding correctly, this would be like building a DDP proxy into the app.

Um, what?

[Sam Hatoum]

Apologies. I thought you were answering my question: "How would you go about scaling that to increase requests? I'm assuming an array of headless browsers". I was trying to get my head around your answer in that context, but I just realized you're referring to Adrian's code and I'm totally out of context! My bad :)

[Andrew Wilcox]

On Monday, April 15, 2013 8:54:33 PM UTC-4, Sam Hatoum wrote:

Apologies. I thought you were answering my question: "How would you go about scaling that to increase requests? I'm assuming an array of headless browsers". I was trying to get my head around your answer in that context, but I just realized you're referring to Adrian's code and I'm totally out of context! My bad :)

Oh, well... to your question  "How would you go about scaling that to increase requests?" :-)

Yes, once you have a plain Meteor app which is generating load for you, then you can run multiple copies of that Meteor app in any way which is most convenient for you, such as by running multiple instances of phantomjs.

[Adrian Lanning]

For one-off testing of a specific app, I do think in-app triggers are simplest.  For this project I'm trying to get something general enough that the system under test would not have to be modified at all in order to be load tested.

Practically speaking there may end up being little difference between adding code to something like 'loadmeteor.js' or to the app.  But separating it out like this keeps the source code clearly differentiated and also will let us include boiler-plate code in the 'load test runner' script which can do gathering and reporting of metrics.

Regarding running via the browser, I really don't think that's going to produce significant enough load.  You can have a test app that is very 'chatty', sending and requesting lots of data.  But that still wouldn't accurately reflect load from a lot of different users.

That's actually the part where I'm stuck at the moment.  Re: Sam's question, I don't know much about forking threads/processes and such.  

Bees with Machine Guns provides the code for scaling out, infrastructure-wise by spinning up EC2 instances.  But then it kicks off Apache ab which does the actual pounding (via many threads).  

I want to have something that can replace ab in that scenario so we can reuse the Bees part but apply it to Meteor apps.  Anyone have ideas on efficiently executing a lot of phantomjs processes on a single machine?  I think that's going to be the biggest bottleneck as far as cost/resources goes.

[Sam Hatoum]

It's difficult to say without tests, but I think it's going to be a struggle to get enough oomph from a grid of phantomjs. Each Phantom footprint can grow to around 100mb (and likely when doing lots of work). So on a micro instance, I'd say you're looking at 3-4 MAX. I'm not sure how much each instance can generate, but I would suspect a rather large number. Again, this is purely speculative on my part. 

This is an interesting read on how this dude managed to get 1million concurrent connections to a node server from 500 EC2 nodes. There's some food for thought here. He's using Micro instances (I had to look hard for this but he mentions 2 cents per hour, which gives it away!).

In the interest of keeping the swarms cheap, I think micro instances should be the bees.

It seems to me that phantom would be good for doing page analysis type tasks, like resource reporting and such, but for generatins serious load,  I think Tom hit the nail on the head with his suggested approach of a DDP client. I can imagine a micro-instance would be able to generate quite some buzz from clients + node threading, as opposed to a few phantoms. Here's a great post I read on  threading with node . Looks easy enough for this task :)

That'd be my take, but experimentation is king of course.

[z...@percolatestudio.com]

I guess it's my turn to chime in. I've been load testing a meteor prototype for a client over the last week. For a highly scalable solution, I think what Tom suggested (i.e building some sort of DDP based client) is the right approach.

For the time being though, it's relatively easy to overwhelm a meteor server so I've been finding quick success using phantomjs. I haven't taken a strictly scientific approach, just wanted to get some ball park numbers and a feel for how the server performs under load. I've been using an m1.large EC2 instance running meteor 0.6.0 and mongodb as the server and spinning up an m3.2xlarge client to run the tests (as people mentioned, phantomjs is very hungry for ram). The phantom test scripts are quite trivial to write as you can write your tests in Meteor.client and just call into them from phantom.

I'm happy to provide more info if people are interested.

P.S phantomjs has a bug where it doesn't respect the command line parameters you pass in to override the local storage directory/quota (it's a documented issue). As meteor stores state in local storage and each test client needs it's own state this is a problem. The workaround I used was finding the location of phantom's local storage file and placing a 0 byte file there with it's permissions set to be unreadable by the phantomjs user.

Zoltan

[Adrian Lanning]

Pairing Grinder with this Java DDP client may be just what we're looking for...

  https://github.com/kutrumbo/java-ddp-client

More info... yes please!  Could you share some of your results?  

If not against the actual client app, perhaps against the simple example app from here:

  https://github.com/alanning/meteor-load-test/tree/master/sut

[Zoltan Olah]

Sure, I'll elaborate further but as a disclaimer please don't use the results to draw any numerical conclusions regarding meteor, they are just for some ballpark figures regarding our expected usage and exploration of behavior at load.

For our dataset, we have 1000 users each having roughly 10 tasks. Each tasks has 30 comments and is shared by 30 different users. The subscriptions are intelligently setup to subscribe to only the data each user needs to see, that is 10 tasks each with 30 comments in them. We use the standard Accounts authentication.

Setup

---------

Server: m1.large EC2 instance with mongo+meteor on the same machine.

Client: m3.2xlarge running many phantomjs instances.

Test 1

----------

Log in and idle (obviously the client will be reacting to it's subscriptions but the data won't be changing).

Findings: We were able to support around 700 logged in and idling users before the server crashed. The app was quite responsive (logging in/out manually) right up to the crash. The crash manifested itself by Node's CPU usage suddenly spiking upto ~100% and the server refusing to send any data in response to http queries. We disconnected the test clients but the server never recovered.

Test 2

----------

We complemented Test 1 by adding a server side function that would run on startup and in a loop create a comment in a random task with X ms delay between additions. We gradually decreased the delay to 250ms. Our reasoning was to test meteor's ability to calculate the changes in each client's subscribed dataset.

Interestingly, our findings were very similar to Test 1. The server still crashed in the same fashion at roughly the same number of connected users, however the CPU load was higher during the running of the test.

Note: We didn't notice any unusually high memory usage either by mongo or node.

Conclusion

----------------

The behavior under load of a realtime app is significantly different than that of a stateless web-app so we must adapt our methods, tools, expectations and metrics to match. By it's very nature, a reactive realtime app will be significantly more expensive to host than a stateless web-app.

For us so far, I think the results are proving positive and the client's business model will most likely be ok with the cost of hosting. Concerns I have are:

- It's unclear how to decide to spin up more instances. Response times and CPU utilization are ok until a handful more users log in and very suddenly the server is hosed. With a traditional web app you typical have more warning and time to spin up extra resources.

- Our tests ramp up the the number of users gradually until the server fails. When the server crashes and is restarted, suddenly it's hit by the full force of all the users trying to re-establish their connections and it promptly crashes again. We will need some sort of plan to deal with this case in production.

Hope that helps,

-Zoltan

--
You received this message because you are subscribed to a topic in the Google Groups "meteor-talk" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/meteor-talk/M9waYvcFufs/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to meteor-talk...@googlegroups.com.

[Sam Hatoum]

Nice find. I had previously looked at Nathan Rajilch Websocket implementation to use with RhinoJS in Jmeter, but being native Java, this should allow spinning up quite a few workers within a VM.

I would say Jmeter is a better option than Grinder. You can also use JMeter to launch Java clients and there  (forum posts/blogs etc). Here's a jmeter-ec2 command line shell you'll find useful.

I think it's worth externalizing the messages and expected responses to scenario files that are language agnostic. That way, we can have a "player" that execute a set of scenarios. Doing this would give us learnings to inform us of how to write a recorder, which I'm thinking might be some custom middleware.

--

[Adrian Lanning]

@Zoltan

Thank you for sharing and congrats on joining Percolate!  Didn't know you and Tom worked together.  Gotta update your "Who" page though.  :-)

Are you planning on probing further to narrow down the cause of the crash?  Its worrisome that it couldn't recover even after the load went away.  What were you using to measure resource usage?

I'd be curious what the results would be if you placed something like:

  https://github.com/lloyd/node-toobusy

... in front of your node server.  Hopefully there would be enough of a change in event loop response time that toobusy could kick in and avoid the server crash.

@Sam

That jmeter-ec2 project looks really useful.  How would we go about integrating the DDL client into the JMeter testing flow?  I think with Grinder its pretty straight forward as the test scripts are just Jython / Clojure so you can just load the library.  What about JMeter?

It seems our next goal is to be able to write a test script that can then be run by JMeter/Grinder against a meteor app.  If we use:

  https://github.com/alanning/meteor-load-test/blob/master/test-scripts/loadmeteor.js

... as an example, the script would need to:

  1. Connect to the SUT

  2. Wait until initial top 10 entries were received

  3. Make a Meteor method call to add more entries

  4. Wait until entries data changes were received

  5. (optional) Stay connected and receive additional updates as more clients add entries

Does that sound right?

On Tuesday, April 16, 2013 6:53:00 PM UTC-4, Sam Hatoum wrote:

Nice find. I had previously looked at Nathan Rajilch Websocket implementation to use with RhinoJS in Jmeter, but being native Java, this should allow spinning up quite a few workers within a VM.

I would say Jmeter is a better option than Grinder. You can also use JMeter to launch Java clients and there  (forum posts/blogs etc). Here's a jmeter-ec2 command line shell you'll find useful.

I think it's worth externalizing the messages and expected responses to scenario files that are language agnostic. That way, we can have a "player" that execute a set of scenarios. Doing this would give us learnings to inform us of how to write a recorder, which I'm thinking might be some custom middleware.

[Zoltan Olah]

@Adrian

Thanks, Tom and I have been friends and worked together on and off for 10 years now so it's great to be forging ahead together once more. Will get around to updating our team page soon!

Alas, I was measuring resource usage using per/process metrics from top - not very scientific. At this stage we have too much on our plate to investigate the server crash/load in detail but I imagine we'll return to looking at it in depth in a few months time when we go into production. Thanks for the heads up on node-toobusy, I suspect this may help in alleviating the problem although it seems to operate at the request level so not sure whether it will help with permanent web sockets connections.

[Sam Hatoum]

Hey Adrian

That jmeter-ec2 project looks really useful.  How would we go about integrating the DDL client into the JMeter testing flow?  I think with Grinder its pretty straight forward as the test scripts are just Jython / Clojure so you can just load the library.  What about JMeter?

Have a look at this this. You basically implement a custom sampler and put in JMeter's classpath, and you'll be able to use JMeter's awesome assertions, reporting and scaling. I guess the one thing that needs thinking about, is the way of deploying the DDP sampler into agents. I'm thinking a custom VM, but this is a problem for later. It can be all done locally to start.

It seems our next goal is to be able to write a test script that can then be run by JMeter/Grinder against a meteor app.  If we use:

  https://github.com/alanning/meteor-load-test/blob/master/test-scripts/loadmeteor.js

... as an example, the script would need to:

  1. Connect to the SUT

  2. Wait until initial top 10 entries were received

  3. Make a Meteor method call to add more entries

  4. Wait until entries data changes were received

  5. (optional) Stay connected and receive additional updates as more clients add entries

Does that sound right?

Sounds great. Let's get the first part working (steps 1-4). From 5 onwards, I think some inter-agent chatter for assertions would be awesome. 

 

--

You received this message because you are subscribed to the Google Groups "meteor-talk" group.

To unsubscribe from this group and stop receiving emails from it, send an email to meteor-talk...@googlegroups.com.

[Adrian Lanning]

@Zoltan

FYI, Meteor doesn't use websockets for client connections.
I thought so too until Avital set me straight.  I had been avoiding nginx due to its (at the time) lack of websocket support.  :-)

Matt discusses a bit here:

  https://groups.google.com/d/msg/meteor-talk/lR1hF4l2gE4/bwz9KN5qQjEJ

...and the relevant source code:

  https://github.com/meteor/meteor/blob/master/packages/livedata/stream_client_sockjs.js#L143

In 0.6.2 there is the experimental server-to-server connect which uses Websockets but that would (hopefully) be a negligible amount of load. 

[Zoltan Olah]

@Adrian

Oh cool - that's good to know, thanks. 

[matt debergalis]

To be clear:

* The Meteor DDP server supports both Websockets and SockJS. That's
been true for a long time.
* Meteor DDP web client uses SockJS. We've disabled the websocket
option, so it'll use one of the long polling strategies.
* 0.6.2 adds support for running a DDP *client* in Meteor server code,
so that you can connect to another DDP server. This runs over raw
websockets.

> --
> You received this message because you are subscribed to the Google Groups
> "meteor-talk" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[Zoltan Olah]

@Matt

Thanks for the clarity.

You received this message because you are subscribed to a topic in the Google Groups "meteor-talk" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/meteor-talk/M9waYvcFufs/unsubscribe?hl=en.

To unsubscribe from this group and all its topics, send an email to meteor-talk...@googlegroups.com.

[Adrian Lanning]

For those tracking this discussion, there is now a load testing tool available which uses DDP to communicate with Meteor applications.  You can read more in the announcement on meteor-core.