Load tests for an Open edX production environment

[Pierre Mailhot]

Maybe this should be a question for the General Open edX mailing list, but I will try the Open edX operations list first.

What are most of you using in order to generate load tests for the LMS and the CMS before going into production? We will be running in the AWS environment and we want to make sure we have enough capacity for the LMS, the CMS and the databases (MySQL and Mongo).

Any rules of thumb available for specific number of users, courses, etc?

Any hints would be greatly appreciated.

[Jesse Zoldak]

At edX, we've been using JMeter scripts, though we are investigating switching over to using locust.io to generate the load.

We run against an environment that is sized like production, attached to a production replica database but with auto_auth turned on so that new users can be created as active, logged-in, and auto-enrolled in specified courses.

For the transaction volume and ratio of particular transactions, we base that on an analysis of transaction data from production.

For execution results analysis we look at the percentile distribution for transaction times, and the error rate from the load generator tool. On the back end we use our usual monitoring tools (AWS, MongoHQ, NewRelic, DataDog).

HTH.

-- JZ

[Pierre Mailhot]

Thanks Jesse. It gives me a base to work from.

[Pierre Mailhot]

Jesse,

Any particular reason for the potential switch from JMeter to locust.io?

Some of our folks here are using The Grinder for other projects and were already thinking about moving to JMeter. We might be interested in knowing why you want to switch to locust.io.

Any information would be greatly appreciated. Thanks.

Have a nice day.

On Monday, November 3, 2014 12:04:38 PM UTC-5, Jesse Zoldak wrote:

[Jesse Zoldak]

Pierre --

JMeter scripts have been working fine for us for actually driving the load, but there are a couple usability issues with using, maintaining, and updating them. For example:

• The configuration of the test that you want to run (test plan, in jmeter lingo) is all in one huge xml file, which includes the thread groups, logic controllers, etc. This makes it difficult to put under version control system.
• It's a lot easier to code (and for the next developer to read/understand) the logic in python vs. drag and drop in the JMeter GUI (or by hand-crafting the XML).

Also we're looking to extend our load/performance testing from one-off events (e.g. full scale load test against production-sized load test environment) into a framework that is incorporated in the development process. Some potential use cases below. For these a python-based solution is definitely preferable to us. From preliminary discovery, locust seems like it will fit our needs and for any gaps it looks like it would be easy to modify/extend and contribute upstream.

• A developer performing "do no harm" verification against devstack or a sandbox when refactoring or adding new features.
• An automatically triggered performance test against a deployed instance after (or before?) every merge to master.
• Driving a baseline level of continuous traffic against our staging environment that is used for manual testing.

Thoughts and comments welcome.

-- JZ

[Donny Davis]

The largest problem with Jmeter is the machine you run it on plays a large part in the actual load it can place on a server. If you are looking for load testing and dont mind spending a couple dollars, I would fire up a digital ocean instance and use jMeter from there... There is also https://www.blitz.io/

[Nate Aune]

We recently ran some load tests for a customer using Loader.io and later Locust.io using 1 master and 3 slaves.

Here is an example of the output that Loader.io gives you: http://ldr.io/1zPrhFi

This was testing 10 unauthenticated users hitting the Open edX homepage for 1 minute.

The machine we used is a m3.medium instance on AWS.

Here's the same test but with 100 unauthenticated users: http://ldr.io/1vQ6Qrn

[cgol...@edx.org]

On Monday, November 3, 2014 at 10:46:33 AM UTC-5, Pierre Mailhot wrote:

What are most of you using in order to generate load tests for the LMS and the CMS before going into production?

sorry for the reply on such an old thread :)

At edX we are now using locust.io pretty heavily for load testing.

The main 2 advantages (IMO) of Locust over JMeter are:

1) For Locust, virtual user scripts are written in Python.  This is much more powerful than the declarative style (XML) that JMeter uses.  You have the full capability of Python to create more complex logic in your scripts.  Also, all of our developers are comfortable with Python.

2)  Locust is much more scalable then JMeter for generating a high number of virtual users.  Locust uses async i/o, whereas JMeter allocates a thread per virtual user.  Locust is also trivial to setup with many slaves generating load and reporting back to a master node.


regards,

-Corey

 

[TJ Keemon]

Hi Corey-

We're also looking to use locust.io for load testing our Open edX production deployments. Is there any chance you'd be able to share the scripts you guys have written? I've attached the current version of our script. It's still a bit crude, but we were able to load test a site using two Digital Ocean droplets as slaves. 

Thanks.

-TJ

[Corey Goldberg]

Hi TJ,

> Is there any chance you'd be able to
> share the scripts you guys have written?

Our locust scripts are on GitHub, but right now they are in a private repo :/

I definietly think they should be public, for other devs/ops to use... but I'll need to check the repo and see if anything needs to be sanitized.

Once I make the repo public, I'll follow up on this thread with details.


regards,
-Corey

--
You received this message because you are subscribed to the Google Groups "Open edX operations" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openedx-ops...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[TJ Keemon]

That would be awesome. Thanks Corey.

-TJ

[Pierre Mailhot]

Never too late to reply :) Especially since I haven't been reading this group myself in recent days...

We will be looking at Locust in the future. Right now, I still need to use JMeter in order to compare oranges with oranges when comparing our old system based on Sakai and ou new system based on Open edX.

On Friday, February 13, 2015 at 2:53:19 PM UTC-5, cgol...@edx.org wrote:

[Pierre Mailhot]

Count me in the group of those interested by these locust scripts.

On Tuesday, February 17, 2015 at 3:47:38 PM UTC-5, Corey Goldberg wrote:

[Nate Aune]

Corey - were you able to sanitize the Locust scripts? We have a big customer launch in a couple weeks, and we want to run the load tests soon so that we have ample time to optimize the infrastructure if the tests report back poor performance under heavy loads.

--
You received this message because you are subscribed to a topic in the Google Groups "Open edX operations" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openedx-ops/2CJTHjptpPM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openedx-ops...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nate Aune - nate...@gmail.com

http://www.nateaune.com

+1 (617) 517-4953

[Corey Goldberg]

Hi Nate,

> Corey - were you able to sanitize the Locust scripts?

I haven't yet, but will get to it soon.

I'll let you know how it progresses.

-Corey

[Anagha Mudigonda]

We'd be interested in the locust.io scripts also when you are able to get to them.

Best

Anagha

[Nate Aune]

Hey Corey,

Any progress on making the Locust scripts generally available? We'd really love to blog about them (hint, hint)! :

Nate