REST API perf/load testing

[Manoj Agarwal]

We discussed about testing REST API here about an year ago, when people recommended SoapUI/LoadUI, JMeter and frisbyjs.

 

I am wondering if there are any new tools available now that you would recommend for REST API testing, or any significant enhancements in SoapUI/LoadUI and JMeter that makes them your favorite over others.

 

Thanks,

Manoj

[Frode Rosand]

If you looking for something simple to do load/per testing check out http://loader.io from SendGrid. They support all verbs now and you can scale your tests pretty high.

Frode

[Sarfraz Ahmed]

You can try with Apache Bench, its very lightweight and has nice features for load testing. 

Another, you can try with The Grinder, it is a JavaTM load testing framework.

Regards,

Sarfraz

--
You received this message because you are subscribed to the Google Groups "API Craft" group.
To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+...@googlegroups.com.
Visit this group at http://groups.google.com/group/api-craft.
For more options, visit https://groups.google.com/groups/opt_out.

[Felipe Sere]

Do any of the *js load/perf testing toe support sequences of calls? I know I can do so using JMeter but it's a nuisance to maintain is far from being as readable as code...

Sent from my iPhone

[Kyle Conroy]

TechEmpower has done a series of fantastic web framework benchmarks. They use wrk for their load testing. I've also had success using vegeta, which is written in Go and easy to use.

[Peter Salas]

Here is what I've tried and what I've liked and didn't like about each:

1. SoapUI/LoadUI - GUI based performance (and functional) benchmarking tool. I have not used it extensively and recently, so I won't go too much into it. I think pretty much JMeter offered the same features, so I have opted to use JMeter over this, but I think that's a personal preference versus technical reason.
2. JMeter - Another GUI based performance benchmarking tool. It's really easy to use for simple API testing but a little bit troublesome to do more complex chaining of requests when a workflow is necessary. I've also suffered with one PUT request that needed a new unique ID as part of the request; there was a problem with that feature in JMeter at that time, and I think I ended up abandoning it for a different tool (e.g. Grinder or Faban)
3. ApacheBench - Really easy CLI-based tool for very quick load/perf testing. It won't be able to do complex API's or simulate a workflow.
4. Grinder - Built on jython and easily extensible using Java to do anything you want. OOTB, it supports testing web applications and support for record-and-playback feature for GUI performance testing. I've used it for both front and back-end performance testing. One negative has been scaling  when you need multiple agents to generate the load; and aggregating report data across multiple agents after a run. This is not hard to do, but it would have been nice for Grinder to support that feature so that I didn't have to do post-processing.
5. Faban - Requires a bit more work to setup because you must compile a "performance profile" before you can execute it. However, it is highly customizable, extensible, and simple to kick off a new run after properly configuring. An advantage/disadvantage is that the tool does the testing at the TCP layer; therefore, it's not for the faint-of-heart for development if you are not comfortable dealing with testing at that layer.

It looks like Vegeta might be something I should look into. It looks like a combination of ApacheBench and Grinder, which is a good thing in my book! I prefer tools that are easy-to-use OOTB, and simple-and-elegant extensibility; I haven't really found that, so hence the myriad of tools that I trade back-and-forth on.

- Peter

[Emmanuel Paraskakis (@manp)]

John Sheehan of Runscope gave a great talk on the subject of testing at APIStrat last week - here are the slides:

https://speakerdeck.com/johnsheehan/free-api-debugging-and-testing-tools-you-should-know-about

VCR and frisby are mentioned

[Kevin Brown]

Another candidate for load testing with complex scenarios:

http://tsung.erlang-projects.org/

On Wed, Oct 30, 2013 at 6:46 PM, Kris Kleva <kris...@gmail.com> wrote:

I know your question was specifically about tools.  However, I'd like to understand your perspective on when it's appropriate to load test your apis and perhaps with those requirements in mind the tooling choice would be easier.

One opportunity for improvement I've been promoting is to integrate concise, use case based performance / load tests as part of the continuous integration process.   The ideal state is that when a commit is pushed to a "origin" or "master" branch it would be automatically deployed, then kick off load tests and fail (or assertively warn) me if a particular version has been demonstrating a decline in performance.

So, IMHO go with the tool with the best API to plug into your existing continuous integration processes and don't wait to perf tests a few weeks before launch or peak season (like I have before).  

My current tool of choice:

http://www.compuware.com/en_us/application-performance-management/products/lifecycle-performance-management/web-load-testing.html

More pricey than some of the other solutions mentioned.  However, when I've compared to other options (LoadUI/JMeter/ApacheBench) it produced the most realistic load/perf results for what we're expecting from the real world public facing API traffic.