[Cucumber] Using cucumber as the basis for a load testing application.

[RurouniJones]

Howdy everyone.

I am looking at load testing my applications and, as well as the GUI
load testing programs out there I found out about trample (http://
github.com/jamesgolick/trample), a command-line, written in ruby, load-
testing gem. Unfortunately Trample development is a bit...er,
fractured, it is a bit limited at the moment and I am having
miscellaneous trouble with it.

It also irked me that I have to enter URLs to test all over again in
Trample's own DSL when I had a full set of cucumber features that
interact with my application.

Then I started thinking about how cucumber could be modified /
extended to form the base of a load testing application. We already
have a way of defining user journeys through-out the system (gherkin)
so we can create virtual users as a single "feature", we already have
a way of executing them and checking responses (Webrat / Capybara
etc.) and application specific steps are already defined in our step
definitions as well as the basic web steps.

The only thing that appears to be missing is a harness that would
allow a user to setup the parallel tests and then do some benchmarking
on response times etc.

Now I am a journeyman dev at best, I have started looking into the
cucumber source-code and it is obvious that, if I were to do this
myself, it will take a while. So before I start down that road I
wanted to ask the experts on this list.

1. Does what I am suggesting have a ring of sanity to it or am I
missing something obvious that would make this a waste of time to
pursue
2. Has anyone else thought of something along similar lines? If so,
any information about it anywhere? I have googled about but didn't see
anything obvious.

Cheers

RJ

[Chuck van der Linden]

I'm not a giant fan of any UI based loadtesting tools because they
don't tend to scale well due to the overhead of having to render the
UI, do clientside processing, etc etc. That's why most of the well
regarded loadtesting tools operate at the protocol level, where you
can support often upwards of 1000 or more vusers from a single system.
(just try running 500 to 1000 simultaneous UI sessions on a single
box.. )

Also with loadtesting you may not be concerned with the same sorts of
validations as with functional, and surely you are far more interested
in instrumenting the performance stats of the various servers in the
system under test. (get the free download of "Performance Testing
Guidance" that the ms patterns and practices folks put out.. it covers
a lot of this and is surprisingly neutral in terms of specific
technology)

The other problem is that loadtests need lots of repeatable data, e.g.
the test for 'new user registration' needs to be able to register
thousands of new users over the duration of the test, that means a
thousand rows of unique data.. Where you scenarios in cucumber to
create a new user likely have only a single row, or a small handful at
most.

If you try to just run your functional tests multiple times against a
load environemnt, you'll find lots of tests that just start failing..
tests to delete things fail because the thing already got deleted,
tests to create new unique things fail because the thing already
exists. Tests to buy an item fail because the prior 50 runs of that
test have now reduced the inventory to zero (and besides who wants
your loadtest users all buying the one same item over and over and
over) .

You end up re-writing everything to pull large quantities of unique
data from a csv file or database, and then you have to coordinate so
parallel threads don't try to use the same data, etc, etc,

OTOH, most of the loadtest tools I've used are setup to cope with all
that. Most of them support a 'session recorder' that will capture all
the HTTP traffic from a sample session, which you then use as the
basis for the loadtest (setting up parameterization and capturing of
the critical data that changes like session ID's, usernames, item
numbers, etc. ) With a good tool like VSTS, an experience person can
create the test pretty rapidly that way

So you might well be able to just run specific cucumber features as a
way to generate those. Fire up the recorder, execute the cucumber
features that represent the user actions you want to simulate, and go
from there using the stuff that's in the tool to parameterize inputs,
expected results, etc.

[Rory Gibson]

You could try using JMeter (free, Apache), which comes with a
recording proxy.

Run the proxy on your local machine, point your Selenium / Webrat
driver at the proxied
port, and run the Cucumber tests.

This wil give you a set of JMeter scripts that precisely represent
your Cucumber specs, and can then be run using JMeter's distributed
testing agents, with results gathered and viewed via the GUI.

Best of both worlds.

Hope that's useful,
Rory Gibson

[George Dinwiddie]

Hi, Chuck,

I'd just like to challenge what seems to be a few assumptions behind
your comments.

On 8/28/10 2:16 AM, Chuck van der Linden wrote:
> I'm not a giant fan of any UI based loadtesting tools because they
> don't tend to scale well due to the overhead of having to render the
> UI, do clientside processing, etc etc. That's why most of the well
> regarded loadtesting tools operate at the protocol level, where you
> can support often upwards of 1000 or more vusers from a single system.
> (just try running 500 to 1000 simultaneous UI sessions on a single
> box.. )

Cucumber tests do not have to access the application through the UI.
And not all UI testing clients do rendering.

> Also with loadtesting you may not be concerned with the same sorts of
> validations as with functional, and surely you are far more interested
> in instrumenting the performance stats of the various servers in the
> system under test. (get the free download of "Performance Testing
> Guidance" that the ms patterns and practices folks put out.. it covers
> a lot of this and is surprisingly neutral in terms of specific
> technology)

It seems perfectly feasible for a Cucumber test to validate performance
stats. You would not likely use functional-test scenarios to do your
load testing.

> The other problem is that loadtests need lots of repeatable data, e.g.
> the test for 'new user registration' needs to be able to register
> thousands of new users over the duration of the test, that means a
> thousand rows of unique data.. Where you scenarios in cucumber to
> create a new user likely have only a single row, or a small handful at
> most.

There's no need to hand-code a thousand rows of unique data. You have
all the power of ruby to generate them.

> If you try to just run your functional tests multiple times against a
> load environemnt, you'll find lots of tests that just start failing..
> tests to delete things fail because the thing already got deleted,
> tests to create new unique things fail because the thing already
> exists. Tests to buy an item fail because the prior 50 runs of that
> test have now reduced the inventory to zero (and besides who wants
> your loadtest users all buying the one same item over and over and
> over) .

If you can test more creatively with other tools, you can certainly do
so with Cucumber also.

There may be efficiency limits with using Cucumber, but I think that
most of the limitations you see are products of your own vision, not of
Cucumber.

cheers,
George

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

[RurouniJones]

Thank you for the insight for the various parties.

I just wanted to clarify that I am not planning on literally re-using
my functional tests in load testing but using gherkin along with the
step definitions already created for the functional tests which would
allow us to quickly, easily, and in a business language create load
tests.

Regarding random data, that would be easily solved using sequences in
factories and things like faker / populator to create semi or totally
random data.

I have used various GUI load testing tools before and always found
them wanting on various levels, I usually loath their data
substitution / scripting languages and would much rather use the
flexibility of Ruby combined with Cucumber (Gherkin).

With regards to using other CLI tools: again I can see that they are
useful and have used some of them before but again that means
reinventing the wheel in terms of writing scripts / checks / paths
etc.

Regards

RJ