Feeder file generation from a Gatling script?

[Nadine Whitfield]

I'd like to find out if it is possible to write  a Feeder file using a Gatling script.

I am using Gatling to test a RESTful API that offers methods for Users to create objects in the target system. 

Each object created by a user has a GUID that subsequently becomes associated with that user's ID number for a rather long time (6 mo).

I have a csv file of unique userIds as well as a test data generation script that can create objects for Users. However, the objectIDS are not currently captured; they go straight to the database.  This is good for the service being tested, but not so good for me.

What I would like to do is to have my script capture the GUID as soon as it is created and somehow link it to the owning UserId so I can use both values for later tests. Out of the UserIds I have, a few might get used twice in a single pass of the script, but the majority will get just one objectID per script run.

 Since these objects are likely to expire before the Users do, I would like to keep my User file intact and have this new file be available "on demand". 

What would I need to do to implement something like this?

 

[Stéphane Landelle]

Why do you want it it be a file? Why not a ConcurrentHashMap?

--
You received this message because you are subscribed to the Google Groups "Gatling User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gatling+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nadine Whitfield]

I was thinking of using a file because that is something I already know how to do, and it 

fits in well with the current implementation patterns I've been using.  

I was also concerned about resource usage, because lots of objects will be created during this test (upwards of 10k during any given test run).

If resources are no concern, I would certainly look into it deeper. 

What would be the advantages for use with Gatling? 

[John Arrowwood]

Are you needing this cache of GUIDS to live between runs of Gatling?  If no, then just throw memory at the problem and do it in RAM.  

If you need the IDs to be persisted and accessed in future runs, then a file or a database is the right thing to do.  

HOWEVER:  if you need both, you need a hybrid solution.  Because writing it to the file does not make it immediately available for a feeder, because feeders load everything into memory at once (most of the time, I believe).

[Nadine Whitfield]

The values will be used by future runs, so it looks like some kind of persistent storage is the best way to go.

 I found some forum posts that indicate I could access the GUIDS by using a Session function, but I'm a little mystified about how to get them from Gatling  Session to a file. 

[John Arrowwood]

In your simulation, before you define your scenario, you would open the file, something like this:

  val guid_data = {
        val fos = new java.io.FileOutputStream( "guid.csv" )
        val pw = new java.io.PrintWriter( fos, true )
        pw.println( "list,of,column,headers" )
        pw
      }

In the simulation, you would need to write to the file, like

guid_data.println( ... )

However, in this simplified model, you don't mix reads and writes. You run a scenario that generates the file, and a different scenario would depend on the data in the file.

If you need to mix reads and writes (like, I need the GUID, and if I get an error that says the GUID has expired, then get a new one and save it for next time), then you need to use a database, or use files on the filesystem as a cache.

[Nadine Whitfield]

Ah......ok. Now the lights are coming on!

So, this looks likes pretty regular code. The biggest "gotcha" is putting it in the right place to execute. 

I have a script that is dedicated for generating test data to be used by other Simulations, so this will work well for my use case. 

Thank you very much. 

[Nadine Whitfield]

@Stephane I'm revisiting this topic because now I actually have a chance to work on it  :)

I was thinking of putting the values into a CSV file so I could capture the data and retain it for multiple test runs. The associations I intend to capture are stored in multiple databases, so creating this file will help us test just this one microservice in isolation. 

However, I am curious about the ConcurrentHashMap. Are there any examples of this in the Gatling docs or on GitHub where I could see some implementations? Also, what would be the advantages over using a feeder file? 

[Stéphane Landelle]

I was only suggesting ConcurrentHashMap if you didn't need to retain the data between runs (like only storing in memory to share some data between scenarios).

Depending on how much you want to store on disk and how fast, you might want to properly buffer, or even retain everything in memory in a ConcurrentQueue, and only dump on file in the "after" hook.

Cheers,

Stéphane

[nadine.w...@nike.com]

Ah, ok. 

Well, I got the script to work beautifully. I can tell it how many rows to create, and it writes the CSV file just fine. Script consists of 7 requests that are chained together and share various attributes from the session object. 

 I now have a requirement to run this Scenario multiple times for a single virtual user. The number of times to repeat has to be a random Integer between 1 and some random max value that is provided at runtime (in this case from Jenkins). 

 

I thought I could accomplish this by wrapping the entire Scenario inside a repeat block, but now I'm getting a runtime error from Gatling with text like this>

[ERROR] [02/10/2015 18:16:41.450] [GatlingSystem-akka.actor.default-dispatcher-2] [akka://GatlingSystem/user/$a] requirement failed: Scenario CreateFullDataSet is empty
java.lang.IllegalArgumentException: requirement failed: Scenario CreateFullDataSet is empty
	at scala.Predef$.require(Predef.scala:233)
	at io.gatling.core.scenario.Simulation$$anonfun$scenarios$2.apply(Simulation.scala:40)
	at io.gatling.core.scenario.Simulation$$anonfun$scenarios$2.apply(Simulation.scala:40)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at io.gatling.core.scenario.Simulation.scenarios(Simulation.scala:40)
	at io.gatling.core.controller.Controller$$anonfun$1.applyOrElse(Controller.scala:80)
	at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
	at io.gatling.core.akka.BaseActor.aroundReceive(BaseActor.scala:23)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
	at akka.actor.ActorCell.invoke(ActorCell.scala:487)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
	at akka.dispatch.Mailbox.run(Mailbox.scala:220)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

What would be the best way to fix this error? 

Basic stripped layout of my code is below. My attempted changes are in Red. 

val scnMyFullScenario = scenario("CreateFullDataSet")

    // generate random number to set number of repeats

    val start = 1

    val rnd = new scala.util.Random

    val hits = start + rnd.nextInt( (maxRepeatsPerUser - start) + 1 )

      repeat( hits ) {  

         feed(senderFeeder)

        .feed(receiverFeeder)

        .feed(traceIdFeeder)

        .feed(timestamp)

        .feed(actionObjectFeeder)

        // place part of payload into Session for further processing

        .exec( session => { // doing stuff with session variables here 

         session })

        

       .exec( http("Request1"))).asJSON

            .check(status.is(200)

              , jsonPath("$.ids").saveAs("PostId"))

        )

        .pause(50 milliseconds)

        // remove unwanted characters from PostId

        .exec(session => { // more session work 

        session  })

        .exec( http("Request2" ))).asJSON

            .check(status.is(204))

        )

        .exec(http("Request3"))).asJSON

        .check(status.is(204))

        )

        .exec(http("Request4"))).asJSON

        .check(status.is(204))

        )

        .exec(http("Request5"))).asJSON

        .check(status.is(204))

        )

        .exec(

          http("Request6"))).asJSON

            .check(status.is(204))

        )

        .exec( http("Request7") )).asJSON

            .check(status.is(200),

              jsonPath("$.id").saveAs("CommentId"))

        )

        // remove unwanted characters from CommentId

        .exec(session => { // doing stuff with session attributes 

         session  })

        // extract values from Session and write to file

        .exec(session => { // extract attributes and set them up to be written in CSV format

        session })

    }

setUp(

      scnMyFullScenario.inject(rampUsers(numberRows) over (runTime seconds))

        .protocols(httpProtocol)

    )

[Nadine Whitfield]

Problem fixed. I had the right bits, but in the wrong order. 

The biggest problem was the placement of the repeat block. I moved this a couple of lines later so the feed statements could set up a SenderId and ReceiverId first. Then I used the repeat block to wrap the following feed statements and subsequent code that creates a post, manipulates session attributes and writes the line to a file. 

Then of course, it is important to be extra aware of placement of the dot operator. I have not seen any formal documentation of this, but have noticed a correlation between placement of dot operators and sections of block code like .group, .repeat, .asLongAs, etc - the first statement after the opening has to have the dot operator removed. 

After working on this problem, I've learned to make dot operators one of the first places I look when there are problems. 

[Srinivas D]

Hello nadine.w...@nike.com

could you post the code how the session attributes are extracted and written to csv format

ref in the section 

 // extract values from Session and write to file

        .exec(session => { // extract attributes and set them up to be written in CSV format

        session })

Would of great help.

Regards

[Nicholas Ling]

please. It's lack of documentation.

[infomaven]

Hi Srinivas, Nicholas Ling  - 

So sorry! I've been off the mailing list for this group and just found your questions. 

You probably figured it out by now, but I did want respond. 

The solution I came up with uses the Gatling session (which is essentially a giant Map structure) to capture the values. 

Session is then passed to a custom Scala function that extracts data and forms it into a comma-delimited row of data that can be written to a file.

Code is here on Github> https://gist.github.com/infomaven/1d9393c83d57eef7ae72e2b2818be718#file-gatlinggeneratedfeeder-scala

Disclaimer - this code has only been tested  with Gatling 2.0. 

Some of the syntax might be different now, but the basic  concepts should not change very much. 

Regards. 

- infomaven