Sometimes scenario not terminating

1,651 views
Skip to first unread message

John Arrowwood

unread,
Mar 4, 2015, 1:31:57 PM3/4/15
to gat...@googlegroups.com
I am doing bulk data loading through Gatling.  I need to step through 5.5 million records in a .csv file.  WAY too many to load in at once, so I broke the data into 1000 line .csv files, and use a symlink to point to the "current" chunk of the data.  I have a scenario (see below) that plows through the list and makes the necessary requests.  When the feeder runs out, everything shuts down, and it's all good.  USUALLY.

In a wrapper script, I consult a configuration file (so I can turn on and off the data load based on time of day), and if the time of day says no-data-load, then it just sleeps.  Otherwise, it points the symlink to the next file, and restarts the scenario.

After a couple of weeks of working just fine, something weird happened.  Over the weekend, Gatling sat there doing nothing for over 200,000 seconds.  So Monday I added a maxDuration of 2,000 seconds.  Last night, around midnight, it locked up again and in spite of the maxDuration, it sat there and spun until around 8 am.  So far this morning, it has done it at least 3 more times.

NORMALLY, when it exits "cleanly" there is a stack trace right after the line that says "13:12:59.163 [ERROR] i.g.c.a.SingletonFeed - Feeder is now empty, stopping engine"

When it doesn't exit cleanly, there are log entries about dead letters, and then it sits there indefinitely.

Any ideas about what could be causing this?  Or how to fix it?  (I am on 2.0.1 at the moment, I will update to 2.2 when it goes gold)

Here's what the simulation looks like:


val findUser =
    scenario
("Find User")
   
.exec( SessionVariables.initialize )
   
.exec( RTDE.Login.useExistingTrustedClientToken )
   
.exitHereIfFailed
   
.forever {
       feed
( csv("add-to-pvs.csv") )
     
.exec( _.set( INDIVIDUAL_ID, "*NOT-FOUND*" ) )
     
.exec( search( "CED" ) )
     
.exec( search( "HCMS" ) )
     
.exec( search( "PYSL" ) )
     
.exec( search( "FML" ) )
     
.exec( search( "GWFC" ) )
     
.exec( search( "CBHR" ) )
     
.exec( search( "GADB" ) )
     
.exec( search( "CBH" ) )
     
.exec( search( "DMND" ) )
     
.exec( search( "BEN" ) )
     
.exec( search( "ACLM" ) )
     
.exec( search( "IFP" ) )
     
.exec( search( "CPTK" ) )
     
.exec( search( "QCRE" ) )
     
.exec( search( "WDEN" ) )
     
.exec( search( "CWEB" ) )
     
.exec( search( "CIEB" ) )
     
.exec( search( "RA" ) )
     
.exec( search( "PCLG" ) )
     
.exec( search( "MRR" ) )
     
.exec( search( "CPI" ) )
   
}

  setUp
(
   
TrustedLogin.flow.inject( atOnceUsers(1) ),
    findUser
.inject(
      nothingFor
( 30 seconds ),
      rampUsers
( Test.numUsers ) over ( Test.rampTime )
   
)
 
)
 
.maxDuration( Test.duration )



John Arrowwood

unread,
Mar 5, 2015, 1:54:25 PM3/5/15
to gat...@googlegroups.com
Still suffering with this problem.  I'm leaning towards Gatling doing the "wrong thing" when the feeder is empty.  To see if that could be the source, I have a question:

What EXACTLY happens when virtual user 7 of 9 reaches a .feed() node, and the feeder says "I got nothin'"?  Does it terminate THAT virtual user, or does it terminate the whole scenario?

I assert that it SHOULD only terminate that virtual user.  Let virtual user 7 terminate now, let the rest of them terminate when they need data and can't get it.  But it does not look like that is what is happening, at least not in 2.0.1.

Thoughts?

Stéphane LANDELLE

unread,
Mar 6, 2015, 3:42:27 AM3/6/15
to gat...@googlegroups.com
No, the opposite happens: simulation is terminated. Not being able to feed is an unexpected behavior, and most likely a bug. Would you really want to have a 48hr simulation keep on running while it's not able to feed logins and passwords?

Stéphane Landelle
Lead developer


--
You received this message because you are subscribed to the Google Groups "Gatling User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gatling+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

John Arrowwood

unread,
Mar 6, 2015, 4:29:51 PM3/6/15
to gat...@googlegroups.com
There are different behaviors that make sense in different circumstances.

USE CASE:  Single flow scenario - The user logs in, flows through the application, and then terminates.  The simulation is driven by constantUsersPerSec()

In this case, it makes sense to allow the user to finish its flow and let it terminate itself.

USE CASE: Looping flow - In a loop, the user gets the data, and uses it to log in, do its flow, and then loop back.  The simulation has a certain number of users.

In this case, it makes sense to terminate when the virtual user gets back to the .feed() 

USE CASE: Do something forever - read from the feeder once, then in a loop do some actions using that user.

In this case, what if the tester tries to inject 100 users, but there are only 98 records in the feeder?  Then it makes sense to terminate the simulation, in a way.  At the same time, using .exitHereIfFailed would be a perfectly viable alternative.  The number of active users would only be 98, but at least you would get a test run with 98 users, instead of a test run that dies because you don't have enough data.

IN MY OPINION - and it is only my opinion, of course - I should have the OPTION of continuing the simulation until it naturally terminates.  

One way to get what I'm thinking would be to have another feeder behavior.  In addition to .random and .circular, there could be a .exitWhenEmpty.  So, for example, 

  .feed( csv( some-file ).exitWhenEmpty )

Then when .feed() tries to read from the feeder, and the feeder is empty, then the virtual user exits rather than terminating the whole simulation.

Thoughts?

John Arrowwood

unread,
Mar 6, 2015, 4:31:28 PM3/6/15
to gat...@googlegroups.com
Addendum - see below ...


On Friday, March 6, 2015 at 1:29:51 PM UTC-8, John Arrowwood wrote:
There are different behaviors that make sense in different circumstances.

USE CASE:  Single flow scenario - The user logs in, flows through the application, and then terminates.  The simulation is driven by constantUsersPerSec()

In this case, it makes sense to allow the user to finish its flow and let it terminate itself

BUT THE user injection should stop, obviously...

Stéphane LANDELLE

unread,
Mar 8, 2015, 5:46:29 PM3/8/15
to gat...@googlegroups.com
Doable, but requires quite some work, and I don't feel like making it a priority without more community demand.

Cheers,

Stéphane Landelle
Lead developer


John Arrowwood

unread,
Mar 8, 2015, 5:49:26 PM3/8/15
to gat...@googlegroups.com
I can work around it.  But if you ever find yourself in that code... :)
You received this message because you are subscribed to a topic in the Google Groups "Gatling User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gatling/uJOaKoKrVos/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gatling+u...@googlegroups.com.

András Gaál

unread,
Dec 1, 2016, 4:09:15 AM12/1/16
to Gatling User Group
Hi John,

what is your workaround please?

I would like to do the same configuration in gatling.

Run tests until there is data in the feeder. If feeder is empty stop the tests, but do not fail them.

Regards,
Andras

Javier Ferrer González

unread,
Feb 20, 2017, 10:04:03 AM2/20/17
to Gatling User Group
+1 for the feature request of gracefully ending the test when the feeder doesn't have more items.

Barry Perez

unread,
Feb 20, 2017, 2:45:29 PM2/20/17
to Gatling User Group

+1 to making the behaviour configurable.

Taking the example of a big load test with a large number of scenarios - if a single scenario runs out of data then I would generally prefer to be able to have that throw errors/warnings and allow the wider simulation to continue. Any results are better than no results!

Barry

adrian...@hushmail.com

unread,
Feb 20, 2017, 3:28:01 PM2/20/17
to gat...@googlegroups.com
Why not use the circular method?

csv("urls.txt").circular

Aidy


On 2/20/2017 at 3:04 PM, "Javier Ferrer González" <javier.m...@gmail.com> wrote:
>
>+1 for the feature request of gracefully ending the test when the
>feeder
>doesn't have more items.
>
>On Thursday, December 1, 2016 at 10:09:15 AM UTC+1, András Gaál
>wrote:
>>
>> Hi John,
>>
>> what is your workaround please?
>>
>> I would like to do the same configuration in gatling.
>>
>> Run tests until there is data in the feeder. If feeder is empty
>stop the
>> tests, but do not fail them.
>>
>> Regards,
>> Andras
>>
>> On Sunday, March 8, 2015 at 10:49:26 PM UTC+1, John Arrowwood
>wrote:
>>>
>>> I can work around it. But if you ever find yourself in that
>code... :)
>>>
>>> On 3/8/15 2:46 PM, Stéphane LANDELLE wrote:
>>>
>>> Doable, but requires quite some work, and I don't feel like
>making it a
>>> priority without more community demand.
>>>
>>> Cheers,
>>>
>>> *Stéphane Landelle*
>>> *Lead developer*
>>> slan...@gatling.io
>>>>>> *Stéphane Landelle*
>>>>>> *Lead developer*
>>>>>> slan...@gatling.io

Barry Perez

unread,
Feb 21, 2017, 4:48:19 AM2/21/17
to Gatling User Group
Hi Aidy,

That would work for some people, but bearing in mind that some data can only be used once, configurable behaviour when all data has been used would make Gatling a lot more flexible (doesn't impact me too much as I tend to use Redis for any burnable data).

Barry

Stéphane LANDELLE

unread,
Feb 21, 2017, 5:33:52 AM2/21/17
to gat...@googlegroups.com
Hey,

First, I think we can all agree that hitting a feeder starvation is a test design flaw. Even if we were to keep on running and make users who hit a starved feeder exit, you'd get less load than we you expected.
Then, as a user, how would you realize that you hit a feeder starvation? Check the logs? Most users would fail to do that and realize it, and we would end up with tons of invalid issues.

Stéphane Landelle
GatlingCorp CEO


To unsubscribe from this group and stop receiving emails from it, send an email to gatling+unsubscribe@googlegroups.com.

John Arrowwood

unread,
Feb 21, 2017, 7:10:58 AM2/21/17
to gat...@googlegroups.com
Yes, if we are using Gatling exclusively as a load testing tool, then it might be said to be a test design flaw.

But you made such an awesome tool that not all of us use it exclusively for generating load.  It is perfectly suited to parallel process automation, too, e.g. data loading.  Which is why this behavior is annoying: it makes it harder to use Gatling for something it wasn't designed for. :)

To unsubscribe from this group and all its topics, send an email to gatling+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
John Arrowwood
John (at) Irie (dash) Inc (dot) com
John (at) Arrowwood Photography (dot) com
John (at) Hanlons Razor (dot) com
--
http://www.irie-inc.com/
http://arrowwood.blogspot.com/
Reply all
Reply to author
Forward
0 new messages