Check Kafka Queue while running Simulation

[Nathan]

Hello,

We are doing a load test on the distributed system using Gatling.

We send a request to Rest endpoint, which sends a bunch of messages to Kafka Topic for the consumers to process it.

One approach we are doing is to send 1000 request to the endpoint and want to check the Kafka Consumer lag using custom code, once all the messages are processed i.e lag is 0 will end the simulation.

how do I implement this in Gatling?

My First cut is

// First Scenario to post to Http Endpoint

val scn: ScenarioBuilder = scenario(scenarioName).repeat(10, "n") {
  exec { session =>
    session.set("id", random.nextInt(10000))
  }
    .exec(http("testendpoint")
      .post("/data")
      .body(ElFileBody("data.json"))
      .headers(httpHeaders)
    )
}

// Custom Action Builder 
val readLabelsBuilder = new ActionBuilder {
  override def build(ctx: ScenarioContext, next: Action): Action = {
    new LagMonitorAction()
  }
}

//Action. ( Ideally I want to check the Lag in Kafka Topics, but for now I just want to print something for debugging)
class LagMonitorAction extends ChainableAction  {
  override def name: String = "LagMonitor"
  override def next: Action = null
  override def execute(session: Session): Unit = {
    println ("I'm a custom Action")
 
  }
}

// Scenario with my custom Action Builder
val checkForKafkaConsumption = scenario("KafkaConsumption").exec(readLabelsBuilder)

val end2End = scenario("LoadTest").exec(scn).exec(checkForKafkaConsumption)

//Set up to execute the scenarios
setUp(
  end2End.inject(atOnceUsers(noOfUsers)).protocols(httpConf)
)

When I run the simulation, it executes the test i.e post 10 requests to the HTTP endpoint, then prints I'm a custom Action

After that the simulation continues does not end right after printing 'I'm a custom action', it waits for for-ever 

What am I doing wrong? 

How can I handle testing like this?

-Siva

[Nathan]

I'm using Gatling 2.3.0 along with Spring Boot.

[Cynthia Hertzer]

Did you get this working? I am doing something very similar (gatling v2.3) and I found this example helpful:

https://github.com/softmentor/gatling-examples/blob/master/gatling-custom-protocol-demo/src/test/scala/custom/protocol/test/PerfCustomProtocolSimulation.scala

My test is able to measure the throughput I computed with my version of a kafka lag monitor to the console. I don't write it to the actual gatling report. I don't think that the DataWriter.dispatch from the example works with 2.3. Not sure the that sort of thing is even supported in free Gatling.

-CH

On Wednesday, December 13, 2017 at 3:00:41 PM UTC-8, Nathan wrote:

[Stéphane LANDELLE]

There's a community (eg unsupported) plugin for Kafka: https://gatling.io/docs/current/extensions/

No idea what it's worth or it's actually maintained.

We're considering having one for FrontLine.

Stéphane Landelle

GatlingCorp CEO

slan...@gatling.io

--
You received this message because you are subscribed to the Google Groups "Gatling User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gatling+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cynthia Hertzer]

Thanks for the link. The plugin serves a different use case than the one I'm monitoring kafka for, but could come in handy for other scenarios.

-CH

To unsubscribe from this group and stop receiving emails from it, send an email to gatling+u...@googlegroups.com.

[ankita chandel]

I am trying to do something similar, could you find a solution for this ?

[Cynthia Hertzer]

Ankita, 

We use the KafkaConsumer to check lag on all the partitions every 5s. When lag drops to zero, we report the throughput via console logs. I do not post this result to the Gatling report.

Hope this helps,

-CH