SSL throughput issues

[Odin Hole Standal]

We're having throughput issues when using SSL and our theory right now is that the Gatling client is the issue. 

Our setup:

- we have Apache in front of a SOAP web service 

- Apache handles SSL and forwards HTTP requests to the SOAP web service 

- gatling version 2.0.0-RC2

Simulation: Running with no SSL directly against the SOAP web service

- simulation that runs for 1 minute with 10 users: 10000 requests per minute 

Simulation: Running against Apache with SSL

- a simulation that runs for 1 minute with 1 user: 1000 requests per minute

- a simulation that runs for 1 minute with 10 users : 14 requests per minute

When we only get 14 requests per minute; Gatling reports that the max response time was 1 second. So it looks like Gatling hangs for a long time between making requests.

We ran the same SSL simulation with JMeter with 10 concurrent users and got 10000 requests per minute, so we are pretty confident that Apache isn't the bottleneck.

Does anyone have suggestions for configurations that we can tweak or how we can go about debugging this issue?

[Nicolas Rémond]

Simulation: Running with no SSL directly against the SOAP web service

- simulation that runs for 1 minute with 10 users: 10000 requests per minute

It seems that you are recycling the users, that isn't the Gatling way of doing things.

So, really, without seeing the scenario, we can't really help here.

--
You received this message because you are subscribed to the Google Groups "Gatling User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gatling+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Bagehot]

I'd also want to see

3 thread dumps 10 seconds apart
3x netstat -na dumped to file

Thanks
Alex

[Odin Hole Standal]

I failed to say that we ran the client on Windows 7. We just ran the same simulation on Linux and the SSL throughput was 10000 requests per minute. 

Is there some windows based configuration we could tweak perhaps?

By the way, impressed with having two responses in such a short amount of time :)

Sincerely

Odin

[Alex Bagehot]

thanks, Like Nicolas said, it would be useful to see a minimal test case (likely mainly the http protocol options and inject config)

and for my part the thread dumps and netstat :)

there may be other ways but a thread dump is usually a good starting point to provide some visibility on where the delay might be.

thanks

Alex

[Odin Hole Standal]

This is the test case we use (can't share the actual code due to company policy):

package sim

import io.gatling.core.Predef._

import io.gatling.http.Predef._

import scala.concurrent.duration._

class MySimulation extends Simulation {

val myData = csv("some.csv").circular

val httpProtocol = http

.baseURL("https://some-url")

.inferHtmlResources()

.acceptEncodingHeader("gzip,deflate")

.authorizationHeader("Basic casdkajslkqwe=")

.contentTypeHeader("text/xml;charset=UTF-8")

.userAgentHeader("someclient")

val headers = Map("SOAPAction" -> "")

val scn = scenario("My Scenario")

.feed(myData)

.during(1 minutes) {

                    exec(http("req_0")

.post("/some/path")

.headers(headers)

.body(StringBody(""""

...valid soap request ....

""".stripMargin.trim()))

.basicAuth("abc", "123"))

}

setUp(scn.inject(atOnceUsers(10))).protocols(httpProtocol)

}

Additionally, we use the default gatling.conf settings.

[Alex Bagehot]

Hi Odin,

What Java version are you using? ensure the latest JDK(not jre) is installed and on the path.

I can't see anything particular from the script - did you manage to get a thread dump? 

You can use the jstack command in the java jdk bin directory to get a thread dump from the pid of gatling.

when running netstat .... can you look for any lines with "SYN"??

I understand if you don't want to send the whole output but connections in SYN* state would indicate a "hangs for a long time between making requests". Currently gatling doesn't include the connection time in the latency figures.

just to simplify the numbers  and scope, can you run just a plain get on a static file on the apache server (you could copy a file to the document root for diagnostic purposes), and only inject 1 user, and report the throughput and latency? thanks

Thanks

Alex

[Odin Hole Standal]

Hi Alex,

We're using Java version 1.8.0_11-b12. When we ran with only one user instead of 10, we got 1000 requests per minute, so the problem is concurrency related.

netstat -an shows:

- 10 TCP-connections to the server when running without SSL

- 1 (sometimes 2) TCP connections when running with SSL 

I'll try to get the thread dumps and post the details here. 

In the meantime we've solved the problem by running the tests on Linux so this isn't a showstopper for our Gatling proof of concept any more. Thankfully I should add, because Gatling is so much better to work with than JMeter :)

Sincerely

Odin

[Odin Hole Standal]

Hi Alex,

Here are the thread dumps, see https://dl.dropboxusercontent.com/u/296943/gatling_thread_dump.zip

My gut feeling is that there is something with my Windows setup that is causing the issue.

Sincerely

Odin 

On Friday, August 15, 2014 3:43:13 PM UTC+2, Alex Bagehot wrote:

[Alex Bagehot]

looks like you may have been hitting this:

http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6450279

$ grep Inet4AddressImpl.getHostByAddr thread_dump*

thread_dump1.txt:    at java.net.Inet4AddressImpl.getHostByAddr(byte[ ])

thread_dump2.txt:    at java.net.Inet4AddressImpl.getHostByAddr(byte[ ])

thread_dump3.txt:    at java.net.Inet4AddressImpl.getHostByAddr(byte[ ])

  Thread "GatlingSystem-akka.actor.default-dispatcher-19":

    at java.net.Inet4AddressImpl.getHostByAddr(byte[ ])

    at java.net.InetAddress$2.getHostByAddr(byte[ ])

    at java.net.InetAddress.getHostFromNameService(java.net.InetAddress, boolean)

    at java.net.InetAddress.getHostName(boolean)

    at java.net.InetAddress.getHostName()

    at java.net.InetSocketAddress$InetSocketAddressHolder.getHostName()

    at java.net.InetSocketAddress$InetSocketAddressHolder.access$600(java.net.InetSocketAddress$InetSocketAddressHolder)

    at java.net.InetSocketAddress.getHostName()

    at com.ning.http.client.providers.netty.channel.SslInitializer.connectRequested(org.jboss.netty.channel.ChannelHandlerContext, org.jboss.netty.channel.ChannelStateEvent)

...

[Odin Hole Standal]

Hi Alex,

I put a DNS record in my hosts file and ran the simulation again; the problem is gone.

Thank you so much for your help!:)

Sincerely

Odin

[Stéphane Landelle]

Hi,

The problem was that some part of AsyncHttpClient was using InetAddress.getHostName, which can cause a reverse DNS lookup.

I changed targeted JDK to JDK7 so we can use getHostString that doesn't perform this lookup.

Next Gatling RC3 will ship the fix.

Thanks for reporting!

Stéphane