Gatling performance test causing tons of sockets in CLOSE_WAIT

[Siegfried Goeschl]

Hi folks,

I was helping a colleague with a Gatling script to do some pre-production testing - basically sending some JSON over a REST API as part of a server to server communication

Initially we were puzzled that Gatling reported a few long-runners not shown in the server's access.log - re-running the same tests with JMeter showed that the long-runners are caused by the time of connection to the server (this is not directly visible in Gatling AFAIK).

So we had a look at the numbers of connections being used on the load injector box and came across many connections in CLOSE_WAIT (might explain the behaviour above)

• Each & every request seems to leave one socket in CLOSE_WAIT (~5.000 after a short test run)
• The script starts a user & session and only executes a single request so we create a lot of session and virtual users (~5.000)
• We did compare the sockets being used with "Apache AB" and that test run leaves no connections behind so the problem stems from the Gatling script

We set "keepAlive=false" in gatling.conf which IMHO should do the trick but it had not the desired effect 

• we left a lot fewer connection behinds but still growing (probably 80% less but I have no hard number yet) 
• we still don't see the behaviour of "Apache AB" where 20 workers used a maximum of 21 sockets
  
• the test run suddenly had a lot more long-runners 

Having said that I'm a bit puzzled - what I'm doing wrong in terms go HTTP connections and sockets?

Thanks in advance, 

Siegfried Goeschl

PS: My colleague is using Windows (he was given little choice) and Gatling 2.2.3 - not perfect :-(

[Stéphane LANDELLE]

I'm sorry, but I'm afraid we can't do anything without a way to reproduce your issue (with up-to-date Gatling version).

Stéphane Landelle

GatlingCorp CEO

slan...@gatling.io

--
You received this message because you are subscribed to the Google Groups "Gatling User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gatling+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Siegfried Goeschl]

Hi Stéphane, 

understood :) - my colleague moved to 2.3.0 already to no avail. I'm teaming up with him tomorrow to provide more details to reproduce the problem 

[Stéphane LANDELLE]

Good luck :)

Also note that antivirus on Windows are rarely load testing friendly...

Stéphane Landelle

GatlingCorp CEO

slan...@gatling.io

[Siegfried Goeschl]

The follow up - sorry for the dealy but I was sick in the mean time :-()

* I took the liberty to use the wellknown http://computer-database.gatling.io

* I updated to Gatling 2.3.0

* I had a quick look on Mac OS X (no anti-virus)

IP Address

======================================

> nslookup computer-database.gatling.io

Server:     195.34.133.21

Address:    195.34.133.21#53

Non-authoritative answer:

Name:   computer-database.gatling.io

Address: 35.158.229.206

GATLING

======================================

When running a simple test hitting http://computer-database.gatling.io using Gatling I get

Request DefaultFullHttpRequest(decodeResult: success, version: HTTP/1.1, content: EmptyByteBufBE)

GET /computers?p=0 HTTP/1.1

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8

Accept-Encoding: gzip, deflate

Accept-Language: en-US,en;q=0.5

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20100101 Firefox/16.0

Referer: http://computer-database.gatling.io/computers/525

Host: computer-database.gatling.io

Response DefaultHttpResponse(decodeResult: success, version: HTTP/1.1)

HTTP/1.1 200 OK

Server: nginx/1.10.2

Date: Thu, 23 Nov 2017 23:33:09 GMT

Content-Type: text/html; charset=utf-8

Content-Length: 7254

Connection: keep-alive

Keep-Alive: timeout=5

> netstat -a -n | grep 35.158.229.206

tcp4       0      0  192.168.0.10.65392     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.65393     35.158.229.206.80      TIME_WAIT  

Apache AB

======================================

Executing the same request with Apache AB leaves no IP connections in "TIME_WAIT"

> ab -v 4 -n 2 http://computer-database.gatling.io/computers?p=0

Benchmarking computer-database.gatling.io (be patient)...INFO: GET header == 

---

GET /computers HTTP/1.0

Host: computer-database.gatling.io

User-Agent: ApacheBench/2.3

Accept: */*

---

LOG: header received:

HTTP/1.1 200 OK

Server: nginx/1.10.2

Date: Thu, 23 Nov 2017 23:39:11 GMT

Content-Type: text/html; charset=utf-8

Content-Length: 7254

Connection: close

> netstat -a -n | grep 35.158.229.206

>

Conclusion

======================================

* Apache AB uses HTTP 1.0 protocol so the server knowns to close the connection immediately 

* Gatling use HTTP 1.1. and keep alive connection are the default - netstat consequently reports the conenction as "TIME_WAIT"

* Setting "keep-alive" to false has no effect which is not really a surprise since the setting makes only sense with HTTP 1.0

So the question is - is it possible to revert to HTTP 1.0? I did not find any setting in "HttpProtocol"

Thanks in advance

Siegfried Goeschl

[Siegfried Goeschl]

Please note that this is more or less an academic question for my problem

• The test setup for a virtual user only issuing a single request probably does not reflect reality (but I need to double-check)
• I'm still puzzled with the current implementation of "keep-alive" but I also need to double-check hitting the test server

But I still want to provide a complete answer for my colleague :-)

[Stéphane LANDELLE]

Sorry, can't reproduce.

Stéphane Landelle

GatlingCorp CEO

slan...@gatling.io

--

[Siegfried Goeschl]

Hi Stephane,

that is somehow surprising since this would be the expected behaviour - having said that it depends on the operating system, gatling.conf and kernel settings.

In my case running on Mac OS X 10.13.1 with the official Gatling download

gatling-charts-highcharts-bundle-2.3.0> sw_vers

ProductName: Mac OS X

ProductVersion: 10.13.1

BuildVersion: 17B48

gatling-charts-highcharts-bundle-2.3.0> java -version

java version "1.8.0_121"

Java(TM) SE Runtime Environment (build 1.8.0_121-b13)

Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

gatling-charts-highcharts-bundle-2.3.0> ./bin/gatling.sh --simulation computerdatabase.advanced.AdvancedSimulationStep05

gatling-charts-highcharts-bundle-2.3.0> netstat -a -n | grep 35.158.229.206

tcp4       0      0  192.168.0.10.60067     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.60069     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.60070     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.60068     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.60071     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.60072     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.60074     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.60075     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.60076     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.60073     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.60077     35.158.229.206.80      TIME_WAIT  

tcp4       0      0  192.168.0.10.60078     35.158.229.206.80      TIME_WAIT  

[Stéphane LANDELLE]

MacBook-Pro-de-slandelle:bin slandelle$ sw_vers

ProductName: Mac OS X

ProductVersion: 10.11.6

BuildVersion: 15G17023

MacBook-Pro-de-slandelle:bin slandelle$ java -version

java version "1.8.0_152"

Java(TM) SE Runtime Environment (build 1.8.0_152-b16)

Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

MacBook-Pro-de-slandelle:bin slandelle$ ./gatling.sh --simulation computerdatabase.advanced.AdvancedSimulationStep05

MacBook-Pro-de-slandelle:bin slandelle$ netstat -a -n | grep 35.158.229.206

MacBook-Pro-de-slandelle:bin slandelle$

¯\_(ツ)_/¯

Stéphane Landelle

GatlingCorp CEO

slan...@gatling.io

--

[Siegfried Goeschl]

Hi Stephane,

two question 

* is your test server mapped to the same IP address as shown below?

* did you check the connections immediately

> nslookup computer-database.gatling.io

Server:     195.34.133.21

Address:    195.34.133.21#53

[Stéphane LANDELLE]

* is your test server mapped to the same IP address as shown below?

MacBook-Pro-de-slandelle:async-http-client-project slandelle$ nslookup computer-database.gatling.io

Server: 8.8.4.4

Address: 8.8.4.4#53

Non-authoritative answer:

Name: computer-database.gatling.io

Address: 35.158.229.206

 

* did you check the connections immediately

Yes, of course