Question about Concurrent Requests

[bf...@bloomhealthco.com]

When performing a few get requests back to back for a service I am frequently seeing the following error in the client:

java.net.SocketTimeoutException: Read timed out

All the other requests seem okay besides the second request.

Doing some testing with the ab tool against the dropwizard example I saw similar results show below (however, my endpoint shows error with a concurrency level as little as 3 with ab).

Is there a recommended configuration to avoid these types of errors?

Thank you.

ab -c 25 -n 100 -r http://localhost:8080/hello-world

This is ApacheBench, Version 2.3 <$Revision: 655654 $>

Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/

Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient).....done

Server Software:        

Server Hostname:        localhost

Server Port:            8080

Document Path:          /hello-world

Document Length:        39 bytes

Concurrency Level:      25

Time taken for tests:   0.030 seconds

Complete requests:      100

Failed requests:        6

   (Connect: 0, Receive: 6, Length: 0, Exceptions: 0)

Write errors:           0

Total transferred:      17271 bytes

HTML transferred:       3939 bytes

Requests per second:    3305.68 [#/sec] (mean)

Time per request:       7.563 [ms] (mean)

Time per request:       0.303 [ms] (mean, across all concurrent requests)

Transfer rate:          557.54 [Kbytes/sec] received

Connection Times (ms)

              min  mean[+/-sd] median   max

Connect:        1    6   2.6      6      10

Processing:     0    1   1.6      0       7

Waiting:        0    1   1.5      0       7

Total:          2    6   1.8      6      10

Percentage of the requests served within a certain time (ms)

  50%      6

  66%      7

  75%      7

  80%      7

  90%      9

  95%     10

  98%     10

  99%     10

 100%     10 (longest request)

Full strack trace:

java.net.SocketTimeoutException: Read timed out

! at java.net.SocketInputStream.socketRead0(Native Method)

! at java.net.SocketInputStream.read(SocketInputStream.java:129)

! at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)

! at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)

! at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)

! at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)

! at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62)

! at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)

! at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)

[Viral Bajaria]

What's does your configuration yaml look like ? Can you share the "http" section of it ? Or are you just going with the defaults ? I have benchmarked my dropwizard service to approx 7000 rps on a m1.large on EC2. I wasn't running the dropwizard example code but a more real workload where there was some db lookup (with caching) and some extra business logic but I would expect the dropwizard-example to perform much better since it has far less blocking code.

[Chad Small]

@bf (co-worker of mine) used the example.yml configuration from the dropwizard-example project as is.  If anyone has ideas on http configuration setting gotchas, they would be appreciated.

thanks,

chad.

[Viral Bajaria]

You can play around with a lot of settings. Also the example yaml is documented pretty well as to what each setting does, so I would advise playing around with a few of those settings one by one.

A few things that I have changed:

connectorType: nonblocking

acceptorThreads: 16 (dual quad-core with hyperthreading)

minThreads: 100

maxThreads: 1000

Instead of changing all settings in one swoop, play around with them one by one and see how it impacts throughput.

-Viral

[Chad Small]

Looks like this was our bad.  A default setting on the socket timeout (SO_TIMEOUT) of HttpClient wasn't being defaulted correctly.  Got that set correctly and life is good.  And it turns out Apache Bench on OS X Lion has some badness going on with it so it just happened to give us the same error.  (http://simon.heimlicher.com/articles/2012/07/08/fix-apache-bench-ab-on-os-x-lion).