Wiremock for load testing

[sagar dutta]

Hi,

I'm using wiremock to mock dependencies in my load test.

It works fine for sometime but under heavy load wiremock starts throwing exceptions

Remotely closed, Too many files open are some of the exceptions I have seen under heavy load.

I have started the Wiremock server programmatically with 500 container threads. I checked ulimit and it seems not bound by the default.

I was wondering if anyone here tried wiremock for load testing and did any sort of tuning  ?

Or any ideas around how to tune jetty in wiremock to take load ?

Please let me know if any info is needed.

I am working on a docker image which will help reproduce this issue. I will upload it soon.

Thanks for your time.

regards,

Sagar

[sagar dutta]

I have seen a previous post which posed the same question back in Jan, 16.

As soon as I reach 2000 rpm, I see errors from wiremock. (connection timed out exceptions )

I would like to figure out how to get around this limitation.

[Tom Akehurst]

Could you post some details?

Stacktraces?

Stubs you're testing against?

Hardware you're running on?

[sagar dutta]

Hardware : 1 CPU, 3 GB memory

Extensions: VelocityTransformer

Code and instructions to run below

https://github.com/sagardutta/wiremock-load-test

 

I'm using docker to run wiremock server so that it would take care of memory and cpu limits.

When I use apache benchmark to run load tests wiremock server goes down with "apr_socket_connect(): Connection reset by peer (54)" 

Please let me know if you need anything else from me.

[Tom Akehurst]

Without more information I can't tell what's happening exactly. 

However, if you're doing template rendering on a single virtualised CPU, I suspect the throughput you're getting isn't that unreasonable. Depends hugely on the complexity of your templates.

I'd guess that the socket resets are happening because Jetty's bounded request queue is filling up, which you'd expect if you're filling it faster than it can handle. Jetty's behaviour in this instance is to reject the request.

I can't really provide any advice in this instance beyond running a profiler against your WireMock server or throwing more hardware at the problem.

[sagar dutta]

Hi Tom,

Isn't Jetty's request queue unbounded by default ?

Does wiremock use a bounded jetty queue by any chance ?

Also my entire wiremock server code is at the below github repo.

Instructions to run it are in the README.md

Just wanted to check if you got a chance to look at it.

When I run the apache benchmark tests and monitor my container

1. Memory seems to be fine (it reaches 1.8 GB out of 3 GB )
2. No of file descriptors are also within the limit (it never goes beyond 1000 and my limit is way beyond it, I use ' lsof -u root | wc -l ' to monitor the no of open files  )
3. CPU usage also seems under control, during minor GC it spikes up but comes back immediately. 
4. If templating is heavy I would still expect timeouts to happen instead of 'connection reset by peer'. Do you agree?

I will also run a profiler and get back to you with more info.

Please comment on the above.

[Tom Akehurst]

It defaults to being bounded: https://github.com/eclipse/jetty.project/blob/jetty-9.3.x/jetty-util/src/main/java/org/eclipse/jetty/util/thread/QueuedThreadPool.java#L103

I'm going to struggle to find time to check out your project in the near future I'm afraid.

Profiling is definitely the way to go. It seems odd that you're not maxing the CPU. Are you reading large body files from a slow file volume maybe?