Comparing vertx with other web framework

[Buddha Shrestha]

Hi,

I recently created a web server using Vert.x. I did a load test for that server and found the results as shown in the charts.

My server is used to receive GET request and then hit another API to get some data and return back the response to the request sender. 

I want my application to handle as many requests as possible and since vertx supports many concurrent requests, I chose Vert.x.

I then ran a comparison against Grails framework, built a Grails server to do the same task and found the response attached in this post. Vertx was not able to handle as many requests as Grails did (10k concurrent requests were hit)

It turns out that Grails supported 10k worker threads. 

Since my Vert.x application is light weight my memory and CPU are relatively low so I would always prefer Vert.x. But I'd like to still handle as many requests as possible.

 

So, what I would like to know is, what would I need to do in order to achieve 10k concurrent requests to be processed using Vert.x?

Thanks

[Thomas SEGISMONT]

Hi,

If your code is simple enough then it would be worth sharing otherwise we can only do some wild guessing (e.g. event loop blocked, not using http client persistent connections)

Regards,

Thomas

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/2d1a5452-e2f5-4ecb-9a17-d010d949306c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julien Viet]

Hi,

a few questions :

- have you run this on localhost ?

- it seems it involves a backend, is the backend on the same machine as well ?

- how many cores have you machine ?

- I can see two charts, what do they actually measure ?

- can you share your benchmark ?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/2d1a5452-e2f5-4ecb-9a17-d010d949306c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<Vetx_CPU.png><Grails_CPU.png>

[Jez P]

+1

It doesn't look like the vert.x implementation is making much use of your CPU, are you doing a lot of executeBlocking/worker verticle code?

[Buddha Shrestha]

Hi Julien,

1. yes I ran it on local. 

2.the backend was on another local server.

3.my machine had 4 cores.

4.The blue chart measured CPU usage and the red measured memory usage.

5.My benchmark is to see how many concurrent users could a Vert.x application process and compare it with other frameworks.

Please let me know, if you need any other information

[Buddha Shrestha]

Hi Jez,

I'm just working with with eventloops and callbacks(for HTTP request), The backend work is done by another local server, where I hit 'GET' request -- which is triggered by user's request to my app.

[Julien Viet]

On Jun 23, 2017, at 6:46 PM, Buddha Shrestha <shrest...@gmail.com> wrote:

Hi Julien,

1. yes I ran it on local. 

you should not run a benchmark on localhost

what is your actual client doing the benchmark ?

2.the backend was on another local server.

if it has a backend, the backend should not be on the same host

what is your backend ? in particular what is interesting is the typical response time of your backend

3.my machine had 4 cores.

have you deployed multiple instance of your HttpServer verticle ? you may be using only one core on the 4 otherwise

4.The blue chart measured CPU usage and the red measured memory usage.

5.My benchmark is to see how many concurrent users could a Vert.x application process and compare it with other frameworks.

you need to share this benchmark, so we can see if you have done anything wrong with your benchmark

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/5f53c768-9525-427e-af83-1fcbe74cd6a4%40googlegroups.com.

[Buddha Shrestha]

Hi Thomas,

Here is our code : 

https://github.com/shresthaankit7/ResearchVertx/blob/EventLoopBranch/src/main/java/io/research/vertx/MainVerticle.java

On Friday, June 23, 2017 at 3:01:57 PM UTC+5:45, Thomas Segismont wrote:

Thanks,

Buddha

Hi,

If your code is simple enough then it would be worth sharing otherwise we can only do some wild guessing (e.g. event loop blocked, not using http client persistent connections)

Regards,

Thomas

2017-06-22 20:03 GMT+02:00 Buddha Shrestha <shrest...@gmail.com>:

Hi,

I recently created a web server using Vert.x. I did a load test for that server and found the results as shown in the charts.

My server is used to receive GET request and then hit another API to get some data and return back the response to the request sender. 

I want my application to handle as many requests as possible and since vertx supports many concurrent requests, I chose Vert.x.

I then ran a comparison against Grails framework, built a Grails server to do the same task and found the response attached in this post. Vertx was not able to handle as many requests as Grails did (10k concurrent requests were hit)

It turns out that Grails supported 10k worker threads. 

Since my Vert.x application is light weight my memory and CPU are relatively low so I would always prefer Vert.x. But I'd like to still handle as many requests as possible.

 

So, what I would like to know is, what would I need to do in order to achieve 10k concurrent requests to be processed using Vert.x?

Thanks

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[Thomas SEGISMONT]

I only had a quick look but you'd get much better performance keeping one instance of the HttpClient per verticle. Otherwise you can't benefit from connection pooling and keepalive.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/991da799-2f7a-47d0-8767-9a7b690ce514%40googlegroups.com.

[Jez P]

Agreed. You're also not closing your http clients before they go out of scope. But why not reuse them and have one per verticle instance, as Thomas suggests?

[javadevmtl]

Also you are deploying worker verticle to do vertx jdbc. You don't have to. Vertx jdbc handles the blocking stuff underneath...

[Buddha Shrestha]

Hi Julien,

My backend is also a Grails application hosted in a remote server.

As I'm out from my workplace for couple of days, I can send you my benchmark results on Monday only. Sorry for that.

But, in the mean time, could you please check out our code and let us know if our implementation will withstand more requests than a normal Grails app would do for the same purpose?

Here is our code:

1. Vert.x (which connects with another backend API to return the result to user)

https://github.com/shresthaankit7/ResearchVertx/blob/EventLoopBranch/src/main/java/io/research/vertx/MainVerticle.java

2. Grails (which uses its JDBC and calls stored proc to get the same information and return the result to the user)

https://github.com/shresthaankit7/GrailsAngular/blob/master/grails-app/controllers/grailsangular/PersonController.groovy

3. Grails Backend API used by Vert.x to GET information

https://github.com/shresthaankit7/GrailsAngular/blob/research/grails-app/controllers/grailsangular/PersonController.groovy

I'm trying to compare asynchronous vs synchronous multi-threaded application's performance, like in the following blog :

http://callistaenterprise.se/blogg/teknik/2014/04/22/c10k-developing-non-blocking-rest-services-with-spring-mvc/

Is my Implementation sufficient for that comparison? Also, we only have localhost servers available for running our benchmark test. So, could you please suggest what could we do? 

Please let me know.

Thanks,

Buddha

[Buddha Shrestha]

Hi Thomas,

Utilizing connection pool is a good concept, but from my understanding, in case of many concurrent requests (like 10k), the pool gets fully consumed and all the other requests except for the pool gets stuck. 

That is why I chose to create new client for every request.

Please let me know, if I'm wrong.

thank,

Buddha

[Buddha Shrestha]

Hi javadevmtl,

Sorry for the confusion. I wasn't using worker vertical. It was just included in the code and was not used anywhere. I've removed it now.

Also, I'm not using any jdbc connection with Vertx. I'm using HTTP client to connect to another API which gives me back the requests.

thanks,

Buddha

[Tim Fox]

First rule of benchmarks: Never try and create your own benchmark. Unless you really know what you're doing it's almost certainly going to have multiple flaws and won't be telling you anything much about the system your testing.

[Buddha Shrestha]

Hi Tim,

I'm just wanting to compare if my Vert.x application would handle 10k concurrent requests. If not, then I would have choose other platforms like Grails. 

Since there are benefits of using asynchronous frameworks than thread based frameworks, i'm opting to use Vert.x.

So, I'm just comparing these two frameworks and I'm sharing my implementations to make sure that I'm not doing anything wrong.

Please do check out the above implementations.

Hope to hear from you.

thanks, 

buddha

[Jez P]

Have you tried using a single client per verticle, or did you just hypothesise the problem? I.e. have you compared the two approaches in your current "benchmark"? Also, aren't you possibly really testing how many calls your backend server can handle? Or have you already benchmarked the calls to that?

[Tim Fox]

Hi Buddha,

I quite understanding you have the best intentions, but 99% of the time I have seen these kinds of posts (and they are very common) the issue has turned out to be a poorly conceived benchmark, not an actual issue with the underlying system. Creating good benchmarks is hard!

It takes a lot of time and effort from the team to examine the individual benchmark and figure out where the problem lies.

[Tim Fox]

If you want to see how Grails compares to Vert.x with a simple Http benchmark with pipelining you can look at techempower: https://www.techempower.com/benchmarks/#section=data-r14&hw=ph&test=plaintext

[Buddha Shrestha]

Thanks Tim,