Spring Boot VS Vertx - Spring Boot wins

10,549 views
Skip to first unread message

Mayur Thakare

unread,
Oct 30, 2017, 2:25:28 AM10/30/17
to vert.x
I compared Spring MVC application and Vertx application. Both application exposed an API that would fetch all records from PostgreSQL DB (Select *). Record set was kept heavy, (20k records returned per API call) 
In Spring MVC application, an API would make a DB connection, fetch records, sends it via response.
In Vertx. I created 2 verticles, HTTP Vertical that handles http requests and responses, and a DB Verticle that talks with Database via vertx PostGres client. 
Verticles communicate to each other via event bus. (This is the typical flow demonstrated on vertx.io sample application in there getting started guide)

I used Apache Bench as performance testing tool. Bombarded 2000 requests with concurrency set to 50.
Initially with no memory restriction, vertx was marginally faster. (Spring :  9.11 requests handled per second, Vertx (5 instances of both verticals each) : 11.06 requests per second)

But when memory restrictions were applied, vertx performance and relibility went down.
After many event queue timeouts and testing with lot of different values of verticle instances following were the results

Memory restriction 1024m
Spring MVC : 8.41 request per seconds
Vertx App (2 instances of each verticle) : 5.6 request per second

This inreased marginally when instance was reduced to 1 for each verticle
Vertx App (1 instance of each verticle) : 5.7 request per second
but still slower than spring boot app.

When memory was restricted even further to 512m, it was disaster for vertx 
After lot of tries (each failing with ThreadBlock, OutOfMemory errors), I could get one complete result for vertx app
However Spring MVC app never complained.
Following are the results

Spring MVC : 7.8 request per seconds
Vertx App (1 instance each of HTTP and DB verticle) : 2.22 requests per second. With 33 out of 2000 requests failing.


My observation is vertx application is not suitable for application that involve DB CRUD usecases.

Mayur Thakare

unread,
Oct 30, 2017, 2:34:24 AM10/30/17
to vert.x
Let me add, for a non DB API. Vertx was 3 times faster
For an api that would just return json data like application version info which does not make any DB calls following were the results

Spring Boot : 326.41 requests per second.
Vertx : 1295.22 requests per second.

Jeroen Knoef

unread,
Oct 30, 2017, 4:54:56 AM10/30/17
to vert.x
Thanks for this!

I am new to vertx, so I can't say much about these results, except that my conclusion would be that vertx is quite suitable for CRUD, if you allocate enough memory. I suspect 1MB or less for an application is on the low side these days, and never a problem. By the way, 9 req vs 11 req is not marginally faster, it's some 20% faster.

Your results may depend on the way the 2 verticles communicate. Can you show us the code?

Jez P

unread,
Oct 30, 2017, 5:13:43 AM10/30/17
to vert.x
1G not 1M. Also, 11 req/second implies a wider problem. Db access across a decent network with a connection pool of 20 should comfortably outdo that in either technology (Spring Boot/vert.x). Out of interest, do you know your connection pool sizes? 

As well as the code, could you provide some idea of what was deployed where (i.e. where network calls were)? Both sets of benchmarks look very poor to me.

Finally, if you're generally retrieving thousands of rows from a DB. I'd be slightly wary. _Most_ CRUD applications deal with much smaller result sets.

Mayur Thakare

unread,
Oct 30, 2017, 5:14:16 AM10/30/17
to ve...@googlegroups.com
Thanks, Code contains some propriety stuff. Ill create a more generic code and upload on git.

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/vb0logBJbpo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
 To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/b2e90975-7940-4b7a-9630-8849331f6532%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
mAyur,
"out of beta"

Mayur Thakare

unread,
Oct 30, 2017, 5:20:51 AM10/30/17
to ve...@googlegroups.com
Also, memory restriction was 1 GB and not 1 MB, but I guess it was typo and you meant 1 GB. 
Reason of testing it with restricted memory size is because this code will eventually running in a container which will have restricted resources. Load balancer will take care of throughput by spawning containers as required. This takes care of throughput advantage that vertx provides.
I makes sense for me to go for vertx, if it able to perform with restricted resources.

~mayur.

Jeroen Knoef

unread,
Oct 30, 2017, 5:23:51 AM10/30/17
to vert.x
No, sorry about that, that mistake was mine.

Julien Viet

unread,
Oct 30, 2017, 6:36:59 AM10/30/17
to vert.x
Hi,

your feedback is interesting but as any benchmark you should provide a way to reproduce it for others, otherwise it's just useless.

Also the title of your message is very provocative, i.e you are making a free statement that claims that Spring Boot wins, and that statements appears to be universal and apply to all kind of applications and workload.

You should rather say, "I've done this benchmark in these conditions, my results show that Spring Boot has a better performance than Vert.x (elaborating because performance can mean a lot of things) can someone look at it and confirm it or instead prove me wrong" ?

In these conditions if you provide your benchmark, I would be happy to run it in our "Performance Lab" and give my versions of your story and perhaps help you find out what could be wrong the benchmark.

HTH

Julien



--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Mayur Thakare

unread,
Oct 30, 2017, 8:03:02 AM10/30/17
to vert.x
My apologies for selecting provocative title for this post.

Intention was not to conclude on my findings but to understand correct use cases where vertx can be useful and to understand any flaws in my process through which I arrived at result from experts. Wish I could change the topic.

I have uploaded my vertx code  at https://github.com/mthakare/vertx with pre requisites listed. Let me know if instructions not complete or there is any issues in accessing code.

I have changed the code so that any copyright of my org is not violated. However basic pattern remains unchanged.

Postgres 9.6 with which I tested was on a Windows workstation local to the vertx application.

Benchmark tool I used is ApacheBench, Version 2.3 <$Revision: 1807734 $>

Let me know I anything more is needed. 

I will carryout tests with this application and post my results.

Thanks.
~mayur.

Julien Viet

unread,
Oct 30, 2017, 8:29:32 AM10/30/17
to vert.x
Hi,

thanks, that sounds correct.

I won't have personally the time to investigate soon (I'm preparing for KotlinConf this week), but I will give a try as soon as I can.

That does not preclude anyone in this list to have a look of course :-)

Julien

Paulo Lopes

unread,
Oct 30, 2017, 8:44:19 AM10/30/17
to vert.x
For proper reference you should also provide the spring boot source code as you cannot verify that the tests are the same.

Mayur Thakare

unread,
Oct 30, 2017, 9:02:23 AM10/30/17
to vert.x
Give me some time to upload spring boot application. It was written for testing purpose and lot many things are hard coded.

It essentially does following -
1.  Exposes one GET API, that API accesses DB via jdbc driver and returns result.
2.  Points to the same database that vertx app is accessing.

~mayur.

Mayur Thakare

unread,
Oct 31, 2017, 8:42:53 AM10/31/17
to vert.x
Please checkout 


It contains details of test code made available via github and test numbers I again carried out in my environment.


On Monday, October 30, 2017 at 6:14:19 PM UTC+5:30, Paulo Lopes wrote:

Jez P

unread,
Oct 31, 2017, 8:57:35 AM10/31/17
to vert.x
Relative to each other where were the following running?:-

Application
Database
Caller of application (i.e. the test)

Mayur Thakare

unread,
Oct 31, 2017, 9:03:53 AM10/31/17
to vert.x
Both applications and database were on same machine. Vertx API will be hosted on port 1729, spring APIs on 8080.

For test numbers uploaded on blog, I ran Apache bench on same machine where applications were running.

I did't ran both applications concurrently. I first gathered numbers for spring app and then vertx app.

Jez P

unread,
Oct 31, 2017, 9:42:38 AM10/31/17
to vert.x
With respect, it's not a very realistic benchmark in that case. And all the numbers look far too low given the lack of network to the DB. 

DB queries running on the same box (which may well involve significant CPU usage) may well affect the two applications very differently (and affect GC as well). 

A more realistic approach would be the following:

Test rig on one box, good network (but not as good as local host) to a box running test application, and similar network to the db (which resides on a third box). 

Your test will also affect the behaviour of both applications and DB similarly. Vert.x is more likely to be using the CPU heavily than the other two, therefore having to share a CPU with the other two will definitely hamper processing, more so as you deploy more instances (and try and use all the cores). Spring Boot will distribute its load between the cores it can reach. So yes, your test probably will favour Spring Boot under load, by hamstringing the Vert.x application.

Jez P

unread,
Oct 31, 2017, 9:55:55 AM10/31/17
to vert.x
You might still find (I've not gone through the code in great detail) that using a more appropriate architecture still gives better results with Spring Boot. However right now, the output of your benchmarks are basically two meaningless numbers which you're trying to compare. CPU load (via DB/testing tool) will affect both frameworks differently and therefore you cannot draw much of a comparison when your test itself is responsible for imposing that CPU load. Assuming that the effect of the CPU load you impose will be identical for the two frameworks is almost certainly the wrong thing to do. 

Mayur Thakare

unread,
Oct 31, 2017, 10:15:41 AM10/31/17
to vert.x
Let me setup test environment such way that Test Machine (Client), Machine on which applications are loaded(server) and DB are three different machine in same LAN.

Jez P

unread,
Oct 31, 2017, 10:33:37 AM10/31/17
to vert.x
I think those results will be enlightening. My prediction is that actually performance will end up being relatively close, because the database (and connection pools) may well prove to be the limiting factor.

However, it will be interesting if I'm proved wrong and you see identical results (but I'd be very surprised in the latter case). 

Julien Viet

unread,
Nov 1, 2017, 12:14:51 AM11/1/17
to vert.x
in such case, indeed the connection to the DB is usually the limiting factor and you can consider the DB as a queue (whose concurrency is determined by the connection pool size, the connection protocol and with jdbc the worker pool size)

Vert.x can very shine however, when you have HTTP requests from different connections doing a single statement (like a simple query) and with a non blocking database driver able to pipeline (like https://github.com/vietj/vertx-pg-client), you can get much better numbers


Mayur Thakare

unread,
Nov 1, 2017, 1:55:15 AM11/1/17
to ve...@googlegroups.com
Thanks for the input, I was using vertx-mysql-postgresql-client 3.4.2 driver. I assumed it to be non blocking.
Also, I will run performance test with the setup recommended by Jex P.

I am vertx champion in my org, and trying to sell vertx to my team who is inclined using spring boot. This exercise and its outcome is important in concluding whether vertx fits our usecase.
~Mayur.

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/vb0logBJbpo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/F287A042-748C-458D-830C-121B26BBB448%40julienviet.com.

For more options, visit https://groups.google.com/d/optout.



--
mAyur,
"out of beta"

Paulo Lopes

unread,
Nov 1, 2017, 5:26:38 AM11/1/17
to vert.x
Hi,

Just a tip!

You're trying to compare Vert.x to Spring boot, why not use an existing benchmark that has been peer reviewed instead of using your own?

For example look at:

https://github.com/TechEmpower/FrameworkBenchmarks

Specifically at:

https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Java/spring

vs

https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Java/vertx

There are good instructions on how to run the benchmark (in dev mode using vagrant) or on real production using 3 machines (wrk + db + app).

I think you will some results that are more comparable as the benchmark requirements are clear and have been reviewed by many people.

The final note is that, do remember that the benchmark does not really reflect a real application, as usually you would not just perform a query but probably do some authentication/authorization, data transformation, etc...

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/vb0logBJbpo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

Julien Viet

unread,
Nov 1, 2017, 6:56:47 AM11/1/17
to vert.x
Hi,

vertx-mysql-postgres-client is non blocking too (but it cannot pipeline as far as I know)

next week I will try give a try to your proof of concept to see what can be fairly compared.

this week I'm too busy travelling for a conf.

Julien

Alex Z.

unread,
Nov 4, 2017, 11:57:25 PM11/4/17
to vert.x
Hi Mayur,

I am a little bit confused by your description when you say you are comparing "Spring Boot VS Vert.X". 

You are not actually comparing "just" Spring Boot - what web container are you running under the hood? 
Would you be able to disclose - is your SpringBoot app running on Tomcat? Jetty? Undertow? Which one of them is being used and what version? 

I appreciate your clarifications,

Regards,
Alex

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/vb0logBJbpo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

Léo Silva

unread,
Nov 5, 2017, 9:28:09 AM11/5/17
to vert.x
I will check out the code used, but anyway it sounds strange.
I have benchmarking of a real-api using vert.x for payment transaction execution, acessing DynamoDB, Mysql and SQS, with 100 req/sec responding around 56ms with ONE api-instance running on ec2-micro.
We have reached in the last result around 27k req/sec.
The case can be checked here (in portuguese, sorry): https://aws.amazon.com/pt/blogs/aws-brasil/846-2/

For me it is understandable that using vert.x, at some point, you just cannot scale just with adding more memory, since it does not rely on creating infinite threads, but using CPU on a more smart way, for example, in this 100 req/sec we reached around 85% of CPU usage.

One important point is that, our benchmarking was limited to 27k req/sec because of relational database bottleneck, our api's were completely fine (cpu and memory usage)
Since we had already a good result, we did not went further with infrastructure changes (eg database, cache, etc)

[]'s

Mayur Thakare

unread,
Nov 5, 2017, 9:30:07 AM11/5/17
to ve...@googlegroups.com
Its embedded tomcat 8.5.16 that comes default with spring framework.



To unsubscribe from this group and all its topics, send an email to vertx+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/c6339a3c-1821-493b-9f2b-e308336a0aa0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Mayur Thakare

unread,
Nov 5, 2017, 9:38:14 AM11/5/17
to ve...@googlegroups.com
Hi Leo,

I am assuming you carried out test on MySQL cluster on AWS. I have not done that yet. I was running everything, my test client, db and apps on same machine. And since test environment was same for spring and vertx apps, I am assuming that I am making a correct relative comparison. As far as absolute numbers are concerned, a MySQL cluster on AWS will give lot better results.
 
And as it was realized in the discussion on this thread, memory was a probable bottleneck for vertx application. My next test would have three different machines for each one (test client, DB and applications).

~mayur.

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/vb0logBJbpo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/24b66596-3d28-40d5-99ed-e954a4fb10fe%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Laputa

unread,
Nov 10, 2017, 7:01:51 AM11/10/17
to vert.x
It's funny because we only justify vert-x performance by non blocking thing. The nature of json itself is not fit in java. Vertx keep returning json as row witch is contains unnecessary column information as key in json. I don't say that vertx approach is wrong but in this test schenario that's what happened. I think it's just wrong to benchmark any framework or tool by implementing it in really wrong way.

Ladislav Jech

unread,
Jan 25, 2019, 10:33:30 PM1/25/19
to vert.x
I recommend to look at light4j framework if you seek performance...

Mukesh kumar

unread,
Mar 25, 2019, 12:47:24 PM3/25/19
to vert.x
I saw an article on Reactive Programming Comparison between Spring 5 and Vert.x and the result is totally different from your view, And I am totally agreed with this article too.
Reply all
Reply to author
Forward
0 new messages