Quarkus + Vertx not performing as well as Vertx alone

[lurumbo]

Hi everybody!

I have 2 projects that retrieve data from the same postgres DB, running in a docker container, obtained from Docker Hub

1 - Full Vertx

https://github.com/lurumbo/vertx-demo

2 - Quarkus + Vertx (it is actually the Vertx project, plus some adjustments to make it work with Quarkus)

https://github.com/lurumbo/quarkus-reactive-pg-vertx-demo

I performed some stress tests on them with Jmeter and I can't wrap my head around the results

Vertx Report

https://github.com/lurumbo/vertx-demo/blob/master/jmeter/vertx.png

Quarkus Report

https://github.com/lurumbo/quarkus-reactive-pg-vertx-demo/blob/master/jmeter/quarkus-reactive-pg-vertx.png

The tests were executed with these params:

- Number of Threads (users): 1000

- Ramp-Up Period (in seconds): 0

- Loop Count: 10

What troubles me the most is the diff in the throughput values.

- Vertx: 1776.5/sec

- Quarkus: 1032.0/sec

Can you please help me understand the results?

Shouldn't the metrics be roughly the same?

Why am I getting such a diff in throughput?

Some notes:

- I tested it using the runner.jar

- The tests were made in JVM mode

Technical info:

- Processor Intel® Core™ i7-8565U CPU @ 1.80GHz × 8

- Memory 15,5 GiB

Thank you very much!

[Loïc MATHIEU]

Hello,

Using Quarkus you will have a more complex stack than Vert.x, and I saw that you didn't use any rampup period and you perform very little iteration for your test scenario.

Can you re-run the test with a rampup period (you always need to have a rampup period for benchmark scenario, at least for the JIT and the different caches to kick off), and more iterations ?

Maybe this will give less differences between raw Vert.x usage and Quarkus reactive routes.

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/a7f2cad0-52e0-4766-a91b-ad7445a4dd36n%40googlegroups.com.

[William Burke]

 There's no magic that Quarkus is doing with Vertx, although I think that Quarkus might introduce its own worker pool implementation as I saw some email threads about that somewhere.  

Is this all JVM based?  Or is the Quarkus app built with Graal?  You may require some Graal tuning if so.  I suggest doing a Quarkus JVM run too to compare to native.  Finally, make sure you have the same number of DB connections and other config set up for each project   I know I've made that simple error in the past!

@Loïc MATHIEU Quarkus is not a more complex stack than Vert.x.  There's no magic/additions other than bootstrap and maybe the worker pool I mentioned.  More often than not, if you're porting an app to quarkus which has specific extensions, those extensions improve performance and reduce overhead.  The OP is right to be concerned.  So should we be!

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJLxjVEvRDzaHgbDGATS9EOgGWOoHX-eL_7jvSy%2BNjUPgzir9A%40mail.gmail.com.

--

Bill Burke

Red Hat

[Loïc MATHIEU]

@luru...@gmail.com are you running your test from a different machine and with the headless mode of JMeter ?

JMeter opens one thread by user, you opened 1000 threads so better check that it's not the machine running JMeter that is the bottleneck (check cpu usage, run queue and context switches at least ...).

If you're curious you can also create profiles for both runs to analyse what's going on, if you do so, please post them to the mailing list, I'm sure it will be of interest to some people here ;). 

We have a page explaining how to create such performance profile : https://github.com/quarkusio/quarkus/blob/master/TROUBLESHOOTING.md

@William Burke I know that Quarkus is not very complex but it still have more bootstrapping code (and as the test has no rampup and runs for a few seconds it can mix up with the throughput) and it still have a slightly more complex HTTP stack (I don't know the details but reactive routes are translated to Vert.x handler so there is an indirection layer).

[Thomas SEGISMONT]

Hi,

As I told you on SO where this question was originally posted, I only had a brief look at the code but I see some differences:

- the Quarkus impl uses a mix of Reactive Web routes with DB verticle, whereas the Vert.x impl uses two verticles

- the Vert.x impl generates the reply by concatenating strings, where as the Quarkus impl creates a JsonArray that needs to be encoded with Jackson

So it doesn't seem like an apples to apples comparison.

Le ven. 17 juil. 2020 à 17:25, lurumbo <luru...@gmail.com> a écrit :

--

[Paul Carter-Brown]

Running 1000 threads and 10 iterations per thread is fairly meaningless. You can see that the TPS of each is more impacted by the length of the first calls (up to 2 seconds) than actual steady state throughput. You are measuring the speed of the container to start up a large thread pool ant not really its throughput.

Change to 20 threads and 100000 iterations per thread. Then run 2 tests and use the value of the second one. Send those results and they will be more meaningful.

Paul

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAJLxjVHtX0O9C_LFtkhPFh7R920CQ81m-q5zP1hvTuvy0JtR0g%40mail.gmail.com.

[John O'Hara]

There was also an issue opened for this question:  https://github.com/quarkusio/quarkus/issues/10785  There are a number of issues with this test, my response to the results is on the issue.

If you have future issues like this, please can you open an issue, and then link to that issue in other channels, rather than asking the same question via different channels. This would mean that all the responses should be in one location and anybody else trying to find the response will see all the comments

Thanks. 

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAO8YPTR5RE2odeCBEDu2VQGBYR4P_EeJtYKb%3DY44wk-kFgFRJQ%40mail.gmail.com.

--

John O'Hara

Principal Software Engineer, Middleware (Performance)

Red Hat