Performance issues upgrading from Vert.x 3.9.8 to 4.x
jeffrey meredith
We recently upgraded from Vert.x 3.9.8 to 4.1.4 (and then later 4.2.x and 4.3.1). In subsequent load testing we noticed some issues. Execution times for a simple workload were higher, old gen memory was growing much more quickly and as a result operations per second were down. The percentage changes in timings was on the order of a 38% slowdown (~18msec to ~25msec) while old gen memory grew at twice the previous rate from roughly 200Mb per hour to 400Mb.
We started looking at heap dumps. We found ~1.1Gb of uncollected ContextScheduler TimedAction objects. These are created and run when RxJava executes its observeOn operator which our load test does frequently. It seemed that these objects were now surviving a GC and moving to old gen at lot more often.
Looking at why that might be happening we instrumented our own copy of TimedAction from RxJava3 (see Gist) to measure latency of these tasks. We found that it was twice as long as the latency in Vert.x 3.9.8 for our test. Under load we saw latencies on the order of 400msec vs. 200 previously.
Finally, we worked to simplify the test and remove RxJava (and most of our code base). The Gist has 3 additional classes that comprise the test: a REST service and a simple vertical that consumes a message then executes runOnContext 150 times for each one. Measuring the latency of the tasks submitted to runOnContext showed similar performance differences between the releases under load. The higher latencies in Vert.x 4 showed up around 16 concurrent users and got quite a bit larger at 32 users.
We can work on a version of the test that removes the remaining dependencies on our code base, but I wanted to first find out if others have seen something like this between the major versions of Vert.x?
Julien Viet
if you can provide two benchmarks with 3.9 and 4.1 we can have a look of course.
Julien
> You received this message because you are subscribed to the Google Groups "vert.x" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/f36d00a1-70fc-46f3-8674-08d25aeffc8fn%40googlegroups.com.