HTTP+SSL performance issue

[Nicolas Pouvesle]

Hello,

When I enable SSL on vertx 3.0 there is a big drops in performance, more than just the overhead due to SSL itself. I'm using the same request against http and https:

wrk -c 200 -d 10s https://localhost:8443/string --timeout 100000s -t 8

This is how SSL is enabled:

HttpServerOptions options = new HttpServerOptions().setSsl(true).setPemKeyCertOptions(

                new PemKeyCertOptions().

                        setKeyPath("key8.pem").

                        setCertPath("cert.pem")

        );

        vertx.createHttpServer(options).requestHandler(router::accept).listen(8443);

And the results for HTTP first:

Running 10s test @ http://localhost:8080/string

  8 threads and 200 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev

    Latency     3.48ms  521.40us  30.89ms   95.22%

    Req/Sec     7.21k   530.04     8.28k    89.47%

  579148 requests in 10.10s, 22.65MB read

Requests/sec:  57323.97

Transfer/sec:      2.24MB

And HTTPS:

Running 10s test @ https://localhost:8443/string

  8 threads and 200 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev

    Latency    62.24ms  132.56ms   1.04s    88.12%

    Req/Sec     1.72k     0.85k    2.39k    79.91%

  79213 requests in 10.06s, 3.10MB read

  Socket errors: connect 82, read 0, write 0, timeout 0

Requests/sec:   7874.11

Transfer/sec:    315.27KB

Am I missing something in server config?

Thanks,

Nicolas

[Tim Fox]

On 04/07/15 16:01, Nicolas Pouvesle wrote:

Hello,

When I enable SSL on vertx 3.0 there is a big drops in performance, more than just the overhead due to SSL itself.

Can you substantiate this?

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nicolas Pouvesle]

I've been doing the exact same benchmark with multiple java frameworks sending the same request and receiving the same response. They are all between 35k and 20k req/sec on http and 20k to 10k req/sec on https using the same test machine.

Vertx 3.0.0 is doing an astonishing 58k req/sec on http but only 8k req/sec when ssl is enabled.

I've performed the same test using Netty 4.0.28 (same as vertx) using the demo http server and I have the following results:

HTTP:

Running 10s test @ http://localhost:8080/string

  8 threads and 200 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev

    Latency     3.82ms  591.76us  21.12ms   84.92%

    Req/Sec     6.57k   683.54     7.24k    79.68%

  527600 requests in 10.10s, 50.32MB read

  Socket errors: connect 0, read 48, write 0, timeout 0

Requests/sec:  52216.67

Transfer/sec:      4.98MB

HTTPS:

Running 10s test @ https://localhost:8443/string

  8 threads and 200 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev

    Latency     6.85ms    2.06ms  70.55ms   88.94%

    Req/Sec     3.66k   430.61     4.32k    77.55%

  289888 requests in 10.03s, 27.65MB read

Requests/sec:  28913.63

Transfer/sec:      2.76MB

That's why I suspect Vertx results should be much higher.

Nicolas

[Nicolas Pouvesle]

So I tracked down the issue and finally figured out what was happening. The createContex method in SSLHelper.java is using the JVM SSLEngine. Netty can switch between the JVM SSLEngine and the OpenSSL SSLEngine.

I replace the code to use Netty's fork and a predefined cert (would need to reimplement all the option code too) and now the performance are in  line with what I would expect:

Running 10s test @ https://localhost:8443/string

  8 threads and 200 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev

    Latency     4.31ms    1.19ms  28.93ms   78.84%

    Req/Sec     5.80k     1.31k   24.74k    93.27%

  462505 requests in 10.10s, 18.08MB read

  Socket errors: connect 34, read 0, write 0, timeout 0

Requests/sec:  45791.10

Transfer/sec:      1.79MB

45k req/sec instead of 8k. The JVM implementation really isn't that great :-)

[Tim Fox]

Can you show your code that shows the OpenSSL implementation being used? I'm having trouble googling docs on this from Nety (getting just 404s)

Yep, it's known the JDK impl is poor (it also has big ugly synchronized blocks which is not good when there are many concurrent connections). If we can use the OpenSSL version that would be great.

[Nicolas Pouvesle]

I attached the diff to use Netty's SslContext instead. It's just a test code as it generates the certificate on the fly (from Netty's example).

I'm still trying to figure out how to replace the current KeyStoreHelper/createContext code with TrustManagerFactory/KeyManagerFactory to use SslContextBuilder directly.

[Manish Kumar]

Hey Guys,

Is there is way in Vertx to tell which SSL Engine to use ? 

Thanks,

Manish

[kim young ill]

do you mean "other" network-javaframeworks (eg Undertow, jetty) also use openssl as backend ? if it's not the case then problem is somewhere in vertx

[kim young ill]

if i read the result correctly  plain netty with jdk's ssl -engine: 28k req/sec, vertx 3: 8k req/sec, so the overhead is pretty high

[Manish Kumar]

I think I don't low level details. I was thinking that Vertx determines which SSL engine to use; but from your comment it looks like that Vertx is using Netty, which in turn uses JDK SSL Engine.

Do you mean 3.8 req/sec with Vertx ?

[Tim Fox]

Thanks,

Can you open an issue for this and attach the diff so it doesn't get forgetten?

[Klausen Schaefersinho]

Hi,

can I expect this fix to be included in 3.1 release? Or is there somehow a workaround?

Cheers,

Klaus

[Tim Fox]

I was waiting for an issue to be filed with the relevant information, then someone can investigate.

Visit this group at http://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/5401c48a-fc36-4633-8b57-a7360be6ccec%40googlegroups.com.

[Tim Fox]

Folks,

We're getting close to the cut-off for Vert.x 3.1. I'm still waiting for an issue that captures all the information so we can investigate and add a new feature (if necessary). Right now the information on this thread is not clear to me.

If we don't get that in the next day or so, it will probably be pushed to 3.2. Thanks.

[Tim Fox]

https://github.com/vert-x3/issues/issues/62

This is not going to make it for 3.1 but I opened an issue anyway.

Please feel free to attach stuff or make comments on the issue.

Pull requests are welcome! :)

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+unsubscribe@googlegroups.com.