Keep alive not working

[Hemanth Shekar]

Hi team 

The I specifically enabled keep alive in my j meter test when working with a vertx  API  that I built. The API connects to couchbase and sends a response based on the request body . 

Is there any thing other than keep alive header in the request that must be done because the tcp dump is showing that connection still closes after every request . 

In other words is there any thing to do in the API ? 

[Shiva Ramagopal]

You could check the value of the keepalive timeout (getKeepAliveTimeout() on HttpClientOptions), and increase it.

[Blake]

Did you set HttpServerOptions.setTcpKeepAlive(true)? It is false by default. DEFAULT_IDLE_TIMEOUT = 0 for connections, which means "do not time out". You may want to adjust that too.

[Hemanth Shekar]

Thanks Blake . 

I will try this . one more thing i observed is ssl handshake takes lot of time almost 60 - 80 ms 

is there any server option that will reduce this time ? 

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.
To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/283059b7-f2d4-4e1d-bce0-d0a7ebc49a08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Blake]

The Java SSL implementation is pretty slow, apparently. You can setup vertx/netty to work with OpenSSL. https://vertx.io/docs/vertx-core/java/#ssl  -- gotta scroll a bit from that point, but can search for ssl engine or openssl.

You will need to add netty-tcnative to the classpath as well. Keep reading in that section for more info. If you have problems with tcnative, you can find some more info in the google groups searching it.

On Sunday, March 31, 2019 at 2:41:43 AM UTC-5, Hemanth Shekar wrote:

Thanks Blake . 

I will try this . one more thing i observed is ssl handshake takes lot of time almost 60 - 80 ms 

is there any server option that will reduce this time ? 

On Sat, Mar 30, 2019 at 5:51 PM Blake <blakeh...@gmail.com> wrote:

Did you set HttpServerOptions.setTcpKeepAlive(true)? It is false by default. DEFAULT_IDLE_TIMEOUT = 0 for connections, which means "do not time out". You may want to adjust that too.

On Friday, March 29, 2019 at 11:41:06 PM UTC-5, Hemanth Shekar wrote:

Hi team 

The I specifically enabled keep alive in my j meter test when working with a vertx  API  that I built. The API connects to couchbase and sends a response based on the request body . 

Is there any thing other than keep alive header in the request that must be done because the tcp dump is showing that connection still closes after every request . 

In other words is there any thing to do in the API ? 

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ve...@googlegroups.com.

[Hemanth Shekar]

Hi Blake 

If i enable open SSL i run into the below issue in the run time while deployment. 

I am using Vertx 3.5.3 . I cant upgrade due to some of the dependencies .

Any pointers will help

io.vertx.core.VertxException: OpenSSL is not available 

 at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:74)

at io.vertx.core.net.impl.SSLHelper.<init>(SSLHelper.java:163)

at io.vertx.core.http.impl.HttpServerImpl.<init>(HttpServerImpl.java:140)

at io.vertx.core.impl.VertxImpl.createHttpServer(VertxImpl.java:289)

at io.vertx.reactivex.core.Vertx.createHttpServer(Vertx.java:232)

at com.aexp.csrt.qs.dp.MainVerticle.start(MainVerticle.java:65)

at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8(DeploymentManager.java:483)

at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:339)

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

at java.lang.Thread.run(Thread.java:748)

 

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/78c3e1bc-5178-4e36-b4a1-dfcc5a6fdf2b%40googlegroups.com.

[Blake]

First, make sure you've got tcnative on the classpath. After that, you can be pretty sure that it has to do with the tcnative binding. Either openssl is not on your system or there's a problem with tcnative finding it .

You can use tcnative-boringssl which contains Google's openssl fork in it -- probably the easiest way.

Check.

https://netty.io/wiki/forked-tomcat-native.html

[Hemanth Shekar]

Thanks Blake . 

One last question

If I enable keep alive and run load tests i find some of the requests failing . 

I was using the below configuration on the http server 

     //   .setTcpKeepAlive(true)

                   //     .setIdleTimeout(60)

Javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)

at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)

at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)

at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)

at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)

at org.apache.jmeter.protocol.http.sampler.hc.LazyLayeredConnectionSocketFactory.connectSocket(LazyLayeredConnectionSocketFactory.java:92)

at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)

at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$JMeterDefaultHttpClientConnectionOperator.connect(HTTPHC4Impl.java:331)

at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)

at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394)

at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)

at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)

at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)

at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)

at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)

at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:832)

at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:570)

at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:67)

at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1231)

at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1220)

at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:622)

at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:546)

at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:486)

at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:253)

at java.lang.Thread.run(Unknown Source)

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/e9edf7cb-f3c7-4294-b05e-171f7bbbe655%40googlegroups.com.

[Blake]

Is this under really high load? Maybe server is closing b/c it can't keep that many open? Maybe this is happening if maxConcurrentStreams is exceeded? If not under high load then i'm not sure.

I'd setup logging vertx side as well to see what's up. Can call HttpServerOptions#setLogActivity to log the http activity.

[Hemanth Shekar]

I AM running 10 threads . 

How do i set the maxConcurrentStreams  for a http server . 

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/8ef2a358-b07a-45ff-b88e-071f110db35e%40googlegroups.com.

[Blake]

Might not be the issue, but it's under HttpServerOptions.setInitialSettings.

[Hemanth Shekar]

Hi Team , 

I am trying to use open ssl in my vertx based application but  I am running into the following issues . 

Any help would be greatly appreciated as we have prod deployment of the code lined up .

I have added the below dependencies as described in the web site . 

https://mvnrepository.com/artifact/io.netty/netty-tcnative/2.0.0.Final 

 

<dependency>

  <groupId>io.netty</groupId>

  <artifactId>netty-tcnative</artifactId>

  <version>2.0.25.Final</version>

  <classifier>${os.detected.classifier}</classifier>

</dependency>

<dependency>

  <groupId>io.netty</groupId>

  <artifactId>netty-tcnative-boringssl-static</artifactId>

  <version>2.0.25.Final</version>

  <classifier>${os.detected.classifier}</classifier>

</dependency>

Error message

io.vertx.core.VertxException: OpenSSL is not available

        at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:74)

        at io.vertx.core.net.impl.SSLHelper.<init>(SSLHelper.java:163)

        at io.vertx.core.http.impl.HttpServerImpl.<init>(HttpServerImpl.java:140)

        at io.vertx.core.impl.VertxImpl.createHttpServer(VertxImpl.java:289)

        at io.vertx.reactivex.core.Vertx.createHttpServer(Vertx.java:232)

        at com.aexp.csrt.qs.cb.StartMain.startServer(StartMain.java:56)

        at com.aexp.csrt.qs.cb.MainVerticle.start(MainVerticle.java:78)

        at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$8(DeploymentManager.java:483)

        at io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:339)

        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)

        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)

        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)

        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)

        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

        at java.lang.Thread.run(Unknown Source)

Caused by: java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_windows_x86_64, netty_tcnative_x86_64, netty_tcnative]

        at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:93)

        at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:421)

        at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:89)

        at io.vertx.core.net.impl.SSLHelper.resolveEngineOptions(SSLHelper.java:73)

        ... 14 more

        Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_tcnative_windows_x86_64

                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:205)

                at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)

                ... 17 more

        Caused by: java.io.FileNotFoundException: META-INF/native/netty_tcnative_windows_x86_64.dll

                at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)

                ... 18 more

On Sun, Mar 31, 2019 at 3:44 PM Blake <blakeh...@gmail.com> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/e9edf7cb-f3c7-4294-b05e-171f7bbbe655%40googlegroups.com.

[Hemanth Shekar]

Hi team 

Any help on below will be greatly appreciated

Thanks a lot for your time on this

Thanks 

[Thomas SEGISMONT]

Java SSL is slower than native SSL for sure, but still 20-30ms for a handshake sounds a lot for me. Did you observe this on production? Or on a low-resource machine?

HTTP1.1 connections should be kept alive by default so there might an issue in your JMeter scenario with respect to the payload config.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CAB4KzzbzhHnSnWfS5C6TN1kQkaSczxi3qWcHc47CggkxEqS7%3Dw%40mail.gmail.com.

[Hemanth Shekar]

Hi Thomas , 

This change to Open SSL is not due to observed Slowness . 

It was a decision that team took for all the Api we have in Vertx and some of them are in production. 

need some pointers on how to convert from Java ssl to open ssl  in vertx as we are running into errors doing the change . 

Please check the trailing mail for the error message .  I added the below dependencies .

Also while starting the http  server i am enabling open ssl . 

Am i missing something?

HttpServer server = vertx

.createHttpServer(new HttpServerOptions().setSsl(config.getBoolean("security.require-ssl"))

.setKeyStoreOptions(new JksOptions().setPath(config.getString("ssl.key-store"))

.setPassword(config.getString("ssl.key-store-password")))

.setOpenSslEngineOptions(new OpenSSLEngineOptions())

.setAcceptBacklog(10000)

.setSendBufferSize(9 * 1024)

.setReceiveBufferSize(9 * 1024)

                        .setTcpKeepAlive(true

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CACiEr_Qzeitwhz%2BA8ChKgfhEFasxcVO4oP4RAeTqaRT%2BLTHPoA%40mail.gmail.com.

[Blake]

Hey Hemanth,

Try using the boringssl static library without the os classifier to see if it works . This will contain ALL of the statically linked libraries, rather than just one OS's.  I don't know if you're running this on a different machine than the one you're compiling on? I.e., if you're compiling on windows, but running on linux you will have the wrong os libs of boringssl.

https://netty.io/wiki/forked-tomcat-native.html

I don't think this is your problem, but this might be a tcnative to netty version problem. I had a problem in the past with grpc-java (uses grpc-netty) and it requires certain versions of tcnative for certain versions of grpc-netty and netty-handler (https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty)

Netty 4.1.35 requires tcnative version 2.0.25.Final which you are using, but vertx is probably using a netty version less than that. Try my first suggesion with the fll boringssl and then try using tcnative 2.0.20.Final or 2.0.17.Final if that doesn't work (again, i don't think this is your problem, but might as well start eliminating possibilities).

I think your config is fine, but I'm running some setup on my stuff now and will be testing OpenSSL and other HTTP stuff. Will get back to you with my exact config and such if it's working and all.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/e9edf7cb-f3c7-4294-b05e-171f7bbbe655%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ve...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/CAB4KzzbzhHnSnWfS5C6TN1kQkaSczxi3qWcHc47CggkxEqS7%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ve...@googlegroups.com.
Visit this group at https://groups.google.com/group/vertx.

[Hemanth Shekar]

Thanks Blake

Appreciate it 

Will get back to you after running your changes. 

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/d413f965-7075-4537-906a-7c4708ecb8f0%40googlegroups.com.

[Hemanth Shekar]

Hi Blake / Team . 

I am still getting the same error message. 

I am attaching the full error log if any one can make sense and give some pointers .

Below are the dependencies i added . 

<dependency>

  <groupId>io.netty</groupId>

  <artifactId>netty-tcnative</artifactId>

  <version>2.0.17.Final</version>

</dependency>

<dependency>

  <groupId>io.netty</groupId>

  <artifactId>netty-tcnative-boringssl-static</artifactId>

  <version>2.0.17.Final</version>

</dependency>

Thanks a lot for your help on this . 

Thanks & Regards

Hemanth H S

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/d413f965-7075-4537-906a-7c4708ecb8f0%40googlegroups.com.

[Blake]

How are you running this? I'd suggest trying to create a jar with everything packaged in it first and make sure it's working. Can use shade (maven) / shadow (gradle) plugin to do this.

Have you tried to run this on linux too? Maybe try to put it on a linux box to see what happens. Even just in docker on windows.

Are you using windows for your production system?

Working on some other dev-deployment stuff right now, but when that's done will be testing the SSL setup and stuff and will get back to you with the settings and sucht. I'm using mac and linux though. Haven't had problems in the past. Should be done within the next 24hrs.

[Hemanth Shekar]

Hi blake , 

I am running this packaged as shade jar on my local windows . 

The application is actually supposed to run in open shift in a prod environment. 

I can try deploying in lower environment and get back to you but  this is not working in my local at this moment

Thanks 

Hemanth H S 

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/17ded471-77f0-49a0-bb76-140e66ed32cd%40googlegroups.com.

[Hemanth Shekar]

Hi blake ,

Just wanted to update you, 

The deployment failed in openshift environment as well . 

Wanted to keep you posted . 

Thanks a lot for your time. 

Thanks 

Hemanth H S

On Thu, Apr 25, 2019 at 3:31 PM Blake <blakeh...@gmail.com> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

Visit this group at https://groups.google.com/group/vertx.

To view this discussion on the web, visit https://groups.google.com/d/msgid/vertx/17ded471-77f0-49a0-bb76-140e66ed32cd%40googlegroups.com.

[Blake]

Hey Hemanth,

I've created a minimal example that uses ALPN with openssl (boringssl).

Try to run this and see what happens:

https://github.com/bhowell2/VertxOpenSslTest