SSLEngine closed already

[Henri Tremblay]

Hi,

When running my tests (a bunch of users hammering a server without pause), after a while (usually about 150 seconds), I get this error "SSLEngine closed already". So

•  None before a precise moment
• A bunch at a given moment
• Then some every 5 seconds

So far, I thought it was my server that was having a problem but I'm not so sure anymore.

If I look at the Netty code (https://searchcode.com/codesearch/view/10525295/), it looks more like the server that decides to close the connexion and then the client that should just reopen one. I'm guessing a bit here.

So it shouldn't be a real KO. It's some kind of normal behavior.

Strangely, when using Apache Benchmark, I also have some similar errors on stdout but then Apache Benchmark doesn't consider it to be a failure in the final report.

Am I making any sense? Can someone explain the error?

Thanks,

Henri

[Stéphane LANDELLE]

Salut Henri,

Do you some kind of reproducer, please?

Thanks,

Stéphane Landelle

GatlingCorp CEO

slan...@gatling.io

--
You received this message because you are subscribed to the Google Groups "Gatling User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gatling+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Henri Tremblay]

The script is quite basic

class MySimulation extends Simulation {

  val users = Integer.getInteger("users",  1).toInt
  val ramp = Integer.getInteger("ramp", 1).toInt
  val duration = Integer.getInteger("duration", 10).toLong
  val baseUrl = System.getProperty("baseUrl")

  val httpProtocol = http
    .baseURL(baseUrl)
    .doNotTrackHeader("1")
    .disableCaching
    .userAgentHeader("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.49 Safari/537.36")
    .acceptHeader("text/html,application/xhtml+xml,application/xml;")
    .acceptEncodingHeader("gzip, deflate, sdch, br")

  val scn = scenario("test")
    .during(duration seconds) {
      exec(http("fake")
        .get("/img.png"))
    }

  setUp(scn.inject(heavisideUsers(users) over(ramp seconds))).protocols(httpProtocol)
}

It hits an https url (load balancer in front handling the SSL termination and then balancing on two servers, with a new https connection).

I will try to reproduce locally.

--
You received this message because you are subscribed to a topic in the Google Groups "Gatling User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gatling/vB3pzHYY8-s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gatling+u...@googlegroups.com.

[Henri Tremblay]

But in fact, I'm pretty sure I won't be able to reproduce out of my environment. If it can be of any use, the LB is managing the keep-alive correctly, does SSL session caching but do not support SSL tickets.

So question is: "How is the Netty piece of code below handled by Gatling?"

Right now, I fell that if the connection is closed, the client should recreate it without complaining. Because it's normal to get a closed connection after a while (I kinda think/guess). WDYT?

                SSLEngineResult result = wrap(engine, buf, out);

                if (!buf.isReadable()) {
                    buf.release();
                    promise = (ChannelPromise) pending.recycleAndGet();
                    pendingUnencryptedWrites.remove();
                } else {
                    promise = null;
                }

                if (result.getStatus() == Status.CLOSED) {
                    // SSLEngine has been closed already.
                    // Any further write attempts should be denied.
                    for (;;) {
                        PendingWrite w = pendingUnencryptedWrites.poll();
                        if (w == null) {
                            break;
                        }
                        w.failAndRecycle(SSLENGINE_CLOSED);
                    }
                    return;
                }

[Stéphane LANDELLE]

There's such a mechanism implemented for retrying such failure, but it might not work in this case.

I'd need to be able to reproduce, and will only be able to investigate when I'm back to the office in 2 weeks.

Stéphane Landelle

GatlingCorp CEO

slan...@gatling.io

[Stéphane LANDELLE]

Also, if you have some exceptions in the logs, it would help.

Stéphane Landelle

GatlingCorp CEO

slan...@gatling.io

[Henri Tremblay]

Sure. I will send you everything I have

--
You received this message because you are subscribed to a topic in the Google Groups "Gatling User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gatling/vB3pzHYY8-s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gatling+u...@googlegroups.com.

[Stéphane LANDELLE]

Please do :)

Stéphane Landelle

GatlingCorp CEO

slan...@gatling.io

[Henri Tremblay]

So the error is: j.n.s.SSLException: SSLEngine closed already

It is logged as a warning like this: 

11:39:10.971 [DEBUG] i.g.h.a.AsyncHandler - Request 'my-webfont.ttf' failed for user 86

javax.net.ssl.SSLException: SSLEngine closed already

11:39:10.973 [WARN ] i.g.h.a.ResponseProcessor - Request 'my-webfont.ttf' failed: j.n.s.SSLException: SSLEngine closed already

11:39:10.978 [DEBUG] i.g.h.a.ResponseProcessor -

>>>>>>>>>>>>>>>>>>>>>>>>>>

Request:

my-webfont.ttf: KO j.n.s.SSLException: SSLEngine closed already

=========================

Session:

.... (the session)

=========================

HTTP request:

GET https://site.com/my-webfont.ttf

headers=

Accept: */*

DNT: 1

Accept-Language: en-CA,en;q=0.8,fr;q=0.6

Accept-Encoding: gzip, deflate, sdch, br

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.49 Safari/537.36

Referer: https://site.com/thing

Cookie: ...

Host: site.com

=========================

HTTP response:

<<<<<<<<<<<<<<<<<<<<<<<<<

No stack trace when the log is enabled for errors

[Muse Mekuria]

Henri, I'm seeing the same issue. Did you figure this out?

[Henri Tremblay]

Sadly no. I never managed to get a fully reproductible use-case after the fact (not my fault, I wasn't there anymore). The root cause seemed to be the IDS that was closing connection but they should then have been recreated by gatling. 

To unsubscribe from this group and all its topics, send an email to gatling+unsubscribe@googlegroups.com.

[yva]

Hi
I have the same problem, have you found a
olution
Thanks

[Henri Tremblay]

No. I haven't. I tune the proxy that was causing issues and it disappeared. But that obviously isn't the solution from gatling side.

In my case, a load balancer was misbehaving and closing my ssl connections

[yva]

Thanks

[Kapil]

Hi Henri

I am also getting the same error and raised it in gatling forum but no reply so thought to check with you as well if you managed to resolve that or now. if yes then how? it looks like issue from Gatling side?

[Henri Tremblay]

No. As I said, something was closing the connection. But it was allowed to. We reconfigured the appliance. But Gatling wasn't recreating the connection for some reason.

[Stéphane LANDELLE]

Again, here isn't much we can do without being able to reproduce.

The only time we saw this error, the frequency was 1/1,000,000 in customer environment so not something we could debug.

Stéphane Landelle

GatlingCorp CEO

slan...@gatling.io

--

You received this message because you are subscribed to the Google Groups "Gatling User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gatling+unsubscribe@googlegroups.com.

[Kapil]

Yes Stephane, Exactly same thing in happening in my environment. out of 34000 transaction we got 1 SSLException error but because script is running continuous mode so other transactions are failing.

is there anyway in gatling to stop the iteration and start new iteration if any failure occurs ?

[Henri Tremblay]

If I recall correctly a way to reproduce would be to establish an SSL connection. Then to have the server cutting the connection because of an SSL connection timeout. Check if the client reconnects every time. 

[Abisha Sahaya]

Hi All, I am also facing the same problem . Anyone have solution for this issue.

To unsubscribe from this group and all its topics, send an email to gatling+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Gatling User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gatling+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Gatling User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gatling/vB3pzHYY8-s/unsubscribe.

To unsubscribe from this group and all its topics, send an email to gatling+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Gatling User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gatling/vB3pzHYY8-s/unsubscribe.

To unsubscribe from this group and all its topics, send an email to gatling+u...@googlegroups.com.