Update jetty from 9.2.5.v20141112 to 9.3.6.v20151106, testGzipCompression in JettyTest fail.

[宾莉金]

@Test

  public void testGzipCompression() throws Exception

  {

    final URL url = new URL("http://localhost:" + port + "/default");

    final HttpURLConnection get = (HttpURLConnection) url.openConnection();

    get.setRequestProperty("Accept-Encoding", "gzip");

    Assert.assertEquals("gzip", get.getContentEncoding());

   ......

}

I am curious how get.getContentEncoding() return gzip? 

And if i update jetty from 9.2.5.v20141112 to 9.3.6.v20151106, get.getContentEncoding() will return null, does this matter?

Any one can help?

[Charles Allen]

If I recall correctly you'll want to look for how io.druid.server.initialization.jetty.JettyServerInitUtils#defaultGzipFilterHolder is handled to follow the "how is it gzip" codepaths.

[Charles Allen]

I would love to update jetty so we can experiment with http2 stuff (see https://github.com/metamx/druid/tree/http2 for an experiment I did many months ago)

[Xavier Léauté]

Updating to Jetty 9.3.x would require building against Java 8, which is something Druid is not ready to move to yet mainly due to other dependencies.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/1faf76ab-660c-4ab0-9ef0-e806e62a6928%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Fangjin]

Maybe it is time to revisit the Java 8 requirement. I think for 0.10.0 we can force Java 8.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAKdCgy-sRkq-_kMD0K899XLoLvHi%2BhGbueOTeqNnmk_yq6mxyQ%40mail.gmail.com.

[Kurt Young]

Java8  +1

We found a jetty 9.2.* bug which will cause historical's all jetty threads stuck for nearly (queryTimeout * 1.5) time.

It's bad for our heavy load production system, so will choose to upgrade to jetty 9.3 in druid 0.8.3

[Fangjin Yang]

That's very bad. Any steps to reproduce? We saw a similar problem in older versions of Jetty.

[Kurt Young]

We can reproduce when druid in heavy loads( large datasets, reduce cache, make most query visit disk), normal query will take 30-60 seconds to finish. And when we increase the client number(300+), it will easy reproduce the bug.

[Fangjin Yang]

Kurt, can you confirm that with the jetty update, the problem goes away?

[Kurt Young]

Yes, we already done that. Jetty version is jetty-9.3.6.v20151106, some lower version of 9.3 still has the bug.

[Fangjin]

Kurt, do you have the JIRA ticket of the bug that was fixed? We can ask the Jetty folks to see if they can create a 9.2.x version of jetty with the fix.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/0d167eb8-eb73-405b-80d2-75c336b8b93a%40googlegroups.com.

[Kurt Young]

Here is the bug in 9.3 version: https://bugs.eclipse.org/bugs/show_bug.cgi?id=478923

I think this bug also effected 9.2.*, not exactly the same, but the theory are almost alike. I can't find any bug related to this in 9.2.* version.

Here is the stack when thread stuck in 9.2.*:

"qtp362370312-874" daemon prio=10 tid=0x00007f7d460ee000 nid=0x6c80 waiting on condition [0x00007f7d0f9da000]

   java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x0000000738d2d000> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176)

        at org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:213)

        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:133)

        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:347)

        at org.eclipse.jetty.servlets.gzip.DeflatedOutputStream.deflate(DeflatedOutputStream.java:74)

        at org.eclipse.jetty.servlets.gzip.DeflatedOutputStream.write(DeflatedOutputStream.java:64)

        at org.eclipse.jetty.servlets.gzip.GzipOutputStream.write(GzipOutputStream.java:46)

[宾莉金]

Yes, to use jetty 9.3.x we have upgrade our java to java 8.

在 2016年2月18日星期四 UTC+8上午2:19:14，Xavier写道：

[宾莉金]

Yes, i see the io.druid.server.initialization.jetty.JettyServerInitUtils#defaultGzipFilterHolder, but still don't know why jetty 9.3.6.v20151106 fail the test.

在 2016年2月18日星期四 UTC+8上午1:21:18，Charles Allen写道：

[Prashant Deva]

+1
seen this issue many times on our druid install.
didnt know jetty was the culprit.

[Nishant Bangarwa]

@Kurt, We also saw issues at scale with jetty 9.2.11 and later versions specifically, issue reported is here - 

https://github.com/eclipse/jetty.project/issues/235

which 9.2.x version of jetty were you using when you faced this issue ? 

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/b74075ab-84d6-4d09-b649-d87a8c7a91b2%40googlegroups.com.

[Fangjin Yang]

@Nishant is there a version of Jetty we can downgrade to to get around the issue?

On Monday, February 22, 2016 at 6:51:07 AM UTC-8, Nishant Bangarwa wrote:

@Kurt, We also saw issues at scale with jetty 9.2.11 and later versions specifically, issue reported is here - 

https://github.com/eclipse/jetty.project/issues/235

which 9.2.x version of jetty were you using when you faced this issue ? 

On Fri, Feb 19, 2016 at 11:34 PM Prashant Deva <prasha...@gmail.com> wrote:

+1
seen this issue many times on our druid install.
didnt know jetty was the culprit.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.

[Nishant Bangarwa]

In our testing, the Jetty version 9.2.5 (currently used by druid) seems to not have this issue. 

On Mon, Feb 22, 2016 at 11:35 PM Fangjin Yang <fan...@imply.io> wrote:

@Nishant is there a version of Jetty we can downgrade to to get around the issue?

On Monday, February 22, 2016 at 6:51:07 AM UTC-8, Nishant Bangarwa wrote:

@Kurt, We also saw issues at scale with jetty 9.2.11 and later versions specifically, issue reported is here - 

https://github.com/eclipse/jetty.project/issues/235

which 9.2.x version of jetty were you using when you faced this issue ? 

On Fri, Feb 19, 2016 at 11:34 PM Prashant Deva <prasha...@gmail.com> wrote:

+1
seen this issue many times on our druid install.
didnt know jetty was the culprit.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/b74075ab-84d6-4d09-b649-d87a8c7a91b2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Druid Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/510dba6d-14c5-4fa4-bf30-81f67c18adf6%40googlegroups.com.

[Fangjin]

Hmm, Kurt reports this version of Jetty is still bad though.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CACW6ntf54wTk62iNrgwDu8f7g0rA0hf%2BZqOusC6zXoUqjOhn3g%40mail.gmail.com.

[Fangjin]

Hi Kurt/Nishant, can you confirm if 9.2.5 is good or bad?

[Kurt Young]

we were using 9.2.5, and it's bad

[Charles Allen]

What node does this appear on?

[Charles Allen]

Also note https://github.com/eclipse/jetty.project/issues/217 gzip compression looks like it changed in 9.3

On Tuesday, February 16, 2016 at 6:57:52 PM UTC-8, 宾莉金 wrote:

[Charles Allen]

Oh i see > historical's all jetty threads

[宾莉金]

Thanks very much, @Charles Allen.

在 2016年2月25日星期四 UTC+8上午8:12:32，Charles Allen写道：

[Fangjin Yang]

I think we should talk about this during the next dev sync.

[Charles Allen]

Agreed. @FJ you're pretty active in this thread, can you do a brief pitch of the issue on Tuesday?

[Fangjin Yang]

We're doing a user poll to see how much updating from java 7 to 8 will impact folks.