Question on Broker WARN

[Jae Hyeon Bae]

Why am I seeing this WARN from time to time?

WARN  2013-06-22 03:26:16,516 [243424006@qtp-182912053-8] com.metamx.druid.http.QueryServlet: Exception occurred on request [TimeseriesQuery{dataSource='nf_errors_log', querySegmentSpec=LegacySegmentSpec{intervals=[2013-06-22T03:16:00.000Z/2013-06-22T03:26:00.000Z]}, dimFilter=app = chukwatracker, granularity='DurationGranularity{length=60000, origin=0}', aggregatorSpecs=[CountAggregatorFactory{name='resCount'}], postAggregatorSpecs=[], context=null}]

com.fasterxml.jackson.databind.JsonMappingException: [no message for java.util.NoSuchElementException]

        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:131)

        at com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:610)

        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:497)

        at com.metamx.druid.http.QueryServlet.doPost(QueryServlet.java:108)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)

        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)

        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)

        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

        at org.mortbay.jetty.Server.handle(Server.java:326)

        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)

        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)

        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)

        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)

        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)

        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)

        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Caused by: java.util.NoSuchElementException

        at java.util.LinkedList.removeFirst(LinkedList.java:268)

        at com.metamx.http.client.io.AppendableByteArrayInputStream.scanThroughBytesAndDoSomething(AppendableByteArrayInputStream.java:119)

        at com.metamx.http.client.io.AppendableByteArrayInputStream.read(AppendableByteArrayInputStream.java:60)

        at com.fasterxml.jackson.dataformat.smile.SmileParserBootstrapper.ensureLoaded(SmileParserBootstrapper.java:253)

        at com.fasterxml.jackson.dataformat.smile.SmileParserBootstrapper.constructParser(SmileParserBootstrapper.java:96)

        at com.fasterxml.jackson.dataformat.smile.SmileFactory._createParser(SmileFactory.java:427)

        at com.fasterxml.jackson.dataformat.smile.SmileFactory.createJsonParser(SmileFactory.java:306)

        at com.fasterxml.jackson.dataformat.smile.SmileFactory.createJsonParser(SmileFactory.java:27)

        at com.metamx.druid.client.DirectDruidClient$JsonParserIterator.init(DirectDruidClient.java:270)

        at com.metamx.druid.client.DirectDruidClient$JsonParserIterator.hasNext(DirectDruidClient.java:233)

        at com.metamx.common.guava.BaseSequence.makeYielder(BaseSequence.java:84)

        at com.metamx.common.guava.BaseSequence.toYielder(BaseSequence.java:74)

        at com.metamx.common.guava.MappedSequence.toYielder(MappedSequence.java:46)

        at com.metamx.common.guava.MergeSequence$2.accumulate(MergeSequence.java:68)

        at com.metamx.common.guava.MergeSequence$2.accumulate(MergeSequence.java:64)

        at com.metamx.common.guava.YieldingAccumulators$1.accumulate(YieldingAccumulators.java:32)

        at com.metamx.common.guava.BaseSequence.makeYielder(BaseSequence.java:85)

        at com.metamx.common.guava.BaseSequence.toYielder(BaseSequence.java:74)

        at com.metamx.common.guava.BaseSequence.accumulate(BaseSequence.java:63)

        at com.metamx.common.guava.MergeSequence.toYielder(MergeSequence.java:61)

        at com.metamx.common.guava.YieldingSequenceBase.accumulate(YieldingSequenceBase.java:15)

        at com.metamx.common.guava.LazySequence.accumulate(LazySequence.java:37)

        at com.metamx.common.guava.ConcatSequence$1.accumulate(ConcatSequence.java:45)

        at com.metamx.common.guava.ConcatSequence$1.accumulate(ConcatSequence.java:41)

        at com.metamx.common.guava.YieldingAccumulators$1.accumulate(YieldingAccumulators.java:32)

        at com.metamx.common.guava.BaseSequence.makeYielder(BaseSequence.java:85)

        at com.metamx.common.guava.BaseSequence.toYielder(BaseSequence.java:74)

        at com.metamx.common.guava.BaseSequence.accumulate(BaseSequence.java:63)

        at com.metamx.common.guava.ConcatSequence.accumulate(ConcatSequence.java:39)

        at com.metamx.druid.query.MetricsEmittingQueryRunner$1.accumulate(MetricsEmittingQueryRunner.java:83)

        at com.metamx.druid.guava.CombiningSequence.accumulate(CombiningSequence.java:66)

        at com.metamx.common.guava.MappedSequence.accumulate(MappedSequence.java:40)

        at com.metamx.druid.jackson.DefaultObjectMapper$4.serialize(DefaultObjectMapper.java:118)

        at com.metamx.druid.jackson.DefaultObjectMapper$4.serialize(DefaultObjectMapper.java:112)

        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:120)

        ... 18 more

[Eric Tschetter]

Hrm, I'm not sure why that would happen.  It just generally shouldn't happen.  It *could* happen if you have multiple threads reading from the same InputStream, but it doesn't look like that would be the case.  Do you have any information about what kind of circumstances it happens in, or is it just random?  Like, are there any similarities between the logs at the various different times that it happens?

Fwiw, the code that appears to be erroring out is in our http-client:

https://github.com/metamx/http-client

There are unit tests for the class AppendableByteArrayInputStream at AppendableByteArrayInputStreamTest, if you can reproduce it in a unit test, we should be able to fix it.

--Eric

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/e4fb0876-1724-453f-bfb7-64c2d353baa9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.