"com.metamx.common.ISE: Cannot have a null result!" in historical node.

85 views
Skip to first unread message

zhao weinan

unread,
Jan 14, 2015, 9:34:26 AM1/14/15
to druid-de...@googlegroups.com
Hi,

I've encountred a strange issue: while do groupby query with some select filter, some of my historical throw "com.metamx.common.ISE: Cannot have a null result!". The full stack is attached below.The more stranger thing is the exception is occured randomly.

After some digging into the source code(druid code is pretty functional, maybe you should consider implementing it using functional language such as scala...), i cannot figure out the root of problem, but i've noticed the request that received from broker with "bySegment:true", which is wired. Then i checked my broker config, there is a: druid.broker.cache.unCacheable=["select"] , when i changed to druid.broker.cache.unCacheable=["select","groupBy"], the issue gone!

I think maybe historical node cannot handle bySegment groupby query well? Even though it's true maybe it's better return a error message instead of a Http 500?

I'm using the druid-0.6.160 for broker an historical. The full stack:

com.fasterxml.jackson.databind.JsonMappingException: Cannot have a null result!
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:129)
        at com.fasterxml.jackson.databind.ObjectWriter._configAndWriteValue(ObjectWriter.java:681)
        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:534)
        at io.druid.server.QueryResource$2.write(QueryResource.java:202)
        at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
        at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
        at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1479)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132)
        at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129)
        at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
        at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
        at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:300)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:497)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:610)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:539)
        at java.lang.Thread.run(Thread.java:722)
Caused by: com.metamx.common.ISE: Cannot have a null result!
        at io.druid.query.FinalizeResultsQueryRunner$1.apply(FinalizeResultsQueryRunner.java:80)
        at com.metamx.common.guava.MappingYieldingAccumulator.accumulate(MappingYieldingAccumulator.java:57)
        at com.metamx.common.guava.BaseSequence.makeYielder(BaseSequence.java:104)
        at com.metamx.common.guava.BaseSequence.access$100(BaseSequence.java:29)
        at com.metamx.common.guava.BaseSequence$3.next(BaseSequence.java:135)
        at io.druid.jackson.DruidDefaultSerializersModule$5.serialize(DruidDefaultSerializersModule.java:140)
        at io.druid.jackson.DruidDefaultSerializersModule$5.serialize(DruidDefaultSerializersModule.java:130)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:118)
        ... 42 more
2015-01-14 21:54:25,765 WARN [qtp1483627138-27] org.eclipse.jetty.server.HttpChannel - Could not send response error 500: com.fasterxml.jackson.databind.JsonMappingException: Cannot have a null result!

Fangjin Yang

unread,
Jan 14, 2015, 12:34:02 PM1/14/15
to druid-de...@googlegroups.com
Hi Weinan, can you remove this configuration from your broker settings?

druid.broker.cache.unCacheable

More inline.

On Wednesday, January 14, 2015 at 6:34:26 AM UTC-8, zhao weinan wrote:
Hi,

I've encountred a strange issue: while do groupby query with some select filter, some of my historical throw "com.metamx.common.ISE: Cannot have a null result!". The full stack is attached below.The more stranger thing is the exception is occured randomly.

After some digging into the source code(druid code is pretty functional, maybe you should consider implementing it using functional language such as scala...), i cannot figure out the root of problem, but i've noticed the request that received from broker with "bySegment:true", which is wired. Then i checked my broker config, there is a: druid.broker.cache.unCacheable=["select"] , when i changed to druid.broker.cache.unCacheable=["select","groupBy"], the issue gone!

This is likely a bug somewhere in the logic of caching groupBys. Caching for select and groupbys should be disabled by default.

zhao weinan

unread,
Jan 14, 2015, 11:36:45 PM1/14/15
to druid-de...@googlegroups.com
Hi Fangjin, I see the default settings for druid.broker.cache.unCacheable is ["groupBy", "select"], so I think remove it will be no different?

With the changed setting druid.broker.cache.unCacheable=["select","groupBy"], the problem has gone, so I'll let it hold and watch for a while.

Nishant Bangarwa

unread,
Feb 25, 2015, 12:25:47 PM2/25/15
to druid-de...@googlegroups.com
Hi zhao, 
I tried reproducing above issue but not seeing this in my setup. 
tried issuing different combinations of caching settings with groupby, 
Can you try enabling groupBy caching and see if its still an issue,
also it will be helpful if you can provide any steps on how to reproduce this  ? 

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/d612fc12-4f5f-4275-acfb-0d9a480a83a6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--

Fangjin Yang

unread,
Feb 25, 2015, 7:31:54 PM2/25/15
to druid-de...@googlegroups.com
Hi Nishant, the error is difficult to reproduce, but I believe it can occur related to memory unmapping segments. The best way to reproduce is to probably cache groupBy queries in our metrics cluster, and wait for it to occur. We may also need more meaningful logs as the first step.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.
Nishant
Software Engineer|METAMARKETS
m +91-9729200044

Hagen Rother

unread,
Feb 27, 2015, 8:06:34 AM2/27/15
to druid-de...@googlegroups.com
Out of experience, if you do a groupBy resulting in an empty result set, you get this error.

Has this been fixed in 0.7? I would love to turn it on :)

To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/973635d0-ddc4-4472-998b-8dccc1cd1831%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Hagen Rother
Lead Architect | LiquidM
LiquidM Technology GmbH
Straße der Pariser Kommune 8 | 10243 Berlin | Germany
LinkedIn
Phone:+49 176 15 00 38 77
Internet:www.liquidm.com
Managing Directors | André Bräuer, Philipp Simon, Thomas Hille
Jurisdiction | Local Court Berlin-Charlottenburg HRB 152426 B

The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination, distribution, forwarding, or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited without the express permission of the sender. If you received this communication in error, please contact the sender and delete the material from any computer.

Himanshu

unread,
Feb 27, 2015, 9:50:11 AM2/27/15
to druid-de...@googlegroups.com
+1 for the fix

Just to confirm, the problem is very real. We've seen this too and had set the "uncacheable" property as workaround.

-- Himanshu

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CAMh1h%3DLw4XEph4kCAyki%3DBAsnutuQ%2B%3DXEb4-xioMHwSN3fJrjQ%40mail.gmail.com.

Fangjin Yang

unread,
Feb 27, 2015, 7:53:12 PM2/27/15
to druid-de...@googlegroups.com
Hi Hagen, do you by any chance have a unit test that can reproduce it?

Hagen Rother

unread,
Apr 27, 2015, 8:43:26 AM4/27/15
to druid-de...@googlegroups.com
I am sorry, my stack of unanswered druid mails is way too high... but I can happily annouce this does not seems to be an issue on 0.7.1.1 anymore:

I turned groupBy caching on and I have not seen the exception in the app that loves to trigger it.

Cheers,
Hagen

To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.



--
Hagen Rother
Lead Architect | LiquidM
LiquidM Technology GmbH
Straße der Pariser Kommune 8 | 10243 Berlin | Germany
LinkedIn
Phone:+49 176 15 00 38 77
Internet:www.liquidm.com
Managing Directors | André Bräuer, Philipp Simon, Thomas Hille
Jurisdiction | Local Court Berlin-Charlottenburg HRB 152426 B

The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination, distribution, forwarding, or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited without the express permission of the sender. If you received this communication in error, please contact the sender and delete the material from any computer.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/38cac821-fdbd-4fca-afc9-fbf50d77151d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Hagen Rother
Lead Architect | LiquidM
LiquidM Technology GmbH
Schlesische Straße 29/30, Aufgang M | 10997 Berlin | Germany
Phone:+49 176 15 00 38 77
Internet:www.liquidm.com | LinkedIn

Fangjin Yang

unread,
Apr 28, 2015, 12:13:46 AM4/28/15
to druid-de...@googlegroups.com
Great to hear :)



--
Hagen Rother
Lead Architect | LiquidM
LiquidM Technology GmbH
Straße der Pariser Kommune 8 | 10243 Berlin | Germany
LinkedIn
Phone:+49 176 15 00 38 77
Internet:www.liquidm.com
Managing Directors | André Bräuer, Philipp Simon, Thomas Hille
Jurisdiction | Local Court Berlin-Charlottenburg HRB 152426 B

The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination, distribution, forwarding, or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited without the express permission of the sender. If you received this communication in error, please contact the sender and delete the material from any computer.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.

Zhao Weinan

unread,
May 5, 2015, 11:45:42 PM5/5/15
to druid-de...@googlegroups.com
Sorry for the long disappeared, and glad to hear it.

It seems we should upgrade to 0.7.1.1, though I still cannot reproduced it after turned the groupBy-cache on

在 2015年4月27日星期一 UTC+8下午8:43:26，Hagen Rother写道：
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
Nishant
Software Engineer|METAMARKETS
m +91-9729200044

Fangjin Yang

unread,
May 6, 2015, 8:28:16 PM5/6/15
to druid-de...@googlegroups.com, xcvi...@gmail.com
Great, I closed the original issue some time ago.
Reply all
Reply to author
Forward
0 new messages