Zookeeper failure leads to segment insertion failure

[TJ Giuli]

Howdy, my Zookeeper cluster experienced a failure and when it came back up, my overlord is throwing exceptions.  Here's the error I see in my overlord logs:

2015-05-12T23:40:05,304 INFO [qtp152661048-142] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_realtime_http_2015-05-12T22:00:00.000Z_0_0]: SegmentInsertAction{segments=[DataSegment{size=171007, shardSpec=LinearShardSpec{partitionNum=0}, metrics=[m1, m2], dimensions=[d1, d2], version='2015-05-12T22:02:03.459Z', loadSpec={type=hdfs, path=hdfs:/druid/http/20150512T220000.000Z_20150512T230000.000Z/2015-05-12T22_02_03.459Z/0/index.zip}, interval=2015-05-12T22:00:00.000Z/2015-05-12T23:00:00.000Z, dataSource='http', binaryVersion='9'}]}

2015-05-12T23:40:05,304 INFO [qtp152661048-142] io.druid.indexing.overlord.MetadataTaskStorage - Logging action for task[index_realtime_http_2015-05-12T22:00:00.000Z_0_0]: SegmentInsertAction{segments=[DataSegment{size=171007, shardSpec=LinearShardSpec{partitionNum=0}, metrics=[m1, m2], dimensions=[d1, d2], version='2015-05-12T22:02:03.459Z', loadSpec={type=hdfs, path=hdfs:/druid/http/20150512T220000.000Z_20150512T230000.000Z/2015-05-12T22_02_03.459Z/0/index.zip}, interval=2015-05-12T22:00:00.000Z/2015-05-12T23:00:00.000Z, dataSource='http', binaryVersion='9'}]}

May 12, 2015 11:40:05 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException

SEVERE: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container

com.metamx.common.ISE: Segments not covered by locks for task: index_realtime_http_2015-05-12T22:00:00.000Z_0_0

at io.druid.indexing.common.actions.TaskActionToolbox.verifyTaskLocksAndSinglePartitionSettitude(TaskActionToolbox.java:93)

at io.druid.indexing.common.actions.SegmentInsertAction.perform(SegmentInsertAction.java:80)

at io.druid.indexing.common.actions.SegmentInsertAction.perform(SegmentInsertAction.java:32)

at io.druid.indexing.common.actions.LocalTaskActionClient.submit(LocalTaskActionClient.java:62)

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:255)

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:244)

at io.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:482)

at io.druid.indexing.overlord.http.OverlordResource.doAction(OverlordResource.java:241)

at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)

at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)

at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)

at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278)

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268)

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)

at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132)

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129)

at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206)

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:69)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)

at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

at org.eclipse.jetty.server.Server.handle(Server.java:497)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620)

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540)

at java.lang.Thread.run(Thread.java:745)

2015-05-12T23:40:05,355 WARN [qtp152661048-142] org.eclipse.jetty.servlet.ServletHandler - /druid/indexer/v1/action

com.metamx.common.ISE: Segments not covered by locks for task: index_realtime_http_2015-05-12T22:00:00.000Z_0_0

at io.druid.indexing.common.actions.TaskActionToolbox.verifyTaskLocksAndSinglePartitionSettitude(TaskActionToolbox.java:93) ~[druid-indexing-service-0.7.1.1.jar:0.7.1.1]

at io.druid.indexing.common.actions.SegmentInsertAction.perform(SegmentInsertAction.java:80) ~[druid-indexing-service-0.7.1.1.jar:0.7.1.1]

at io.druid.indexing.common.actions.SegmentInsertAction.perform(SegmentInsertAction.java:32) ~[druid-indexing-service-0.7.1.1.jar:0.7.1.1]

at io.druid.indexing.common.actions.LocalTaskActionClient.submit(LocalTaskActionClient.java:62) ~[druid-indexing-service-0.7.1.1.jar:0.7.1.1]

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:255) ~[druid-indexing-service-0.7.1.1.jar:0.7.1.1]

at io.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:244) ~[druid-indexing-service-0.7.1.1.jar:0.7.1.1]

at io.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:482) ~[druid-indexing-service-0.7.1.1.jar:0.7.1.1]

at io.druid.indexing.overlord.http.OverlordResource.doAction(OverlordResource.java:241) ~[druid-indexing-service-0.7.1.1.jar:0.7.1.1]

at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) ~[?:?]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_75]

at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_75]

at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381) ~[jersey-server-1.17.1.jar:1.17.1]

at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) ~[jersey-servlet-1.17.1.jar:1.17.1]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538) ~[jersey-servlet-1.17.1.jar:1.17.1]

at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716) ~[jersey-servlet-1.17.1.jar:1.17.1]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]

at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278) ~[guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268) ~[guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180) ~[guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93) ~[guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) ~[guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132) ~[guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129) ~[guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206) ~[guice-servlet-4.0-beta.jar:?]

at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129) ~[guice-servlet-4.0-beta.jar:?]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at io.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:69) ~[druid-server-0.7.1.1.jar:0.7.1.1]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) ~[jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) ~[jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.Server.handle(Server.java:497) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]

Any idea what's going on?  Thanks!

--T

[Fangjin Yang]

These messages should be harmless and is the overlord recovering from a disconnect. I don't recall the exact nature why they occur, but perhaps Gian can chime in here.

[Gian Merlino]

I think you're seeing https://github.com/druid-io/druid/issues/709. The quickest "fix" is to restart your middleManagers, which will cause tasks to fail but will at least bring things on the overlord back in sync with the rest of the cluster.

[TJ Giuli]

Ah, ok, thanks.