0.9.2 rc2 kafka indexing service OOM
Visto 157 veces
Saltar al primer mensaje no leído
Linbo Jin
8 nov 2016, 20:17:258/11/16
a Druid User
Hi guys,
We are using kafka indexign service. We know maxRowInMemory in druid 0.9.1.1 is not correct (which works on individual segment), and our use case (real time data ingestion as well as migration old data couple months) will generate a lot of segments(daily, around 180 for each datasouce) within taskDuration(30h). So we suffered a lot of OOM and then we updated to 0.9.2_rc2. It really helped a lot, we didn't see OOM during indexing any more. But we saw OOM again when we query the whole range data (4 months and all have not be published) from the biggest datasource. From indexing log, we saw there are query time out, connection loss and then OOM.
We set maxRowInMemory = 20000 and one peon memory = 1G. When indexing it works well for all the datasources and after indexing the old data we query it works well for other datasources and we saw OOM for this biggest datasource. Supervisor killed the indexing task for this datasource and restarted new one to index again, but we saw Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded.
We thought maxRowInMemory will work on all segments in real-time node(running kafka indexing service) and it should not be OOM even we have a lot of segment. Maybe the problem is when we query the data, druid will pick up data into memory and cause OOM? Can you give any suggestions how to solve this problem.
a part of log:
2016-11-08T15:53:18,016 ERROR [Curator-ServiceProvider-0-EventThread] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (19070)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.doSyncForSuspendedConnection(CuratorFrameworkImpl.java:701) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.suspendConnection(CuratorFrameworkImpl.java:674) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.validateConnection(CuratorFrameworkImpl.java:621) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processEvent(CuratorFrameworkImpl.java:877) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$000(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$1.process(CuratorFrameworkImpl.java:109) [curator-framework-2.11.0.jar:?]
at org.apache.curator.ConnectionState.process(ConnectionState.java:164) [curator-client-2.11.0.jar:?]
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) [zookeeper-3.4.9.jar:3.4.9-1757313]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) [zookeeper-3.4.9.jar:3.4.9-1757313]
2016-11-08T15:53:19,635 ERROR [qtp1141984159-61] com.sun.jersey.spi.container.ContainerResponse - The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
java.lang.OutOfMemoryError: Java heap space
at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:114) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.io.IOContext.allocConcatBuffer(IOContext.java:186) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.<init>(WriterBasedJsonGenerator.java:76) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.JsonFactory._createGenerator(JsonFactory.java:1261) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.JsonFactory.createGenerator(JsonFactory.java:1051) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2338) ~[jackson-databind-2.4.6.jar:2.4.6]
at io.druid.query.DruidMetrics.makeQueryTimeMetric(DruidMetrics.java:104) ~[druid-processing-0.9.2-rc2.jar:0.9.2-rc2]
at io.druid.server.QueryResource.doPost(QueryResource.java:313) ~[druid-server-0.9.2-rc2.jar:0.9.2-rc2]
at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_74]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_74]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.jar:1.19]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) ~[guice-servlet-4.1.0.jar:?]
2016-11-08T15:53:22,026 ERROR [Curator-ServiceProvider-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (24851)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:507) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:203) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:508) [curator-recipes-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) [curator-recipes-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) [curator-recipes-2.11.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_74]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_74]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:22,024 WARN [qtp1141984159-61] org.eclipse.jetty.servlet.ServletHandler -
javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420) ~[jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.jar:1.19]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) ~[guice-servlet-4.1.0.jar:?]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) ~[jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) ~[jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) ~[jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.Server.handle(Server.java:497) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:114) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.io.IOContext.allocConcatBuffer(IOContext.java:186) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.<init>(WriterBasedJsonGenerator.java:76) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.JsonFactory._createGenerator(JsonFactory.java:1261) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.core.JsonFactory.createGenerator(JsonFactory.java:1051) ~[jackson-core-2.4.6.jar:2.4.6]
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2338) ~[jackson-databind-2.4.6.jar:2.4.6]
at io.druid.query.DruidMetrics.makeQueryTimeMetric(DruidMetrics.java:104) ~[druid-processing-0.9.2-rc2.jar:0.9.2-rc2]
at io.druid.server.QueryResource.doPost(QueryResource.java:313) ~[druid-server-0.9.2-rc2.jar:0.9.2-rc2]
at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_74]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_74]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.jar:1.19]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) ~[guice-servlet-4.1.0.jar:?]
2016-11-08T15:53:23,635 ERROR [Announcer-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (24851)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:507) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:203) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:508) [curator-recipes-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) [curator-recipes-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) [curator-recipes-2.11.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_74]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_74]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,635 ERROR [Announcer-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26460)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:507) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:203) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:508) [curator-recipes-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) [curator-recipes-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) [curator-recipes-2.11.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_74]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_74]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,636 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26461)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,637 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26462)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,637 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26462)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,637 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26462)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,637 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26462)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,637 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26462)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,638 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26463)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,638 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26463)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,638 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26463)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,638 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26463)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,639 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26464)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,639 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26464)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,639 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26464)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,639 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26464)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,640 ERROR [Curator-ServiceProvider-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (26465)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:507) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:203) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:40) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:508) [curator-recipes-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) [curator-recipes-2.11.0.jar:?]
at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) [curator-recipes-2.11.0.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_74]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_74]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:23,641 INFO [binnedSessions-incremental-persist] io.druid.segment.realtime.appenderator.AppenderatorImpl - Segment[binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z], persisting Hydrant[FireHydrant{index=io.druid.segment.incremental.OnheapIncrementalIndex@11181ca6, queryable=io.druid.segment.ReferenceCountingSegment@53548ff8, count=25}]
2016-11-08T15:53:23,641 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - Starting persist for interval[2016-11-02T00:00:00.000Z/2016-11-03T00:00:00.000Z], rows[10,160]
2016-11-08T15:53:23,647 WARN [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Ingestion was throttled for [2,423] millis because persists were pending.
2016-11-08T15:53:23,664 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - outDir[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/25/v8-tmp] completed index.drd in 0 millis.
2016-11-08T15:53:23,664 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[system] conversions with cardinality[1] in 0 millis.
2016-11-08T15:53:23,665 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[ctrlMac] conversions with cardinality[2] in 0 millis.
2016-11-08T15:53:23,665 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[ctrlName] conversions with cardinality[2] in 0 millis.
2016-11-08T15:53:23,666 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[ctrlSerial] conversions with cardinality[1] in 0 millis.
2016-11-08T15:53:23,666 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[ctrlModel] conversions with cardinality[1] in 0 millis.
2016-11-08T15:53:23,667 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[domains] conversions with cardinality[4] in 0 millis.
2016-11-08T15:53:23,667 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[zoneName] conversions with cardinality[1] in 0 millis.
2016-11-08T15:53:23,668 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apGroupName] conversions with cardinality[23] in 0 millis.
2016-11-08T15:53:23,669 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apMac] conversions with cardinality[660] in 1 millis.
2016-11-08T15:53:23,670 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apName] conversions with cardinality[659] in 1 millis.
2016-11-08T15:53:23,671 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apSerial] conversions with cardinality[660] in 1 millis.
2016-11-08T15:53:23,671 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apModel] conversions with cardinality[4] in 0 millis.
2016-11-08T15:53:23,672 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apLocation] conversions with cardinality[3] in 0 millis.
2016-11-08T15:53:23,672 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apDescription] conversions with cardinality[2] in 0 millis.
2016-11-08T15:53:23,673 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apIp] conversions with cardinality[660] in 0 millis.
2016-11-08T15:53:23,674 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apExtIp] conversions with cardinality[12] in 0 millis.
2016-11-08T15:53:23,674 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[ssid] conversions with cardinality[1] in 0 millis.
2016-11-08T15:53:23,675 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[radio] conversions with cardinality[2] in 1 millis.
2016-11-08T15:53:23,677 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[clientMac] conversions with cardinality[4,831] in 2 millis.
2016-11-08T15:53:23,680 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[clientIp] conversions with cardinality[4,655] in 2 millis.
2016-11-08T15:53:23,682 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[username] conversions with cardinality[2,487] in 1 millis.
2016-11-08T15:53:23,685 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[hostname] conversions with cardinality[3,446] in 2 millis.
2016-11-08T15:53:23,690 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[sessionId] conversions with cardinality[10,160] in 5 millis.
2016-11-08T15:53:23,691 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[osType] conversions with cardinality[11] in 0 millis.
2016-11-08T15:53:23,692 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[manufacturer] conversions with cardinality[31] in 0 millis.
2016-11-08T15:53:23,692 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[sessionType] conversions with cardinality[2] in 0 millis.
2016-11-08T15:53:23,693 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - outDir[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/25/v8-tmp] completed dim conversions in 29 millis.
2016-11-08T15:53:25,362 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28187)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,362 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28187)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,363 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28188)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,363 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28188)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,363 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28188)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,363 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28188)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,363 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28188)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,364 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28189)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,364 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28189)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,364 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28189)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,364 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28189)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,364 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28189)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,364 ERROR [Curator-Framework-0] org.apache.curator.ConnectionState - Connection timed out for connection string (master-5ce39db4.node.rsa) and timeout (15000) / elapsed (28189)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197) [curator-client-2.11.0.jar:?]
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.11.0.jar:?]
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:116) [curator-client-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:835) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:809) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:64) [curator-framework-2.11.0.jar:?]
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:267) [curator-framework-2.11.0.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_74]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:25,367 INFO [Curator-ServiceProvider-0-SendThread(master-5ce39db4.node.dc1.rsa:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server master-5ce39db4.node.dc1.rsa/10.0.3.13:2181. Will not attempt to authenticate using SASL (unknown error)
2016-11-08T15:53:25,370 INFO [Curator-ServiceProvider-0-SendThread(master-5ce39db4.node.dc1.rsa:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to master-5ce39db4.node.dc1.rsa/10.0.3.13:2181, initiating session
2016-11-08T15:53:25,373 WARN [Curator-ServiceProvider-0-SendThread(master-5ce39db4.node.dc1.rsa:2181)] org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x15842ed22300019 has expired
2016-11-08T15:53:25,373 INFO [Curator-ServiceProvider-0-SendThread(master-5ce39db4.node.dc1.rsa:2181)] org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service, session 0x15842ed22300019 has expired, closing socket connection
2016-11-08T15:53:25,373 WARN [Curator-ServiceProvider-0-EventThread] org.apache.curator.ConnectionState - Session expired event received
2016-11-08T15:53:25,373 INFO [Curator-ServiceProvider-0-EventThread] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=master-5ce39db4.node.rsa sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@47acd13b
2016-11-08T15:53:25,375 INFO [Curator-ServiceProvider-0-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: LOST
2016-11-08T15:53:25,375 INFO [Curator-ServiceProvider-0-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x15842ed22300019
2016-11-08T15:53:25,376 INFO [Curator-ServiceProvider-0-SendThread(master-5ce39db4.node.dc1.rsa:2181)] org.apache.zookeeper.ClientCnxn - Opening socket connection to server master-5ce39db4.node.dc1.rsa/10.0.3.13:2181. Will not attempt to authenticate using SASL (unknown error)
2016-11-08T15:53:25,376 INFO [Announcer-0] io.druid.curator.announcement.Announcer - Node[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480] is added to reinstate.
2016-11-08T15:53:25,376 INFO [Announcer-0] io.druid.curator.announcement.Announcer - Node[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2] is added to reinstate.
2016-11-08T15:53:25,376 INFO [Announcer-0] io.druid.curator.announcement.Announcer - Node[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61] is added to reinstate.
2016-11-08T15:53:25,376 INFO [Announcer-0] io.druid.curator.announcement.Announcer - Node[/druid/listeners/lookups/__default/data-8ce3acae.node.rsa:8100] is added to reinstate.
2016-11-08T15:53:25,376 INFO [Announcer-0] io.druid.curator.announcement.Announcer - Node[/druid/announcements/data-8ce3acae.node.rsa:8100] is added to reinstate.
2016-11-08T15:53:25,378 INFO [Curator-ServiceProvider-0-SendThread(master-5ce39db4.node.dc1.rsa:2181)] org.apache.zookeeper.ClientCnxn - Socket connection established to master-5ce39db4.node.dc1.rsa/10.0.3.13:2181, initiating session
2016-11-08T15:53:25,380 INFO [Curator-ServiceProvider-0-SendThread(master-5ce39db4.node.dc1.rsa:2181)] org.apache.zookeeper.ClientCnxn - Session establishment complete on server master-5ce39db4.node.dc1.rsa/10.0.3.13:2181, sessionid = 0x15842ed2230001a, negotiated timeout = 30000
2016-11-08T15:53:25,380 INFO [Curator-ServiceProvider-0-EventThread] org.apache.curator.framework.state.ConnectionStateManager - State change: RECONNECTED
2016-11-08T15:53:25,383 INFO [Announcer-0] io.druid.curator.announcement.Announcer - Reinstating [/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:25,383 INFO [Announcer-0] io.druid.curator.announcement.Announcer - Reinstating [/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:25,384 INFO [Announcer-0] io.druid.curator.announcement.Announcer - Reinstating [/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:25,384 INFO [Announcer-0] io.druid.curator.announcement.Announcer - Reinstating [/druid/listeners/lookups/__default/data-8ce3acae.node.rsa:8100]
2016-11-08T15:53:25,384 INFO [Announcer-0] io.druid.curator.announcement.Announcer - Reinstating [/druid/announcements/data-8ce3acae.node.rsa:8100]
2016-11-08T15:53:31,794 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - outDir[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/25/v8-tmp] completed walk through of 10,160 rows in 8,101 millis.
2016-11-08T15:53:31,794 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[system] with cardinality[1]
2016-11-08T15:53:31,796 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[system] in 2 millis.
2016-11-08T15:53:31,796 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[ctrlMac] with cardinality[2]
2016-11-08T15:53:31,797 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[ctrlMac] in 1 millis.
2016-11-08T15:53:31,798 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[ctrlName] with cardinality[2]
2016-11-08T15:53:31,799 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[ctrlName] in 1 millis.
2016-11-08T15:53:31,800 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[ctrlSerial] with cardinality[1]
2016-11-08T15:53:31,801 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[ctrlSerial] in 1 millis.
2016-11-08T15:53:31,802 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[ctrlModel] with cardinality[1]
2016-11-08T15:53:31,803 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[ctrlModel] in 2 millis.
2016-11-08T15:53:31,803 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[domains] with cardinality[4]
2016-11-08T15:53:32,597 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[domains] in 794 millis.
2016-11-08T15:53:32,598 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[zoneName] with cardinality[1]
2016-11-08T15:53:32,599 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[zoneName] in 1 millis.
2016-11-08T15:53:32,600 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apGroupName] with cardinality[23]
2016-11-08T15:53:32,601 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apGroupName] in 1 millis.
2016-11-08T15:53:32,602 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apMac] with cardinality[660]
2016-11-08T15:53:32,604 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apMac] in 3 millis.
2016-11-08T15:53:32,604 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apName] with cardinality[659]
2016-11-08T15:53:32,606 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apName] in 2 millis.
2016-11-08T15:53:32,607 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apSerial] with cardinality[660]
2016-11-08T15:53:32,609 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apSerial] in 2 millis.
2016-11-08T15:53:32,609 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apModel] with cardinality[4]
2016-11-08T15:53:32,610 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apModel] in 1 millis.
2016-11-08T15:53:32,611 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apLocation] with cardinality[3]
2016-11-08T15:53:32,612 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apLocation] in 1 millis.
2016-11-08T15:53:32,613 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apDescription] with cardinality[2]
2016-11-08T15:53:32,614 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apDescription] in 1 millis.
2016-11-08T15:53:32,614 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apIp] with cardinality[660]
2016-11-08T15:53:33,411 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apIp] in 797 millis.
2016-11-08T15:53:34,197 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apExtIp] with cardinality[12]
2016-11-08T15:53:34,198 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apExtIp] in 1 millis.
2016-11-08T15:53:34,199 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[ssid] with cardinality[1]
2016-11-08T15:53:34,200 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[ssid] in 2 millis.
2016-11-08T15:53:34,201 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[radio] with cardinality[2]
2016-11-08T15:53:34,202 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[radio] in 2 millis.
2016-11-08T15:53:34,203 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[clientMac] with cardinality[4,831]
2016-11-08T15:53:34,210 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[clientMac] in 7 millis.
2016-11-08T15:53:34,210 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[clientIp] with cardinality[4,655]
2016-11-08T15:53:34,984 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[clientIp] in 774 millis.
2016-11-08T15:53:34,985 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[username] with cardinality[2,487]
2016-11-08T15:53:34,989 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[username] in 4 millis.
2016-11-08T15:53:34,989 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[hostname] with cardinality[3,446]
2016-11-08T15:53:35,774 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[hostname] in 785 millis.
2016-11-08T15:53:35,775 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[sessionId] with cardinality[10,160]
2016-11-08T15:53:36,560 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[sessionId] in 785 millis.
2016-11-08T15:53:37,336 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[osType] with cardinality[11]
2016-11-08T15:53:37,337 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[osType] in 2 millis.
2016-11-08T15:53:37,337 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[manufacturer] with cardinality[31]
2016-11-08T15:53:37,338 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[manufacturer] in 1 millis.
2016-11-08T15:53:37,338 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[sessionType] with cardinality[2]
2016-11-08T15:53:37,339 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[sessionType] in 1 millis.
2016-11-08T15:53:37,340 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - outDir[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/25/v8-tmp] completed inverted.drd and wrote dimensions in 5,545 millis.
2016-11-08T15:53:37,340 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - wrote metadata.drd in outDir[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/25/v8-tmp].
2016-11-08T15:53:37,345 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Converting v8[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/25/v8-tmp] to v9[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/25]
2016-11-08T15:53:37,345 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apDescription.drd]
2016-11-08T15:53:37,345 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apDescription] is single value, converting...
2016-11-08T15:53:37,347 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apExtIp.drd]
2016-11-08T15:53:37,347 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apExtIp] is single value, converting...
2016-11-08T15:53:37,349 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apGroupName.drd]
2016-11-08T15:53:37,349 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apGroupName] is single value, converting...
2016-11-08T15:53:38,121 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apIp.drd]
2016-11-08T15:53:38,121 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apIp] is single value, converting...
2016-11-08T15:53:38,122 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apLocation.drd]
2016-11-08T15:53:38,123 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apLocation] is single value, converting...
2016-11-08T15:53:38,124 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apMac.drd]
2016-11-08T15:53:38,124 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apMac] is single value, converting...
2016-11-08T15:53:38,125 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apModel.drd]
2016-11-08T15:53:38,125 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apModel] is single value, converting...
2016-11-08T15:53:38,127 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apName.drd]
2016-11-08T15:53:38,899 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apName] is single value, converting...
2016-11-08T15:53:38,900 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apSerial.drd]
2016-11-08T15:53:38,900 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apSerial] is single value, converting...
2016-11-08T15:53:38,901 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_clientIp.drd]
2016-11-08T15:53:38,902 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[clientIp] is single value, converting...
2016-11-08T15:53:38,904 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_clientMac.drd]
2016-11-08T15:53:38,904 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[clientMac] is single value, converting...
2016-11-08T15:53:39,675 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_ctrlMac.drd]
2016-11-08T15:53:39,675 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[ctrlMac] is single value, converting...
2016-11-08T15:53:39,678 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_ctrlModel.drd]
2016-11-08T15:53:39,679 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[ctrlModel] is single value, converting...
2016-11-08T15:53:39,680 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_ctrlName.drd]
2016-11-08T15:53:39,680 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[ctrlName] is single value, converting...
2016-11-08T15:53:39,681 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_ctrlSerial.drd]
2016-11-08T15:53:39,681 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[ctrlSerial] is single value, converting...
2016-11-08T15:53:40,456 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_domains.drd]
2016-11-08T15:53:40,460 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_hostname.drd]
2016-11-08T15:53:40,460 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[hostname] is single value, converting...
2016-11-08T15:53:40,462 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_manufacturer.drd]
2016-11-08T15:53:40,463 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[manufacturer] is single value, converting...
2016-11-08T15:53:41,237 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_osType.drd]
2016-11-08T15:53:41,237 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[osType] is single value, converting...
2016-11-08T15:53:41,239 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_radio.drd]
2016-11-08T15:53:41,240 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[radio] is single value, converting...
2016-11-08T15:53:41,243 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_sessionId.drd]
2016-11-08T15:53:41,244 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[sessionId] is single value, converting...
2016-11-08T15:53:42,014 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_sessionType.drd]
2016-11-08T15:53:42,015 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[sessionType] is single value, converting...
2016-11-08T15:53:42,020 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_ssid.drd]
2016-11-08T15:53:42,020 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[ssid] is single value, converting...
2016-11-08T15:53:42,021 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_system.drd]
2016-11-08T15:53:42,022 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[system] is single value, converting...
2016-11-08T15:53:42,799 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_username.drd]
2016-11-08T15:53:42,799 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[username] is single value, converting...
2016-11-08T15:53:42,800 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_zoneName.drd]
2016-11-08T15:53:42,801 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[zoneName] is single value, converting...
2016-11-08T15:53:42,801 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[index.drd]
2016-11-08T15:53:42,801 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[inverted.drd]
2016-11-08T15:53:42,801 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_firstRss_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,801 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_firstSnr_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,801 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_lastRss_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,801 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_lastSnr_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,801 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_maxRss_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,801 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_maxSnr_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,801 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_minRss_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,801 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_minSnr_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,802 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_noiseFloor_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,802 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_rxBytes_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,802 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_throughputEstimate_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,802 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_traffic_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,802 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_txBytes_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,802 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[metadata.drd]
2016-11-08T15:53:42,802 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[spatial.drd]
2016-11-08T15:53:42,802 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[time_LITTLE_ENDIAN.drd]
2016-11-08T15:53:42,802 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Skipped files[[index.drd, inverted.drd, metadata.drd, spatial.drd]]
2016-11-08T15:53:43,582 INFO [binnedSessions-incremental-persist] io.druid.segment.ReferenceCountingSegment - Closing binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z
2016-11-08T15:53:43,582 INFO [binnedSessions-incremental-persist] io.druid.segment.ReferenceCountingSegment - Closing binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z, numReferences: 0
2016-11-08T15:53:43,582 INFO [binnedSessions-incremental-persist] io.druid.segment.realtime.appenderator.AppenderatorImpl - Segment[binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z], persisting Hydrant[FireHydrant{index=io.druid.segment.incremental.OnheapIncrementalIndex@694dfbc3, queryable=io.druid.segment.ReferenceCountingSegment@3cd05253, count=26}]
2016-11-08T15:53:43,582 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - Starting persist for interval[2016-11-02T00:00:00.000Z/2016-11-03T00:00:00.000Z], rows[9,930]
2016-11-08T15:53:44,398 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - outDir[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/26/v8-tmp] completed index.drd in 0 millis.
2016-11-08T15:53:44,398 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[system] conversions with cardinality[1] in 0 millis.
2016-11-08T15:53:44,399 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[ctrlMac] conversions with cardinality[2] in 0 millis.
2016-11-08T15:53:44,399 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[ctrlName] conversions with cardinality[2] in 0 millis.
2016-11-08T15:53:44,399 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[ctrlSerial] conversions with cardinality[1] in 0 millis.
2016-11-08T15:53:44,400 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[ctrlModel] conversions with cardinality[1] in 0 millis.
2016-11-08T15:53:44,400 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[domains] conversions with cardinality[4] in 0 millis.
2016-11-08T15:53:44,400 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[zoneName] conversions with cardinality[1] in 0 millis.
2016-11-08T15:53:44,401 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apGroupName] conversions with cardinality[23] in 0 millis.
2016-11-08T15:53:44,401 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apMac] conversions with cardinality[620] in 0 millis.
2016-11-08T15:53:44,402 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apName] conversions with cardinality[620] in 0 millis.
2016-11-08T15:53:44,403 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apSerial] conversions with cardinality[620] in 1 millis.
2016-11-08T15:53:44,403 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apModel] conversions with cardinality[4] in 0 millis.
2016-11-08T15:53:44,403 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apLocation] conversions with cardinality[3] in 0 millis.
2016-11-08T15:53:44,404 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apDescription] conversions with cardinality[2] in 0 millis.
2016-11-08T15:53:44,404 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apIp] conversions with cardinality[620] in 0 millis.
2016-11-08T15:53:44,405 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[apExtIp] conversions with cardinality[13] in 0 millis.
2016-11-08T15:53:44,405 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[ssid] conversions with cardinality[1] in 0 millis.
2016-11-08T15:53:44,405 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[radio] conversions with cardinality[2] in 0 millis.
2016-11-08T15:53:44,409 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[clientMac] conversions with cardinality[4,766] in 3 millis.
2016-11-08T15:53:44,412 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[clientIp] conversions with cardinality[4,601] in 3 millis.
2016-11-08T15:53:44,414 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[username] conversions with cardinality[2,446] in 2 millis.
2016-11-08T15:53:44,416 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[hostname] conversions with cardinality[3,364] in 2 millis.
2016-11-08T15:53:44,423 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[sessionId] conversions with cardinality[9,930] in 7 millis.
2016-11-08T15:53:45,185 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[osType] conversions with cardinality[12] in 762 millis.
2016-11-08T15:53:45,185 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[manufacturer] conversions with cardinality[29] in 0 millis.
2016-11-08T15:53:45,186 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerV9 - Completed dim[sessionType] conversions with cardinality[2] in 0 millis.
2016-11-08T15:53:45,186 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - outDir[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/26/v8-tmp] completed dim conversions in 788 millis.
2016-11-08T15:53:46,076 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - outDir[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/26/v8-tmp] completed walk through of 9,930 rows in 890 millis.
2016-11-08T15:53:46,076 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[system] with cardinality[1]
2016-11-08T15:53:46,077 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[system] in 1 millis.
2016-11-08T15:53:46,077 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[ctrlMac] with cardinality[2]
2016-11-08T15:53:46,078 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[ctrlMac] in 1 millis.
2016-11-08T15:53:46,078 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[ctrlName] with cardinality[2]
2016-11-08T15:53:46,079 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[ctrlName] in 1 millis.
2016-11-08T15:53:46,079 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[ctrlSerial] with cardinality[1]
2016-11-08T15:53:46,080 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[ctrlSerial] in 1 millis.
2016-11-08T15:53:46,080 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[ctrlModel] with cardinality[1]
2016-11-08T15:53:46,081 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[ctrlModel] in 1 millis.
2016-11-08T15:53:46,081 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[domains] with cardinality[4]
2016-11-08T15:53:46,084 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[domains] in 3 millis.
2016-11-08T15:53:46,084 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[zoneName] with cardinality[1]
2016-11-08T15:53:46,085 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[zoneName] in 1 millis.
2016-11-08T15:53:46,085 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apGroupName] with cardinality[23]
2016-11-08T15:53:46,086 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apGroupName] in 1 millis.
2016-11-08T15:53:46,086 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apMac] with cardinality[620]
2016-11-08T15:53:46,088 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apMac] in 2 millis.
2016-11-08T15:53:46,088 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apName] with cardinality[620]
2016-11-08T15:53:46,089 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apName] in 1 millis.
2016-11-08T15:53:46,090 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apSerial] with cardinality[620]
2016-11-08T15:53:46,091 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apSerial] in 1 millis.
2016-11-08T15:53:46,091 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apModel] with cardinality[4]
2016-11-08T15:53:46,092 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apModel] in 1 millis.
2016-11-08T15:53:46,093 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apLocation] with cardinality[3]
2016-11-08T15:53:46,093 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apLocation] in 1 millis.
2016-11-08T15:53:46,094 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apDescription] with cardinality[2]
2016-11-08T15:53:46,094 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apDescription] in 0 millis.
2016-11-08T15:53:46,095 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apIp] with cardinality[620]
2016-11-08T15:53:46,096 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apIp] in 1 millis.
2016-11-08T15:53:46,096 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[apExtIp] with cardinality[13]
2016-11-08T15:53:46,097 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[apExtIp] in 1 millis.
2016-11-08T15:53:46,097 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[ssid] with cardinality[1]
2016-11-08T15:53:46,098 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[ssid] in 1 millis.
2016-11-08T15:53:46,098 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[radio] with cardinality[2]
2016-11-08T15:53:46,099 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[radio] in 1 millis.
2016-11-08T15:53:46,100 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[clientMac] with cardinality[4,766]
2016-11-08T15:53:46,104 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[clientMac] in 4 millis.
2016-11-08T15:53:46,104 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[clientIp] with cardinality[4,601]
2016-11-08T15:53:46,108 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[clientIp] in 4 millis.
2016-11-08T15:53:46,109 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[username] with cardinality[2,446]
2016-11-08T15:53:46,111 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[username] in 3 millis.
2016-11-08T15:53:46,111 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[hostname] with cardinality[3,364]
2016-11-08T15:53:46,114 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[hostname] in 3 millis.
2016-11-08T15:53:46,115 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[sessionId] with cardinality[9,930]
2016-11-08T15:53:46,121 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[sessionId] in 6 millis.
2016-11-08T15:53:46,122 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[osType] with cardinality[12]
2016-11-08T15:53:46,123 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[osType] in 1 millis.
2016-11-08T15:53:46,123 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[manufacturer] with cardinality[29]
2016-11-08T15:53:46,124 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[manufacturer] in 1 millis.
2016-11-08T15:53:46,125 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Starting dimension[sessionType] with cardinality[2]
2016-11-08T15:53:46,125 INFO [binnedSessions-incremental-persist] io.druid.segment.StringDimensionMergerLegacy - Completed dimension[sessionType] in 0 millis.
2016-11-08T15:53:46,126 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - outDir[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/26/v8-tmp] completed inverted.drd and wrote dimensions in 50 millis.
2016-11-08T15:53:46,126 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexMerger - wrote metadata.drd in outDir[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/26/v8-tmp].
2016-11-08T15:53:46,130 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Converting v8[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/26/v8-tmp] to v9[var/druid/task/index_kafka_binnedSessions_3aa45483b4be612_iehphlgk/work/persist/binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z/26]
2016-11-08T15:53:46,131 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apDescription.drd]
2016-11-08T15:53:46,131 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apDescription] is single value, converting...
2016-11-08T15:53:46,133 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apExtIp.drd]
2016-11-08T15:53:46,133 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apExtIp] is single value, converting...
2016-11-08T15:53:46,135 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apGroupName.drd]
2016-11-08T15:53:46,135 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apGroupName] is single value, converting...
2016-11-08T15:53:46,136 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apIp.drd]
2016-11-08T15:53:46,137 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apIp] is single value, converting...
2016-11-08T15:53:46,137 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apLocation.drd]
2016-11-08T15:53:46,138 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apLocation] is single value, converting...
2016-11-08T15:53:46,139 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apMac.drd]
2016-11-08T15:53:46,140 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apMac] is single value, converting...
2016-11-08T15:53:46,140 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apModel.drd]
2016-11-08T15:53:46,151 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apModel] is single value, converting...
2016-11-08T15:53:46,154 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apName.drd]
2016-11-08T15:53:46,154 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apName] is single value, converting...
2016-11-08T15:53:46,155 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_apSerial.drd]
2016-11-08T15:53:46,155 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[apSerial] is single value, converting...
2016-11-08T15:53:46,156 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_clientIp.drd]
2016-11-08T15:53:46,156 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[clientIp] is single value, converting...
2016-11-08T15:53:46,157 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_clientMac.drd]
2016-11-08T15:53:46,157 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[clientMac] is single value, converting...
2016-11-08T15:53:46,158 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_ctrlMac.drd]
2016-11-08T15:53:46,159 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[ctrlMac] is single value, converting...
2016-11-08T15:53:46,161 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_ctrlModel.drd]
2016-11-08T15:53:46,162 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[ctrlModel] is single value, converting...
2016-11-08T15:53:46,162 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_ctrlName.drd]
2016-11-08T15:53:46,162 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[ctrlName] is single value, converting...
2016-11-08T15:53:46,164 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_ctrlSerial.drd]
2016-11-08T15:53:46,164 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[ctrlSerial] is single value, converting...
2016-11-08T15:53:46,165 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_domains.drd]
2016-11-08T15:53:46,167 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_hostname.drd]
2016-11-08T15:53:46,168 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[hostname] is single value, converting...
2016-11-08T15:53:46,169 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_manufacturer.drd]
2016-11-08T15:53:46,169 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[manufacturer] is single value, converting...
2016-11-08T15:53:46,170 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_osType.drd]
2016-11-08T15:53:46,171 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[osType] is single value, converting...
2016-11-08T15:53:46,172 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_radio.drd]
2016-11-08T15:53:46,172 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[radio] is single value, converting...
2016-11-08T15:53:46,174 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_sessionId.drd]
2016-11-08T15:53:46,174 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[sessionId] is single value, converting...
2016-11-08T15:53:46,176 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_sessionType.drd]
2016-11-08T15:53:46,176 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[sessionType] is single value, converting...
2016-11-08T15:53:46,179 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_ssid.drd]
2016-11-08T15:53:46,180 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[ssid] is single value, converting...
2016-11-08T15:53:46,180 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_system.drd]
2016-11-08T15:53:46,180 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[system] is single value, converting...
2016-11-08T15:53:46,181 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_username.drd]
2016-11-08T15:53:46,181 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[username] is single value, converting...
2016-11-08T15:53:46,182 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[dim_zoneName.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Dimension[zoneName] is single value, converting...
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[index.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[inverted.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_firstRss_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_firstSnr_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_lastRss_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_lastSnr_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_maxRss_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_maxSnr_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_minRss_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_minSnr_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_noiseFloor_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_rxBytes_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,183 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_throughputEstimate_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,184 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_traffic_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,184 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[met_txBytes_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,184 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[metadata.drd]
2016-11-08T15:53:46,184 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[spatial.drd]
2016-11-08T15:53:46,184 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Processing file[time_LITTLE_ENDIAN.drd]
2016-11-08T15:53:46,184 INFO [binnedSessions-incremental-persist] io.druid.segment.IndexIO$DefaultIndexIOHandler - Skipped files[[index.drd, inverted.drd, metadata.drd, spatial.drd]]
2016-11-08T15:53:46,191 INFO [binnedSessions-incremental-persist] io.druid.segment.ReferenceCountingSegment - Closing binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z
2016-11-08T15:53:46,191 INFO [binnedSessions-incremental-persist] io.druid.segment.ReferenceCountingSegment - Closing binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z, numReferences: 0
2016-11-08T15:53:46,192 INFO [binnedSessions-incremental-persist] io.druid.segment.realtime.appenderator.AppenderatorImpl - Committing metadata[FiniteAppenderatorDriverMetadata{activeSegments={index_kafka_binnedSessions_3aa45483b4be612_0=[binnedSessions_2016-06-20T00:00:00.000Z_2016-06-21T00:00:00.000Z_2016-11-08T10:09:13.263Z, binnedSessions_2016-06-21T00:00:00.000Z_2016-06-22T00:00:00.000Z_2016-11-08T10:09:27.747Z, binnedSessions_2016-06-22T00:00:00.000Z_2016-06-23T00:00:00.000Z_2016-11-08T10:11:18.466Z, binnedSessions_2016-06-23T00:00:00.000Z_2016-06-24T00:00:00.000Z_2016-11-08T10:14:53.766Z, binnedSessions_2016-06-24T00:00:00.000Z_2016-06-25T00:00:00.000Z_2016-11-08T10:14:09.400Z, binnedSessions_2016-06-25T00:00:00.000Z_2016-06-26T00:00:00.000Z_2016-11-08T10:19:25.906Z, binnedSessions_2016-06-26T00:00:00.000Z_2016-06-27T00:00:00.000Z_2016-11-08T10:18:03.604Z, binnedSessions_2016-06-27T00:00:00.000Z_2016-06-28T00:00:00.000Z_2016-11-08T10:20:23.175Z, binnedSessions_2016-06-28T00:00:00.000Z_2016-06-29T00:00:00.000Z_2016-11-08T10:22:54.736Z, binnedSessions_2016-06-29T00:00:00.000Z_2016-06-30T00:00:00.000Z_2016-11-08T10:25:17.128Z, binnedSessions_2016-06-30T00:00:00.000Z_2016-07-01T00:00:00.000Z_2016-11-08T10:27:47.942Z, binnedSessions_2016-07-01T00:00:00.000Z_2016-07-02T00:00:00.000Z_2016-11-08T10:31:06.628Z, binnedSessions_2016-07-02T00:00:00.000Z_2016-07-03T00:00:00.000Z_2016-11-08T10:32:29.081Z, binnedSessions_2016-07-03T00:00:00.000Z_2016-07-04T00:00:00.000Z_2016-11-08T10:32:13.446Z, binnedSessions_2016-07-04T00:00:00.000Z_2016-07-05T00:00:00.000Z_2016-11-08T10:33:00.954Z, binnedSessions_2016-07-05T00:00:00.000Z_2016-07-06T00:00:00.000Z_2016-11-08T10:33:49.484Z, binnedSessions_2016-07-06T00:00:00.000Z_2016-07-07T00:00:00.000Z_2016-11-08T10:35:03.291Z, binnedSessions_2016-07-07T00:00:00.000Z_2016-07-08T00:00:00.000Z_2016-11-08T10:35:22.023Z, binnedSessions_2016-07-08T00:00:00.000Z_2016-07-09T00:00:00.000Z_2016-11-08T10:35:59.888Z, binnedSessions_2016-07-09T00:00:00.000Z_2016-07-10T00:00:00.000Z_2016-11-08T10:36:51.145Z, binnedSessions_2016-07-10T00:00:00.000Z_2016-07-11T00:00:00.000Z_2016-11-08T10:38:04.739Z, binnedSessions_2016-07-11T00:00:00.000Z_2016-07-12T00:00:00.000Z_2016-11-08T10:39:29.662Z, binnedSessions_2016-07-12T00:00:00.000Z_2016-07-13T00:00:00.000Z_2016-11-08T10:40:22.950Z, binnedSessions_2016-07-13T00:00:00.000Z_2016-07-14T00:00:00.000Z_2016-11-08T10:42:02.651Z, binnedSessions_2016-07-14T00:00:00.000Z_2016-07-15T00:00:00.000Z_2016-11-08T10:43:04.902Z, binnedSessions_2016-07-15T00:00:00.000Z_2016-07-16T00:00:00.000Z_2016-11-08T10:44:35.665Z, binnedSessions_2016-07-16T00:00:00.000Z_2016-07-17T00:00:00.000Z_2016-11-08T10:45:49.778Z, binnedSessions_2016-07-17T00:00:00.000Z_2016-07-18T00:00:00.000Z_2016-11-08T10:47:38.203Z, binnedSessions_2016-07-18T00:00:00.000Z_2016-07-19T00:00:00.000Z_2016-11-08T10:48:19.393Z, binnedSessions_2016-07-19T00:00:00.000Z_2016-07-20T00:00:00.000Z_2016-11-08T10:49:58.959Z, binnedSessions_2016-07-20T00:00:00.000Z_2016-07-21T00:00:00.000Z_2016-11-08T10:51:41.400Z, binnedSessions_2016-07-21T00:00:00.000Z_2016-07-22T00:00:00.000Z_2016-11-08T10:53:09.966Z, binnedSessions_2016-07-22T00:00:00.000Z_2016-07-23T00:00:00.000Z_2016-11-08T10:54:15.948Z, binnedSessions_2016-07-23T00:00:00.000Z_2016-07-24T00:00:00.000Z_2016-11-08T10:55:41.960Z, binnedSessions_2016-07-24T00:00:00.000Z_2016-07-25T00:00:00.000Z_2016-11-08T10:56:41.041Z, binnedSessions_2016-07-25T00:00:00.000Z_2016-07-26T00:00:00.000Z_2016-11-08T10:58:29.571Z, binnedSessions_2016-07-26T00:00:00.000Z_2016-07-27T00:00:00.000Z_2016-11-08T10:59:29.907Z, binnedSessions_2016-07-27T00:00:00.000Z_2016-07-28T00:00:00.000Z_2016-11-08T11:01:08.815Z, binnedSessions_2016-07-28T00:00:00.000Z_2016-07-29T00:00:00.000Z_2016-11-08T11:02:41.680Z, binnedSessions_2016-07-29T00:00:00.000Z_2016-07-30T00:00:00.000Z_2016-11-08T11:04:55.338Z, binnedSessions_2016-07-30T00:00:00.000Z_2016-07-31T00:00:00.000Z_2016-11-08T11:05:28.321Z, binnedSessions_2016-07-31T00:00:00.000Z_2016-08-01T00:00:00.000Z_2016-11-08T11:07:08.339Z, binnedSessions_2016-08-01T00:00:00.000Z_2016-08-02T00:00:00.000Z_2016-11-08T11:08:09.045Z, binnedSessions_2016-08-02T00:00:00.000Z_2016-08-03T00:00:00.000Z_2016-11-08T11:10:15.210Z, binnedSessions_2016-08-03T00:00:00.000Z_2016-08-04T00:00:00.000Z_2016-11-08T11:11:42.942Z, binnedSessions_2016-08-04T00:00:00.000Z_2016-08-05T00:00:00.000Z_2016-11-08T11:13:33.953Z, binnedSessions_2016-08-05T00:00:00.000Z_2016-08-06T00:00:00.000Z_2016-11-08T11:15:48.289Z, binnedSessions_2016-08-06T00:00:00.000Z_2016-08-07T00:00:00.000Z_2016-11-08T11:16:43.316Z, binnedSessions_2016-08-07T00:00:00.000Z_2016-08-08T00:00:00.000Z_2016-11-08T11:18:03.936Z, binnedSessions_2016-08-08T00:00:00.000Z_2016-08-09T00:00:00.000Z_2016-11-08T11:19:28.834Z, binnedSessions_2016-08-09T00:00:00.000Z_2016-08-10T00:00:00.000Z_2016-11-08T11:20:55.666Z, binnedSessions_2016-08-10T00:00:00.000Z_2016-08-11T00:00:00.000Z_2016-11-08T09:08:30.650Z, binnedSessions_2016-08-11T00:00:00.000Z_2016-08-12T00:00:00.000Z_2016-11-08T09:08:35.017Z, binnedSessions_2016-08-12T00:00:00.000Z_2016-08-13T00:00:00.000Z_2016-11-08T09:08:45.879Z, binnedSessions_2016-08-13T00:00:00.000Z_2016-08-14T00:00:00.000Z_2016-11-08T09:09:02.675Z, binnedSessions_2016-08-14T00:00:00.000Z_2016-08-15T00:00:00.000Z_2016-11-08T09:09:12.818Z, binnedSessions_2016-08-15T00:00:00.000Z_2016-08-16T00:00:00.000Z_2016-11-08T09:09:20.968Z, binnedSessions_2016-08-16T00:00:00.000Z_2016-08-17T00:00:00.000Z_2016-11-08T09:09:36.313Z, binnedSessions_2016-08-17T00:00:00.000Z_2016-08-18T00:00:00.000Z_2016-11-08T09:09:45.037Z, binnedSessions_2016-08-18T00:00:00.000Z_2016-08-19T00:00:00.000Z_2016-11-08T09:10:02.353Z, binnedSessions_2016-08-19T00:00:00.000Z_2016-08-20T00:00:00.000Z_2016-11-08T09:10:17.536Z, binnedSessions_2016-08-20T00:00:00.000Z_2016-08-21T00:00:00.000Z_2016-11-08T09:10:34.559Z, binnedSessions_2016-08-21T00:00:00.000Z_2016-08-22T00:00:00.000Z_2016-11-08T09:10:48.819Z, binnedSessions_2016-08-22T00:00:00.000Z_2016-08-23T00:00:00.000Z_2016-11-08T09:11:03.214Z, binnedSessions_2016-08-23T00:00:00.000Z_2016-08-24T00:00:00.000Z_2016-11-08T09:11:07.551Z, binnedSessions_2016-08-24T00:00:00.000Z_2016-08-25T00:00:00.000Z_2016-11-08T09:11:14.105Z, binnedSessions_2016-08-25T00:00:00.000Z_2016-08-26T00:00:00.000Z_2016-11-08T09:11:28.552Z, binnedSessions_2016-08-26T00:00:00.000Z_2016-08-27T00:00:00.000Z_2016-11-08T09:11:46.259Z, binnedSessions_2016-08-27T00:00:00.000Z_2016-08-28T00:00:00.000Z_2016-11-08T09:11:58.799Z, binnedSessions_2016-08-29T00:00:00.000Z_2016-08-30T00:00:00.000Z_2016-11-08T09:12:20.767Z, binnedSessions_2016-08-30T00:00:00.000Z_2016-08-31T00:00:00.000Z_2016-11-08T09:12:25.186Z, binnedSessions_2016-08-31T00:00:00.000Z_2016-09-01T00:00:00.000Z_2016-11-08T09:12:26.445Z, binnedSessions_2016-09-01T00:00:00.000Z_2016-09-02T00:00:00.000Z_2016-11-08T09:12:44.632Z, binnedSessions_2016-09-02T00:00:00.000Z_2016-09-03T00:00:00.000Z_2016-11-08T09:13:02.260Z, binnedSessions_2016-09-03T00:00:00.000Z_2016-09-04T00:00:00.000Z_2016-11-08T09:13:17.177Z, binnedSessions_2016-09-04T00:00:00.000Z_2016-09-05T00:00:00.000Z_2016-11-08T09:13:34.045Z, binnedSessions_2016-09-05T00:00:00.000Z_2016-09-06T00:00:00.000Z_2016-11-08T09:13:51.338Z, binnedSessions_2016-09-06T00:00:00.000Z_2016-09-07T00:00:00.000Z_2016-11-08T09:14:07.289Z, binnedSessions_2016-09-07T00:00:00.000Z_2016-09-08T00:00:00.000Z_2016-11-08T09:14:14.866Z, binnedSessions_2016-09-08T00:00:00.000Z_2016-09-09T00:00:00.000Z_2016-11-08T09:14:37.991Z, binnedSessions_2016-09-09T00:00:00.000Z_2016-09-10T00:00:00.000Z_2016-11-08T09:14:41.387Z, binnedSessions_2016-09-10T00:00:00.000Z_2016-09-11T00:00:00.000Z_2016-11-08T09:14:55.122Z, binnedSessions_2016-09-11T00:00:00.000Z_2016-09-12T00:00:00.000Z_2016-11-08T09:15:09.722Z, binnedSessions_2016-09-12T00:00:00.000Z_2016-09-13T00:00:00.000Z_2016-11-08T09:15:19.273Z, binnedSessions_2016-09-13T00:00:00.000Z_2016-09-14T00:00:00.000Z_2016-11-08T09:15:36.571Z, binnedSessions_2016-09-14T00:00:00.000Z_2016-09-15T00:00:00.000Z_2016-11-08T09:15:42.735Z, binnedSessions_2016-09-15T00:00:00.000Z_2016-09-16T00:00:00.000Z_2016-11-08T09:15:59.169Z, binnedSessions_2016-09-16T00:00:00.000Z_2016-09-17T00:00:00.000Z_2016-11-08T09:16:16.924Z, binnedSessions_2016-09-17T00:00:00.000Z_2016-09-18T00:00:00.000Z_2016-11-08T09:16:23.787Z, binnedSessions_2016-09-18T00:00:00.000Z_2016-09-19T00:00:00.000Z_2016-11-08T09:16:49.914Z, binnedSessions_2016-09-19T00:00:00.000Z_2016-09-20T00:00:00.000Z_2016-11-08T09:17:00.222Z, binnedSessions_2016-09-20T00:00:00.000Z_2016-09-21T00:00:00.000Z_2016-11-08T09:17:01.413Z, binnedSessions_2016-09-21T00:00:00.000Z_2016-09-22T00:00:00.000Z_2016-11-08T09:17:20.852Z, binnedSessions_2016-09-22T00:00:00.000Z_2016-09-23T00:00:00.000Z_2016-11-08T09:17:51.767Z, binnedSessions_2016-09-23T00:00:00.000Z_2016-09-24T00:00:00.000Z_2016-11-08T09:18:13.896Z, binnedSessions_2016-09-24T00:00:00.000Z_2016-09-25T00:00:00.000Z_2016-11-08T09:18:32.770Z, binnedSessions_2016-09-25T00:00:00.000Z_2016-09-26T00:00:00.000Z_2016-11-08T09:18:14.042Z, binnedSessions_2016-09-26T00:00:00.000Z_2016-09-27T00:00:00.000Z_2016-11-08T09:18:32.598Z, binnedSessions_2016-09-27T00:00:00.000Z_2016-09-28T00:00:00.000Z_2016-11-08T09:19:18.833Z, binnedSessions_2016-09-28T00:00:00.000Z_2016-09-29T00:00:00.000Z_2016-11-08T09:18:58.416Z, binnedSessions_2016-09-29T00:00:00.000Z_2016-09-30T00:00:00.000Z_2016-11-08T09:19:15.564Z, binnedSessions_2016-09-30T00:00:00.000Z_2016-10-01T00:00:00.000Z_2016-11-08T09:19:33.831Z, binnedSessions_2016-10-01T00:00:00.000Z_2016-10-02T00:00:00.000Z_2016-11-08T09:19:48.092Z, binnedSessions_2016-10-02T00:00:00.000Z_2016-10-03T00:00:00.000Z_2016-11-08T09:20:08.136Z, binnedSessions_2016-10-03T00:00:00.000Z_2016-10-04T00:00:00.000Z_2016-11-08T09:20:25.659Z, binnedSessions_2016-10-04T00:00:00.000Z_2016-10-05T00:00:00.000Z_2016-11-08T09:20:51.278Z, binnedSessions_2016-10-05T00:00:00.000Z_2016-10-06T00:00:00.000Z_2016-11-08T09:21:09.985Z, binnedSessions_2016-10-06T00:00:00.000Z_2016-10-07T00:00:00.000Z_2016-11-08T09:21:31.287Z, binnedSessions_2016-10-07T00:00:00.000Z_2016-10-08T00:00:00.000Z_2016-11-08T09:21:53.922Z, binnedSessions_2016-10-08T00:00:00.000Z_2016-10-09T00:00:00.000Z_2016-11-08T09:22:08.140Z, binnedSessions_2016-10-09T00:00:00.000Z_2016-10-10T00:00:00.000Z_2016-11-08T09:22:24.187Z, binnedSessions_2016-10-10T00:00:00.000Z_2016-10-11T00:00:00.000Z_2016-11-08T09:22:45.399Z, binnedSessions_2016-10-11T00:00:00.000Z_2016-10-12T00:00:00.000Z_2016-11-08T09:23:07.509Z, binnedSessions_2016-10-12T00:00:00.000Z_2016-10-13T00:00:00.000Z_2016-11-08T09:23:22.541Z, binnedSessions_2016-10-13T00:00:00.000Z_2016-10-14T00:00:00.000Z_2016-11-08T09:23:48.505Z, binnedSessions_2016-10-14T00:00:00.000Z_2016-10-15T00:00:00.000Z_2016-11-08T09:23:55.532Z, binnedSessions_2016-10-15T00:00:00.000Z_2016-10-16T00:00:00.000Z_2016-11-08T09:24:10.238Z, binnedSessions_2016-10-16T00:00:00.000Z_2016-10-17T00:00:00.000Z_2016-11-08T09:24:13.499Z, binnedSessions_2016-10-17T00:00:00.000Z_2016-10-18T00:00:00.000Z_2016-11-08T09:24:15.951Z, binnedSessions_2016-10-18T00:00:00.000Z_2016-10-19T00:00:00.000Z_2016-11-08T09:24:32.574Z, binnedSessions_2016-10-19T00:00:00.000Z_2016-10-20T00:00:00.000Z_2016-11-08T09:24:54.849Z, binnedSessions_2016-10-20T00:00:00.000Z_2016-10-21T00:00:00.000Z_2016-11-08T09:25:06.222Z, binnedSessions_2016-10-21T00:00:00.000Z_2016-10-22T00:00:00.000Z_2016-11-08T09:25:28.585Z, binnedSessions_2016-10-22T00:00:00.000Z_2016-10-23T00:00:00.000Z_2016-11-08T09:25:53.404Z, binnedSessions_2016-10-23T00:00:00.000Z_2016-10-24T00:00:00.000Z_2016-11-08T09:26:10.238Z, binnedSessions_2016-10-24T00:00:00.000Z_2016-10-25T00:00:00.000Z_2016-11-08T09:26:22.664Z, binnedSessions_2016-10-25T00:00:00.000Z_2016-10-26T00:00:00.000Z_2016-11-08T09:26:53.414Z, binnedSessions_2016-10-26T00:00:00.000Z_2016-10-27T00:00:00.000Z_2016-11-08T09:27:04.394Z, binnedSessions_2016-10-27T00:00:00.000Z_2016-10-28T00:00:00.000Z_2016-11-08T09:27:40.465Z, binnedSessions_2016-10-28T00:00:00.000Z_2016-10-29T00:00:00.000Z_2016-11-08T09:27:24.584Z, binnedSessions_2016-10-29T00:00:00.000Z_2016-10-30T00:00:00.000Z_2016-11-08T09:27:45.887Z, binnedSessions_2016-10-30T00:00:00.000Z_2016-10-31T00:00:00.000Z_2016-11-08T09:28:18.105Z, binnedSessions_2016-10-31T00:00:00.000Z_2016-11-01T00:00:00.000Z_2016-11-08T09:28:28.774Z, binnedSessions_2016-11-01T00:00:00.000Z_2016-11-02T00:00:00.000Z_2016-11-08T09:28:40.571Z, binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z, binnedSessions_2016-11-03T00:00:00.000Z_2016-11-04T00:00:00.000Z_2016-11-03T10:23:07.264Z_1, binnedSessions_2016-11-04T00:00:00.000Z_2016-11-05T00:00:00.000Z_2016-11-04T00:22:15.105Z_3, binnedSessions_2016-11-05T00:00:00.000Z_2016-11-06T00:00:00.000Z_2016-11-05T00:21:25.466Z_2, binnedSessions_2016-11-06T00:00:00.000Z_2016-11-07T00:00:00.000Z_2016-11-06T00:21:25.492Z_1, binnedSessions_2016-11-07T00:00:00.000Z_2016-11-08T00:00:00.000Z_2016-11-07T00:21:24.892Z_2, binnedSessions_2016-11-08T00:00:00.000Z_2016-11-09T00:00:00.000Z_2016-11-08T00:22:16.022Z_1]}, lastSegmentIds={index_kafka_binnedSessions_3aa45483b4be612_0=binnedSessions_2016-08-09T00:00:00.000Z_2016-08-10T00:00:00.000Z_2016-11-08T11:20:55.666Z}, callerMetadata={nextPartitions=KafkaPartitions{topic='binnedSessions', partitionOffsetMap={0=61567755}}}}] for sinks[binnedSessions_2016-08-19T00:00:00.000Z_2016-08-20T00:00:00.000Z_2016-11-08T09:10:17.536Z:26, binnedSessions_2016-10-26T00:00:00.000Z_2016-10-27T00:00:00.000Z_2016-11-08T09:27:04.394Z:35, binnedSessions_2016-08-30T00:00:00.000Z_2016-08-31T00:00:00.000Z_2016-11-08T09:12:25.186Z:44, binnedSessions_2016-10-11T00:00:00.000Z_2016-10-12T00:00:00.000Z_2016-11-08T09:23:07.509Z:80, binnedSessions_2016-08-06T00:00:00.000Z_2016-08-07T00:00:00.000Z_2016-11-08T11:16:43.316Z:15, binnedSessions_2016-10-24T00:00:00.000Z_2016-10-25T00:00:00.000Z_2016-11-08T09:26:22.664Z:34, binnedSessions_2016-06-22T00:00:00.000Z_2016-06-23T00:00:00.000Z_2016-11-08T10:11:18.466Z:63, binnedSessions_2016-11-04T00:00:00.000Z_2016-11-05T00:00:00.000Z_2016-11-04T00:22:15.105Z_3:7, binnedSessions_2016-10-18T00:00:00.000Z_2016-10-19T00:00:00.000Z_2016-11-08T09:24:32.574Z:83, binnedSessions_2016-10-27T00:00:00.000Z_2016-10-28T00:00:00.000Z_2016-11-08T09:27:40.465Z:38, binnedSessions_2016-09-02T00:00:00.000Z_2016-09-03T00:00:00.000Z_2016-11-08T09:13:02.260Z:44, binnedSessions_2016-11-08T00:00:00.000Z_2016-11-09T00:00:00.000Z_2016-11-08T00:22:16.022Z_1:121, binnedSessions_2016-09-24T00:00:00.000Z_2016-09-25T00:00:00.000Z_2016-11-08T09:18:32.770Z:68, binnedSessions_2016-09-20T00:00:00.000Z_2016-09-21T00:00:00.000Z_2016-11-08T09:17:01.413Z:88, binnedSessions_2016-08-26T00:00:00.000Z_2016-08-27T00:00:00.000Z_2016-11-08T09:11:46.259Z:25, binnedSessions_2016-07-10T00:00:00.000Z_2016-07-11T00:00:00.000Z_2016-11-08T10:38:04.739Z:17, binnedSessions_2016-09-03T00:00:00.000Z_2016-09-04T00:00:00.000Z_2016-11-08T09:13:17.177Z:41, binnedSessions_2016-08-12T00:00:00.000Z_2016-08-13T00:00:00.000Z_2016-11-08T09:08:45.879Z:21, binnedSessions_2016-11-01T00:00:00.000Z_2016-11-02T00:00:00.000Z_2016-11-08T09:28:40.571Z:26, binnedSessions_2016-10-10T00:00:00.000Z_2016-10-11T00:00:00.000Z_2016-11-08T09:22:45.399Z:87, binnedSessions_2016-06-29T00:00:00.000Z_2016-06-30T00:00:00.000Z_2016-11-08T10:25:17.128Z:49, binnedSessions_2016-09-01T00:00:00.000Z_2016-09-02T00:00:00.000Z_2016-11-08T09:12:44.632Z:39, binnedSessions_2016-09-05T00:00:00.000Z_2016-09-06T00:00:00.000Z_2016-11-08T09:13:51.338Z:69, binnedSessions_2016-08-16T00:00:00.000Z_2016-08-17T00:00:00.000Z_2016-11-08T09:09:36.313Z:28, binnedSessions_2016-06-30T00:00:00.000Z_2016-07-01T00:00:00.000Z_2016-11-08T10:27:47.942Z:30, binnedSessions_2016-07-03T00:00:00.000Z_2016-07-04T00:00:00.000Z_2016-11-08T10:32:13.446Z:11, binnedSessions_2016-08-27T00:00:00.000Z_2016-08-28T00:00:00.000Z_2016-11-08T09:11:58.799Z:4, binnedSessions_2016-10-04T00:00:00.000Z_2016-10-05T00:00:00.000Z_2016-11-08T09:20:51.278Z:66, binnedSessions_2016-07-30T00:00:00.000Z_2016-07-31T00:00:00.000Z_2016-11-08T11:05:28.321Z:17, binnedSessions_2016-10-13T00:00:00.000Z_2016-10-14T00:00:00.000Z_2016-11-08T09:23:48.505Z:78, binnedSessions_2016-08-29T00:00:00.000Z_2016-08-30T00:00:00.000Z_2016-11-08T09:12:20.767Z:34, binnedSessions_2016-06-23T00:00:00.000Z_2016-06-24T00:00:00.000Z_2016-11-08T10:14:53.766Z:75, binnedSessions_2016-09-27T00:00:00.000Z_2016-09-28T00:00:00.000Z_2016-11-08T09:19:18.833Z:73, binnedSessions_2016-09-11T00:00:00.000Z_2016-09-12T00:00:00.000Z_2016-11-08T09:15:09.722Z:25, binnedSessions_2016-08-05T00:00:00.000Z_2016-08-06T00:00:00.000Z_2016-11-08T11:15:48.289Z:22, binnedSessions_2016-09-21T00:00:00.000Z_2016-09-22T00:00:00.000Z_2016-11-08T09:17:20.852Z:99, binnedSessions_2016-10-19T00:00:00.000Z_2016-10-20T00:00:00.000Z_2016-11-08T09:24:54.849Z:84, binnedSessions_2016-08-11T00:00:00.000Z_2016-08-12T00:00:00.000Z_2016-11-08T09:08:35.017Z:25, binnedSessions_2016-09-22T00:00:00.000Z_2016-09-23T00:00:00.000Z_2016-11-08T09:17:51.767Z:78, binnedSessions_2016-10-31T00:00:00.000Z_2016-11-01T00:00:00.000Z_2016-11-08T09:28:28.774Z:48, binnedSessions_2016-09-30T00:00:00.000Z_2016-10-01T00:00:00.000Z_2016-11-08T09:19:33.831Z:61, binnedSessions_2016-08-07T00:00:00.000Z_2016-08-08T00:00:00.000Z_2016-11-08T11:18:03.936Z:14, binnedSessions_2016-07-18T00:00:00.000Z_2016-07-19T00:00:00.000Z_2016-11-08T10:48:19.393Z:23, binnedSessions_2016-08-04T00:00:00.000Z_2016-08-05T00:00:00.000Z_2016-11-08T11:13:33.953Z:18, binnedSessions_2016-08-22T00:00:00.000Z_2016-08-23T00:00:00.000Z_2016-11-08T09:11:03.214Z:9, binnedSessions_2016-08-20T00:00:00.000Z_2016-08-21T00:00:00.000Z_2016-11-08T09:10:34.559Z:22, binnedSessions_2016-07-19T00:00:00.000Z_2016-07-20T00:00:00.000Z_2016-11-08T10:49:58.959Z:22, binnedSessions_2016-06-28T00:00:00.000Z_2016-06-29T00:00:00.000Z_2016-11-08T10:22:54.736Z:62, binnedSessions_2016-10-02T00:00:00.000Z_2016-10-03T00:00:00.000Z_2016-11-08T09:20:08.136Z:56, binnedSessions_2016-10-12T00:00:00.000Z_2016-10-13T00:00:00.000Z_2016-11-08T09:23:22.541Z:71, binnedSessions_2016-08-02T00:00:00.000Z_2016-08-03T00:00:00.000Z_2016-11-08T11:10:15.210Z:22, binnedSessions_2016-06-21T00:00:00.000Z_2016-06-22T00:00:00.000Z_2016-11-08T10:09:27.747Z:49, binnedSessions_2016-07-09T00:00:00.000Z_2016-07-10T00:00:00.000Z_2016-11-08T10:36:51.145Z:11, binnedSessions_2016-09-09T00:00:00.000Z_2016-09-10T00:00:00.000Z_2016-11-08T09:14:41.387Z:63, binnedSessions_2016-09-17T00:00:00.000Z_2016-09-18T00:00:00.000Z_2016-11-08T09:16:23.787Z:61, binnedSessions_2016-07-05T00:00:00.000Z_2016-07-06T00:00:00.000Z_2016-11-08T10:33:49.484Z:9, binnedSessions_2016-08-01T00:00:00.000Z_2016-08-02T00:00:00.000Z_2016-11-08T11:08:09.045Z:21, binnedSessions_2016-08-08T00:00:00.000Z_2016-08-09T00:00:00.000Z_2016-11-08T11:19:28.834Z:20, binnedSessions_2016-07-25T00:00:00.000Z_2016-07-26T00:00:00.000Z_2016-11-08T10:58:29.571Z:22, binnedSessions_2016-10-08T00:00:00.000Z_2016-10-09T00:00:00.000Z_2016-11-08T09:22:08.140Z:75, binnedSessions_2016-11-03T00:00:00.000Z_2016-11-04T00:00:00.000Z_2016-11-03T10:23:07.264Z_1:5, binnedSessions_2016-09-06T00:00:00.000Z_2016-09-07T00:00:00.000Z_2016-11-08T09:14:07.289Z:34, binnedSessions_2016-10-14T00:00:00.000Z_2016-10-15T00:00:00.000Z_2016-11-08T09:23:55.532Z:68, binnedSessions_2016-09-15T00:00:00.000Z_2016-09-16T00:00:00.000Z_2016-11-08T09:15:59.169Z:46, binnedSessions_2016-07-13T00:00:00.000Z_2016-07-14T00:00:00.000Z_2016-11-08T10:42:02.651Z:17, binnedSessions_2016-09-18T00:00:00.000Z_2016-09-19T00:00:00.000Z_2016-11-08T09:16:49.914Z:63, binnedSessions_2016-07-12T00:00:00.000Z_2016-07-13T00:00:00.000Z_2016-11-08T10:40:22.950Z:15, binnedSessions_2016-10-29T00:00:00.000Z_2016-10-30T00:00:00.000Z_2016-11-08T09:27:45.887Z:25, binnedSessions_2016-07-14T00:00:00.000Z_2016-07-15T00:00:00.000Z_2016-11-08T10:43:04.902Z:18, binnedSessions_2016-07-26T00:00:00.000Z_2016-07-27T00:00:00.000Z_2016-11-08T10:59:29.907Z:19, binnedSessions_2016-06-20T00:00:00.000Z_2016-06-21T00:00:00.000Z_2016-11-08T10:09:13.263Z:3, binnedSessions_2016-09-10T00:00:00.000Z_2016-09-11T00:00:00.000Z_2016-11-08T09:14:55.122Z:34, binnedSessions_2016-10-22T00:00:00.000Z_2016-10-23T00:00:00.000Z_2016-11-08T09:25:53.404Z:35, binnedSessions_2016-10-21T00:00:00.000Z_2016-10-22T00:00:00.000Z_2016-11-08T09:25:28.585Z:61, binnedSessions_2016-08-14T00:00:00.000Z_2016-08-15T00:00:00.000Z_2016-11-08T09:09:12.818Z:26, binnedSessions_2016-07-29T00:00:00.000Z_2016-07-30T00:00:00.000Z_2016-11-08T11:04:55.338Z:17, binnedSessions_2016-08-23T00:00:00.000Z_2016-08-24T00:00:00.000Z_2016-11-08T09:11:07.551Z:15, binnedSessions_2016-07-08T00:00:00.000Z_2016-07-09T00:00:00.000Z_2016-11-08T10:35:59.888Z:9, binnedSessions_2016-08-31T00:00:00.000Z_2016-09-01T00:00:00.000Z_2016-11-08T09:12:26.445Z:46, binnedSessions_2016-09-19T00:00:00.000Z_2016-09-20T00:00:00.000Z_2016-11-08T09:17:00.222Z:87, binnedSessions_2016-10-03T00:00:00.000Z_2016-10-04T00:00:00.000Z_2016-11-08T09:20:25.659Z:72, binnedSessions_2016-07-23T00:00:00.000Z_2016-07-24T00:00:00.000Z_2016-11-08T10:55:41.960Z:16, binnedSessions_2016-10-23T00:00:00.000Z_2016-10-24T00:00:00.000Z_2016-11-08T09:26:10.238Z:35, binnedSessions_2016-10-07T00:00:00.000Z_2016-10-08T00:00:00.000Z_2016-11-08T09:21:53.922Z:57, binnedSessions_2016-10-25T00:00:00.000Z_2016-10-26T00:00:00.000Z_2016-11-08T09:26:53.414Z:35, binnedSessions_2016-08-24T00:00:00.000Z_2016-08-25T00:00:00.000Z_2016-11-08T09:11:14.105Z:21, binnedSessions_2016-09-16T00:00:00.000Z_2016-09-17T00:00:00.000Z_2016-11-08T09:16:16.924Z:42, binnedSessions_2016-09-25T00:00:00.000Z_2016-09-26T00:00:00.000Z_2016-11-08T09:18:14.042Z:77, binnedSessions_2016-06-27T00:00:00.000Z_2016-06-28T00:00:00.000Z_2016-11-08T10:20:23.175Z:62, binnedSessions_2016-08-17T00:00:00.000Z_2016-08-18T00:00:00.000Z_2016-11-08T09:09:45.037Z:24, binnedSessions_2016-09-12T00:00:00.000Z_2016-09-13T00:00:00.000Z_2016-11-08T09:15:19.273Z:29, binnedSessions_2016-09-13T00:00:00.000Z_2016-09-14T00:00:00.000Z_2016-11-08T09:15:36.571Z:27, binnedSessions_2016-10-17T00:00:00.000Z_2016-10-18T00:00:00.000Z_2016-11-08T09:24:15.951Z:77, binnedSessions_2016-07-27T00:00:00.000Z_2016-07-28T00:00:00.000Z_2016-11-08T11:01:08.815Z:21, binnedSessions_2016-10-30T00:00:00.000Z_2016-10-31T00:00:00.000Z_2016-11-08T09:28:18.105Z:35, binnedSessions_2016-07-07T00:00:00.000Z_2016-07-08T00:00:00.000Z_2016-11-08T10:35:22.023Z:8, binnedSessions_2016-07-15T00:00:00.000Z_2016-07-16T00:00:00.000Z_2016-11-08T10:44:35.665Z:21, binnedSessions_2016-07-24T00:00:00.000Z_2016-07-25T00:00:00.000Z_2016-11-08T10:56:41.041Z:15, binnedSessions_2016-08-18T00:00:00.000Z_2016-08-19T00:00:00.000Z_2016-11-08T09:10:02.353Z:26, binnedSessions_2016-10-20T00:00:00.000Z_2016-10-21T00:00:00.000Z_2016-11-08T09:25:06.222Z:77, binnedSessions_2016-07-21T00:00:00.000Z_2016-07-22T00:00:00.000Z_2016-11-08T10:53:09.966Z:22, binnedSessions_2016-11-06T00:00:00.000Z_2016-11-07T00:00:00.000Z_2016-11-06T00:21:25.492Z_1:6, binnedSessions_2016-08-09T00:00:00.000Z_2016-08-10T00:00:00.000Z_2016-11-08T11:20:55.666Z:21, binnedSessions_2016-09-29T00:00:00.000Z_2016-09-30T00:00:00.000Z_2016-11-08T09:19:15.564Z:86, binnedSessions_2016-08-25T00:00:00.000Z_2016-08-26T00:00:00.000Z_2016-11-08T09:11:28.552Z:25, binnedSessions_2016-08-21T00:00:00.000Z_2016-08-22T00:00:00.000Z_2016-11-08T09:10:48.819Z:24, binnedSessions_2016-08-10T00:00:00.000Z_2016-08-11T00:00:00.000Z_2016-11-08T09:08:30.650Z:21, binnedSessions_2016-10-06T00:00:00.000Z_2016-10-07T00:00:00.000Z_2016-11-08T09:21:31.287Z:69, binnedSessions_2016-06-24T00:00:00.000Z_2016-06-25T00:00:00.000Z_2016-11-08T10:14:09.400Z:75, binnedSessions_2016-10-15T00:00:00.000Z_2016-10-16T00:00:00.000Z_2016-11-08T09:24:10.238Z:3, binnedSessions_2016-09-28T00:00:00.000Z_2016-09-29T00:00:00.000Z_2016-11-08T09:18:58.416Z:87, binnedSessions_2016-07-28T00:00:00.000Z_2016-07-29T00:00:00.000Z_2016-11-08T11:02:41.680Z:21, binnedSessions_2016-09-23T00:00:00.000Z_2016-09-24T00:00:00.000Z_2016-11-08T09:18:13.896Z:81, binnedSessions_2016-07-01T00:00:00.000Z_2016-07-02T00:00:00.000Z_2016-11-08T10:31:06.628Z:16, binnedSessions_2016-06-26T00:00:00.000Z_2016-06-27T00:00:00.000Z_2016-11-08T10:18:03.604Z:63, binnedSessions_2016-08-13T00:00:00.000Z_2016-08-14T00:00:00.000Z_2016-11-08T09:09:02.675Z:22, binnedSessions_2016-09-04T00:00:00.000Z_2016-09-05T00:00:00.000Z_2016-11-08T09:13:34.045Z:75, binnedSessions_2016-10-28T00:00:00.000Z_2016-10-29T00:00:00.000Z_2016-11-08T09:27:24.584Z:37, binnedSessions_2016-11-05T00:00:00.000Z_2016-11-06T00:00:00.000Z_2016-11-05T00:21:25.466Z_2:7, binnedSessions_2016-10-01T00:00:00.000Z_2016-10-02T00:00:00.000Z_2016-11-08T09:19:48.092Z:57, binnedSessions_2016-09-26T00:00:00.000Z_2016-09-27T00:00:00.000Z_2016-11-08T09:18:32.598Z:76, binnedSessions_2016-07-11T00:00:00.000Z_2016-07-12T00:00:00.000Z_2016-11-08T10:39:29.662Z:18, binnedSessions_2016-09-08T00:00:00.000Z_2016-09-09T00:00:00.000Z_2016-11-08T09:14:37.991Z:84, binnedSessions_2016-07-22T00:00:00.000Z_2016-07-23T00:00:00.000Z_2016-11-08T10:54:15.948Z:18, binnedSessions_2016-09-07T00:00:00.000Z_2016-09-08T00:00:00.000Z_2016-11-08T09:14:14.866Z:67, binnedSessions_2016-07-17T00:00:00.000Z_2016-07-18T00:00:00.000Z_2016-11-08T10:47:38.203Z:18, binnedSessions_2016-09-14T00:00:00.000Z_2016-09-15T00:00:00.000Z_2016-11-08T09:15:42.735Z:32, binnedSessions_2016-07-04T00:00:00.000Z_2016-07-05T00:00:00.000Z_2016-11-08T10:33:00.954Z:5, binnedSessions_2016-11-07T00:00:00.000Z_2016-11-08T00:00:00.000Z_2016-11-07T00:21:24.892Z_2:88, binnedSessions_2016-07-02T00:00:00.000Z_2016-07-03T00:00:00.000Z_2016-11-08T10:32:29.081Z:13, binnedSessions_2016-10-16T00:00:00.000Z_2016-10-17T00:00:00.000Z_2016-11-08T09:24:13.499Z:31, binnedSessions_2016-07-20T00:00:00.000Z_2016-07-21T00:00:00.000Z_2016-11-08T10:51:41.400Z:24, binnedSessions_2016-08-15T00:00:00.000Z_2016-08-16T00:00:00.000Z_2016-11-08T09:09:20.968Z:25, binnedSessions_2016-10-05T00:00:00.000Z_2016-10-06T00:00:00.000Z_2016-11-08T09:21:09.985Z:77, binnedSessions_2016-07-16T00:00:00.000Z_2016-07-17T00:00:00.000Z_2016-11-08T10:45:49.778Z:19, binnedSessions_2016-08-03T00:00:00.000Z_2016-08-04T00:00:00.000Z_2016-11-08T11:11:42.942Z:24, binnedSessions_2016-06-25T00:00:00.000Z_2016-06-26T00:00:00.000Z_2016-11-08T10:19:25.906Z:72, binnedSessions_2016-10-09T00:00:00.000Z_2016-10-10T00:00:00.000Z_2016-11-08T09:22:24.187Z:91, binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z:27, binnedSessions_2016-07-06T00:00:00.000Z_2016-07-07T00:00:00.000Z_2016-11-08T10:35:03.291Z:9, binnedSessions_2016-07-31T00:00:00.000Z_2016-08-01T00:00:00.000Z_2016-11-08T11:07:08.339Z:15].
2016-11-08T15:53:46,193 INFO [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.FiniteAppenderatorDriver - Persisted pending data in 505,967ms.
2016-11-08T15:53:46,196 INFO [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Shutting down...
2016-11-08T15:53:46,218 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-19T00:00:00.000Z_2016-08-20T00:00:00.000Z_2016-11-08T09:10:17.536Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,218 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-19T00:00:00.000Z_2016-08-20T00:00:00.000Z_2016-11-08T09:10:17.536Z].
2016-11-08T15:53:46,222 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-26T00:00:00.000Z_2016-10-27T00:00:00.000Z_2016-11-08T09:27:04.394Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,223 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-26T00:00:00.000Z_2016-10-27T00:00:00.000Z_2016-11-08T09:27:04.394Z].
2016-11-08T15:53:46,224 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-30T00:00:00.000Z_2016-08-31T00:00:00.000Z_2016-11-08T09:12:25.186Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,224 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-30T00:00:00.000Z_2016-08-31T00:00:00.000Z_2016-11-08T09:12:25.186Z].
2016-11-08T15:53:46,225 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-11T00:00:00.000Z_2016-10-12T00:00:00.000Z_2016-11-08T09:23:07.509Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,226 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-11T00:00:00.000Z_2016-10-12T00:00:00.000Z_2016-11-08T09:23:07.509Z].
2016-11-08T15:53:46,228 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-06T00:00:00.000Z_2016-08-07T00:00:00.000Z_2016-11-08T11:16:43.316Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,228 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-06T00:00:00.000Z_2016-08-07T00:00:00.000Z_2016-11-08T11:16:43.316Z].
2016-11-08T15:53:46,230 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-24T00:00:00.000Z_2016-10-25T00:00:00.000Z_2016-11-08T09:26:22.664Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,231 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-24T00:00:00.000Z_2016-10-25T00:00:00.000Z_2016-11-08T09:26:22.664Z].
2016-11-08T15:53:46,233 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-06-22T00:00:00.000Z_2016-06-23T00:00:00.000Z_2016-11-08T10:11:18.466Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,233 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-06-22T00:00:00.000Z_2016-06-23T00:00:00.000Z_2016-11-08T10:11:18.466Z].
2016-11-08T15:53:46,235 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-11-04T00:00:00.000Z_2016-11-05T00:00:00.000Z_2016-11-04T00:22:15.105Z_3] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,235 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-11-04T00:00:00.000Z_2016-11-05T00:00:00.000Z_2016-11-04T00:22:15.105Z_3].
2016-11-08T15:53:46,237 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-18T00:00:00.000Z_2016-10-19T00:00:00.000Z_2016-11-08T09:24:32.574Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,238 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-18T00:00:00.000Z_2016-10-19T00:00:00.000Z_2016-11-08T09:24:32.574Z].
2016-11-08T15:53:46,239 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-27T00:00:00.000Z_2016-10-28T00:00:00.000Z_2016-11-08T09:27:40.465Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,239 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-27T00:00:00.000Z_2016-10-28T00:00:00.000Z_2016-11-08T09:27:40.465Z].
2016-11-08T15:53:46,240 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-02T00:00:00.000Z_2016-09-03T00:00:00.000Z_2016-11-08T09:13:02.260Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,241 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-02T00:00:00.000Z_2016-09-03T00:00:00.000Z_2016-11-08T09:13:02.260Z].
2016-11-08T15:53:46,242 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-11-08T00:00:00.000Z_2016-11-09T00:00:00.000Z_2016-11-08T00:22:16.022Z_1] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,242 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-11-08T00:00:00.000Z_2016-11-09T00:00:00.000Z_2016-11-08T00:22:16.022Z_1].
2016-11-08T15:53:46,243 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-24T00:00:00.000Z_2016-09-25T00:00:00.000Z_2016-11-08T09:18:32.770Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,244 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-24T00:00:00.000Z_2016-09-25T00:00:00.000Z_2016-11-08T09:18:32.770Z].
2016-11-08T15:53:46,245 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-20T00:00:00.000Z_2016-09-21T00:00:00.000Z_2016-11-08T09:17:01.413Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,245 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-20T00:00:00.000Z_2016-09-21T00:00:00.000Z_2016-11-08T09:17:01.413Z].
2016-11-08T15:53:46,246 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-26T00:00:00.000Z_2016-08-27T00:00:00.000Z_2016-11-08T09:11:46.259Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,247 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-26T00:00:00.000Z_2016-08-27T00:00:00.000Z_2016-11-08T09:11:46.259Z].
2016-11-08T15:53:46,248 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-10T00:00:00.000Z_2016-07-11T00:00:00.000Z_2016-11-08T10:38:04.739Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,248 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-10T00:00:00.000Z_2016-07-11T00:00:00.000Z_2016-11-08T10:38:04.739Z].
2016-11-08T15:53:46,249 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-03T00:00:00.000Z_2016-09-04T00:00:00.000Z_2016-11-08T09:13:17.177Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,250 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-03T00:00:00.000Z_2016-09-04T00:00:00.000Z_2016-11-08T09:13:17.177Z].
2016-11-08T15:53:46,251 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-12T00:00:00.000Z_2016-08-13T00:00:00.000Z_2016-11-08T09:08:45.879Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,251 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-12T00:00:00.000Z_2016-08-13T00:00:00.000Z_2016-11-08T09:08:45.879Z].
2016-11-08T15:53:46,252 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-11-01T00:00:00.000Z_2016-11-02T00:00:00.000Z_2016-11-08T09:28:40.571Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,252 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-11-01T00:00:00.000Z_2016-11-02T00:00:00.000Z_2016-11-08T09:28:40.571Z].
2016-11-08T15:53:46,253 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-10T00:00:00.000Z_2016-10-11T00:00:00.000Z_2016-11-08T09:22:45.399Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,254 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-10T00:00:00.000Z_2016-10-11T00:00:00.000Z_2016-11-08T09:22:45.399Z].
2016-11-08T15:53:46,255 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-06-29T00:00:00.000Z_2016-06-30T00:00:00.000Z_2016-11-08T10:25:17.128Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,255 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-06-29T00:00:00.000Z_2016-06-30T00:00:00.000Z_2016-11-08T10:25:17.128Z].
2016-11-08T15:53:46,257 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-01T00:00:00.000Z_2016-09-02T00:00:00.000Z_2016-11-08T09:12:44.632Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,257 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-01T00:00:00.000Z_2016-09-02T00:00:00.000Z_2016-11-08T09:12:44.632Z].
2016-11-08T15:53:46,258 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-05T00:00:00.000Z_2016-09-06T00:00:00.000Z_2016-11-08T09:13:51.338Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,258 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-05T00:00:00.000Z_2016-09-06T00:00:00.000Z_2016-11-08T09:13:51.338Z].
2016-11-08T15:53:46,260 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-16T00:00:00.000Z_2016-08-17T00:00:00.000Z_2016-11-08T09:09:36.313Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,260 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-16T00:00:00.000Z_2016-08-17T00:00:00.000Z_2016-11-08T09:09:36.313Z].
2016-11-08T15:53:46,261 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-06-30T00:00:00.000Z_2016-07-01T00:00:00.000Z_2016-11-08T10:27:47.942Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,261 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-06-30T00:00:00.000Z_2016-07-01T00:00:00.000Z_2016-11-08T10:27:47.942Z].
2016-11-08T15:53:46,262 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-03T00:00:00.000Z_2016-07-04T00:00:00.000Z_2016-11-08T10:32:13.446Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,263 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-03T00:00:00.000Z_2016-07-04T00:00:00.000Z_2016-11-08T10:32:13.446Z].
2016-11-08T15:53:46,263 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-27T00:00:00.000Z_2016-08-28T00:00:00.000Z_2016-11-08T09:11:58.799Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,264 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-27T00:00:00.000Z_2016-08-28T00:00:00.000Z_2016-11-08T09:11:58.799Z].
2016-11-08T15:53:46,265 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-04T00:00:00.000Z_2016-10-05T00:00:00.000Z_2016-11-08T09:20:51.278Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,265 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-04T00:00:00.000Z_2016-10-05T00:00:00.000Z_2016-11-08T09:20:51.278Z].
2016-11-08T15:53:46,266 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-30T00:00:00.000Z_2016-07-31T00:00:00.000Z_2016-11-08T11:05:28.321Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,266 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-30T00:00:00.000Z_2016-07-31T00:00:00.000Z_2016-11-08T11:05:28.321Z].
2016-11-08T15:53:46,268 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-13T00:00:00.000Z_2016-10-14T00:00:00.000Z_2016-11-08T09:23:48.505Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,268 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-13T00:00:00.000Z_2016-10-14T00:00:00.000Z_2016-11-08T09:23:48.505Z].
2016-11-08T15:53:46,269 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-29T00:00:00.000Z_2016-08-30T00:00:00.000Z_2016-11-08T09:12:20.767Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,269 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-29T00:00:00.000Z_2016-08-30T00:00:00.000Z_2016-11-08T09:12:20.767Z].
2016-11-08T15:53:46,271 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-06-23T00:00:00.000Z_2016-06-24T00:00:00.000Z_2016-11-08T10:14:53.766Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,271 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-06-23T00:00:00.000Z_2016-06-24T00:00:00.000Z_2016-11-08T10:14:53.766Z].
2016-11-08T15:53:46,286 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-27T00:00:00.000Z_2016-09-28T00:00:00.000Z_2016-11-08T09:19:18.833Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,287 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-27T00:00:00.000Z_2016-09-28T00:00:00.000Z_2016-11-08T09:19:18.833Z].
2016-11-08T15:53:46,288 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-11T00:00:00.000Z_2016-09-12T00:00:00.000Z_2016-11-08T09:15:09.722Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,289 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-11T00:00:00.000Z_2016-09-12T00:00:00.000Z_2016-11-08T09:15:09.722Z].
2016-11-08T15:53:46,290 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-05T00:00:00.000Z_2016-08-06T00:00:00.000Z_2016-11-08T11:15:48.289Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,291 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-05T00:00:00.000Z_2016-08-06T00:00:00.000Z_2016-11-08T11:15:48.289Z].
2016-11-08T15:53:46,292 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-21T00:00:00.000Z_2016-09-22T00:00:00.000Z_2016-11-08T09:17:20.852Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,292 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-21T00:00:00.000Z_2016-09-22T00:00:00.000Z_2016-11-08T09:17:20.852Z].
2016-11-08T15:53:46,300 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-19T00:00:00.000Z_2016-10-20T00:00:00.000Z_2016-11-08T09:24:54.849Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,301 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-19T00:00:00.000Z_2016-10-20T00:00:00.000Z_2016-11-08T09:24:54.849Z].
2016-11-08T15:53:46,302 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-11T00:00:00.000Z_2016-08-12T00:00:00.000Z_2016-11-08T09:08:35.017Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,302 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-11T00:00:00.000Z_2016-08-12T00:00:00.000Z_2016-11-08T09:08:35.017Z].
2016-11-08T15:53:46,304 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-22T00:00:00.000Z_2016-09-23T00:00:00.000Z_2016-11-08T09:17:51.767Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,304 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-22T00:00:00.000Z_2016-09-23T00:00:00.000Z_2016-11-08T09:17:51.767Z].
2016-11-08T15:53:46,310 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-31T00:00:00.000Z_2016-11-01T00:00:00.000Z_2016-11-08T09:28:28.774Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,310 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-31T00:00:00.000Z_2016-11-01T00:00:00.000Z_2016-11-08T09:28:28.774Z].
2016-11-08T15:53:46,312 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-30T00:00:00.000Z_2016-10-01T00:00:00.000Z_2016-11-08T09:19:33.831Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,312 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-30T00:00:00.000Z_2016-10-01T00:00:00.000Z_2016-11-08T09:19:33.831Z].
2016-11-08T15:53:46,313 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-07T00:00:00.000Z_2016-08-08T00:00:00.000Z_2016-11-08T11:18:03.936Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,314 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-07T00:00:00.000Z_2016-08-08T00:00:00.000Z_2016-11-08T11:18:03.936Z].
2016-11-08T15:53:46,315 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-18T00:00:00.000Z_2016-07-19T00:00:00.000Z_2016-11-08T10:48:19.393Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,315 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-18T00:00:00.000Z_2016-07-19T00:00:00.000Z_2016-11-08T10:48:19.393Z].
2016-11-08T15:53:46,317 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-04T00:00:00.000Z_2016-08-05T00:00:00.000Z_2016-11-08T11:13:33.953Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,317 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-04T00:00:00.000Z_2016-08-05T00:00:00.000Z_2016-11-08T11:13:33.953Z].
2016-11-08T15:53:46,320 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-22T00:00:00.000Z_2016-08-23T00:00:00.000Z_2016-11-08T09:11:03.214Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,321 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-22T00:00:00.000Z_2016-08-23T00:00:00.000Z_2016-11-08T09:11:03.214Z].
2016-11-08T15:53:46,322 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-20T00:00:00.000Z_2016-08-21T00:00:00.000Z_2016-11-08T09:10:34.559Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,322 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-20T00:00:00.000Z_2016-08-21T00:00:00.000Z_2016-11-08T09:10:34.559Z].
2016-11-08T15:53:46,323 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-19T00:00:00.000Z_2016-07-20T00:00:00.000Z_2016-11-08T10:49:58.959Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,323 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-19T00:00:00.000Z_2016-07-20T00:00:00.000Z_2016-11-08T10:49:58.959Z].
2016-11-08T15:53:46,325 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-06-28T00:00:00.000Z_2016-06-29T00:00:00.000Z_2016-11-08T10:22:54.736Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,325 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-06-28T00:00:00.000Z_2016-06-29T00:00:00.000Z_2016-11-08T10:22:54.736Z].
2016-11-08T15:53:46,326 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-02T00:00:00.000Z_2016-10-03T00:00:00.000Z_2016-11-08T09:20:08.136Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,326 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-02T00:00:00.000Z_2016-10-03T00:00:00.000Z_2016-11-08T09:20:08.136Z].
2016-11-08T15:53:46,328 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-12T00:00:00.000Z_2016-10-13T00:00:00.000Z_2016-11-08T09:23:22.541Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,328 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-12T00:00:00.000Z_2016-10-13T00:00:00.000Z_2016-11-08T09:23:22.541Z].
2016-11-08T15:53:46,330 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-02T00:00:00.000Z_2016-08-03T00:00:00.000Z_2016-11-08T11:10:15.210Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,331 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-02T00:00:00.000Z_2016-08-03T00:00:00.000Z_2016-11-08T11:10:15.210Z].
2016-11-08T15:53:46,332 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-06-21T00:00:00.000Z_2016-06-22T00:00:00.000Z_2016-11-08T10:09:27.747Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,332 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-06-21T00:00:00.000Z_2016-06-22T00:00:00.000Z_2016-11-08T10:09:27.747Z].
2016-11-08T15:53:46,333 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-09T00:00:00.000Z_2016-07-10T00:00:00.000Z_2016-11-08T10:36:51.145Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,333 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-09T00:00:00.000Z_2016-07-10T00:00:00.000Z_2016-11-08T10:36:51.145Z].
2016-11-08T15:53:46,334 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-09T00:00:00.000Z_2016-09-10T00:00:00.000Z_2016-11-08T09:14:41.387Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,335 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-09T00:00:00.000Z_2016-09-10T00:00:00.000Z_2016-11-08T09:14:41.387Z].
2016-11-08T15:53:46,336 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-17T00:00:00.000Z_2016-09-18T00:00:00.000Z_2016-11-08T09:16:23.787Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,336 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-17T00:00:00.000Z_2016-09-18T00:00:00.000Z_2016-11-08T09:16:23.787Z].
2016-11-08T15:53:46,337 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-05T00:00:00.000Z_2016-07-06T00:00:00.000Z_2016-11-08T10:33:49.484Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,338 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-05T00:00:00.000Z_2016-07-06T00:00:00.000Z_2016-11-08T10:33:49.484Z].
2016-11-08T15:53:46,339 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-01T00:00:00.000Z_2016-08-02T00:00:00.000Z_2016-11-08T11:08:09.045Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,339 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-01T00:00:00.000Z_2016-08-02T00:00:00.000Z_2016-11-08T11:08:09.045Z].
2016-11-08T15:53:46,340 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-08T00:00:00.000Z_2016-08-09T00:00:00.000Z_2016-11-08T11:19:28.834Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,341 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-08T00:00:00.000Z_2016-08-09T00:00:00.000Z_2016-11-08T11:19:28.834Z].
2016-11-08T15:53:46,342 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-25T00:00:00.000Z_2016-07-26T00:00:00.000Z_2016-11-08T10:58:29.571Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,342 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-25T00:00:00.000Z_2016-07-26T00:00:00.000Z_2016-11-08T10:58:29.571Z].
2016-11-08T15:53:46,343 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-08T00:00:00.000Z_2016-10-09T00:00:00.000Z_2016-11-08T09:22:08.140Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,344 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-08T00:00:00.000Z_2016-10-09T00:00:00.000Z_2016-11-08T09:22:08.140Z].
2016-11-08T15:53:46,345 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-11-03T00:00:00.000Z_2016-11-04T00:00:00.000Z_2016-11-03T10:23:07.264Z_1] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,345 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-11-03T00:00:00.000Z_2016-11-04T00:00:00.000Z_2016-11-03T10:23:07.264Z_1].
2016-11-08T15:53:46,346 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-06T00:00:00.000Z_2016-09-07T00:00:00.000Z_2016-11-08T09:14:07.289Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,346 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-06T00:00:00.000Z_2016-09-07T00:00:00.000Z_2016-11-08T09:14:07.289Z].
2016-11-08T15:53:46,350 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-14T00:00:00.000Z_2016-10-15T00:00:00.000Z_2016-11-08T09:23:55.532Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,350 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-14T00:00:00.000Z_2016-10-15T00:00:00.000Z_2016-11-08T09:23:55.532Z].
2016-11-08T15:53:46,351 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-15T00:00:00.000Z_2016-09-16T00:00:00.000Z_2016-11-08T09:15:59.169Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,351 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-15T00:00:00.000Z_2016-09-16T00:00:00.000Z_2016-11-08T09:15:59.169Z].
2016-11-08T15:53:46,355 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-13T00:00:00.000Z_2016-07-14T00:00:00.000Z_2016-11-08T10:42:02.651Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,356 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-13T00:00:00.000Z_2016-07-14T00:00:00.000Z_2016-11-08T10:42:02.651Z].
2016-11-08T15:53:46,357 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-18T00:00:00.000Z_2016-09-19T00:00:00.000Z_2016-11-08T09:16:49.914Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,357 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-18T00:00:00.000Z_2016-09-19T00:00:00.000Z_2016-11-08T09:16:49.914Z].
2016-11-08T15:53:46,358 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-12T00:00:00.000Z_2016-07-13T00:00:00.000Z_2016-11-08T10:40:22.950Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,358 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-12T00:00:00.000Z_2016-07-13T00:00:00.000Z_2016-11-08T10:40:22.950Z].
2016-11-08T15:53:46,358 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-29T00:00:00.000Z_2016-10-30T00:00:00.000Z_2016-11-08T09:27:45.887Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,359 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-29T00:00:00.000Z_2016-10-30T00:00:00.000Z_2016-11-08T09:27:45.887Z].
2016-11-08T15:53:46,359 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-14T00:00:00.000Z_2016-07-15T00:00:00.000Z_2016-11-08T10:43:04.902Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,360 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-14T00:00:00.000Z_2016-07-15T00:00:00.000Z_2016-11-08T10:43:04.902Z].
2016-11-08T15:53:46,360 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-26T00:00:00.000Z_2016-07-27T00:00:00.000Z_2016-11-08T10:59:29.907Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,360 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-26T00:00:00.000Z_2016-07-27T00:00:00.000Z_2016-11-08T10:59:29.907Z].
2016-11-08T15:53:46,361 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-06-20T00:00:00.000Z_2016-06-21T00:00:00.000Z_2016-11-08T10:09:13.263Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,361 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-06-20T00:00:00.000Z_2016-06-21T00:00:00.000Z_2016-11-08T10:09:13.263Z].
2016-11-08T15:53:46,362 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-10T00:00:00.000Z_2016-09-11T00:00:00.000Z_2016-11-08T09:14:55.122Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,362 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-10T00:00:00.000Z_2016-09-11T00:00:00.000Z_2016-11-08T09:14:55.122Z].
2016-11-08T15:53:46,363 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-22T00:00:00.000Z_2016-10-23T00:00:00.000Z_2016-11-08T09:25:53.404Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,363 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-22T00:00:00.000Z_2016-10-23T00:00:00.000Z_2016-11-08T09:25:53.404Z].
2016-11-08T15:53:46,364 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-21T00:00:00.000Z_2016-10-22T00:00:00.000Z_2016-11-08T09:25:28.585Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,364 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-21T00:00:00.000Z_2016-10-22T00:00:00.000Z_2016-11-08T09:25:28.585Z].
2016-11-08T15:53:46,365 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-14T00:00:00.000Z_2016-08-15T00:00:00.000Z_2016-11-08T09:09:12.818Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,365 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-14T00:00:00.000Z_2016-08-15T00:00:00.000Z_2016-11-08T09:09:12.818Z].
2016-11-08T15:53:46,365 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-29T00:00:00.000Z_2016-07-30T00:00:00.000Z_2016-11-08T11:04:55.338Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,366 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-29T00:00:00.000Z_2016-07-30T00:00:00.000Z_2016-11-08T11:04:55.338Z].
2016-11-08T15:53:46,366 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-23T00:00:00.000Z_2016-08-24T00:00:00.000Z_2016-11-08T09:11:07.551Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,366 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-23T00:00:00.000Z_2016-08-24T00:00:00.000Z_2016-11-08T09:11:07.551Z].
2016-11-08T15:53:46,367 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-08T00:00:00.000Z_2016-07-09T00:00:00.000Z_2016-11-08T10:35:59.888Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,367 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-08T00:00:00.000Z_2016-07-09T00:00:00.000Z_2016-11-08T10:35:59.888Z].
2016-11-08T15:53:46,368 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-31T00:00:00.000Z_2016-09-01T00:00:00.000Z_2016-11-08T09:12:26.445Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,368 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-31T00:00:00.000Z_2016-09-01T00:00:00.000Z_2016-11-08T09:12:26.445Z].
2016-11-08T15:53:46,371 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-19T00:00:00.000Z_2016-09-20T00:00:00.000Z_2016-11-08T09:17:00.222Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,371 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-19T00:00:00.000Z_2016-09-20T00:00:00.000Z_2016-11-08T09:17:00.222Z].
2016-11-08T15:53:46,372 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-03T00:00:00.000Z_2016-10-04T00:00:00.000Z_2016-11-08T09:20:25.659Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,372 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-03T00:00:00.000Z_2016-10-04T00:00:00.000Z_2016-11-08T09:20:25.659Z].
2016-11-08T15:53:46,373 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-23T00:00:00.000Z_2016-07-24T00:00:00.000Z_2016-11-08T10:55:41.960Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,373 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-23T00:00:00.000Z_2016-07-24T00:00:00.000Z_2016-11-08T10:55:41.960Z].
2016-11-08T15:53:46,374 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-23T00:00:00.000Z_2016-10-24T00:00:00.000Z_2016-11-08T09:26:10.238Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,375 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-23T00:00:00.000Z_2016-10-24T00:00:00.000Z_2016-11-08T09:26:10.238Z].
2016-11-08T15:53:46,375 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-07T00:00:00.000Z_2016-10-08T00:00:00.000Z_2016-11-08T09:21:53.922Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,376 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-07T00:00:00.000Z_2016-10-08T00:00:00.000Z_2016-11-08T09:21:53.922Z].
2016-11-08T15:53:46,376 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-25T00:00:00.000Z_2016-10-26T00:00:00.000Z_2016-11-08T09:26:53.414Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,377 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-25T00:00:00.000Z_2016-10-26T00:00:00.000Z_2016-11-08T09:26:53.414Z].
2016-11-08T15:53:46,378 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-24T00:00:00.000Z_2016-08-25T00:00:00.000Z_2016-11-08T09:11:14.105Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,378 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-24T00:00:00.000Z_2016-08-25T00:00:00.000Z_2016-11-08T09:11:14.105Z].
2016-11-08T15:53:46,379 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-16T00:00:00.000Z_2016-09-17T00:00:00.000Z_2016-11-08T09:16:16.924Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,379 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-16T00:00:00.000Z_2016-09-17T00:00:00.000Z_2016-11-08T09:16:16.924Z].
2016-11-08T15:53:46,380 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-25T00:00:00.000Z_2016-09-26T00:00:00.000Z_2016-11-08T09:18:14.042Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,380 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-25T00:00:00.000Z_2016-09-26T00:00:00.000Z_2016-11-08T09:18:14.042Z].
2016-11-08T15:53:46,381 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-06-27T00:00:00.000Z_2016-06-28T00:00:00.000Z_2016-11-08T10:20:23.175Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,381 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-06-27T00:00:00.000Z_2016-06-28T00:00:00.000Z_2016-11-08T10:20:23.175Z].
2016-11-08T15:53:46,382 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-17T00:00:00.000Z_2016-08-18T00:00:00.000Z_2016-11-08T09:09:45.037Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,382 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-17T00:00:00.000Z_2016-08-18T00:00:00.000Z_2016-11-08T09:09:45.037Z].
2016-11-08T15:53:46,383 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-12T00:00:00.000Z_2016-09-13T00:00:00.000Z_2016-11-08T09:15:19.273Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,383 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-12T00:00:00.000Z_2016-09-13T00:00:00.000Z_2016-11-08T09:15:19.273Z].
2016-11-08T15:53:46,384 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-13T00:00:00.000Z_2016-09-14T00:00:00.000Z_2016-11-08T09:15:36.571Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,384 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-13T00:00:00.000Z_2016-09-14T00:00:00.000Z_2016-11-08T09:15:36.571Z].
2016-11-08T15:53:46,385 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-17T00:00:00.000Z_2016-10-18T00:00:00.000Z_2016-11-08T09:24:15.951Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,385 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-17T00:00:00.000Z_2016-10-18T00:00:00.000Z_2016-11-08T09:24:15.951Z].
2016-11-08T15:53:46,386 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-27T00:00:00.000Z_2016-07-28T00:00:00.000Z_2016-11-08T11:01:08.815Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,386 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-27T00:00:00.000Z_2016-07-28T00:00:00.000Z_2016-11-08T11:01:08.815Z].
2016-11-08T15:53:46,388 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-30T00:00:00.000Z_2016-10-31T00:00:00.000Z_2016-11-08T09:28:18.105Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,388 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-30T00:00:00.000Z_2016-10-31T00:00:00.000Z_2016-11-08T09:28:18.105Z].
2016-11-08T15:53:46,388 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-07T00:00:00.000Z_2016-07-08T00:00:00.000Z_2016-11-08T10:35:22.023Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,389 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-07T00:00:00.000Z_2016-07-08T00:00:00.000Z_2016-11-08T10:35:22.023Z].
2016-11-08T15:53:46,389 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-15T00:00:00.000Z_2016-07-16T00:00:00.000Z_2016-11-08T10:44:35.665Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,390 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-15T00:00:00.000Z_2016-07-16T00:00:00.000Z_2016-11-08T10:44:35.665Z].
2016-11-08T15:53:46,390 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-24T00:00:00.000Z_2016-07-25T00:00:00.000Z_2016-11-08T10:56:41.041Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,390 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-24T00:00:00.000Z_2016-07-25T00:00:00.000Z_2016-11-08T10:56:41.041Z].
2016-11-08T15:53:46,391 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-18T00:00:00.000Z_2016-08-19T00:00:00.000Z_2016-11-08T09:10:02.353Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,391 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-18T00:00:00.000Z_2016-08-19T00:00:00.000Z_2016-11-08T09:10:02.353Z].
2016-11-08T15:53:46,392 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-20T00:00:00.000Z_2016-10-21T00:00:00.000Z_2016-11-08T09:25:06.222Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,392 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-20T00:00:00.000Z_2016-10-21T00:00:00.000Z_2016-11-08T09:25:06.222Z].
2016-11-08T15:53:46,393 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-21T00:00:00.000Z_2016-07-22T00:00:00.000Z_2016-11-08T10:53:09.966Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,393 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-21T00:00:00.000Z_2016-07-22T00:00:00.000Z_2016-11-08T10:53:09.966Z].
2016-11-08T15:53:46,394 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-11-06T00:00:00.000Z_2016-11-07T00:00:00.000Z_2016-11-06T00:21:25.492Z_1] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,394 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-11-06T00:00:00.000Z_2016-11-07T00:00:00.000Z_2016-11-06T00:21:25.492Z_1].
2016-11-08T15:53:46,394 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-09T00:00:00.000Z_2016-08-10T00:00:00.000Z_2016-11-08T11:20:55.666Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,394 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-09T00:00:00.000Z_2016-08-10T00:00:00.000Z_2016-11-08T11:20:55.666Z].
2016-11-08T15:53:46,396 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-29T00:00:00.000Z_2016-09-30T00:00:00.000Z_2016-11-08T09:19:15.564Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,397 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-29T00:00:00.000Z_2016-09-30T00:00:00.000Z_2016-11-08T09:19:15.564Z].
2016-11-08T15:53:46,397 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-25T00:00:00.000Z_2016-08-26T00:00:00.000Z_2016-11-08T09:11:28.552Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,398 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-25T00:00:00.000Z_2016-08-26T00:00:00.000Z_2016-11-08T09:11:28.552Z].
2016-11-08T15:53:46,398 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-21T00:00:00.000Z_2016-08-22T00:00:00.000Z_2016-11-08T09:10:48.819Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,398 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-21T00:00:00.000Z_2016-08-22T00:00:00.000Z_2016-11-08T09:10:48.819Z].
2016-11-08T15:53:46,399 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-10T00:00:00.000Z_2016-08-11T00:00:00.000Z_2016-11-08T09:08:30.650Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,399 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-10T00:00:00.000Z_2016-08-11T00:00:00.000Z_2016-11-08T09:08:30.650Z].
2016-11-08T15:53:46,400 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-06T00:00:00.000Z_2016-10-07T00:00:00.000Z_2016-11-08T09:21:31.287Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,400 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-06T00:00:00.000Z_2016-10-07T00:00:00.000Z_2016-11-08T09:21:31.287Z].
2016-11-08T15:53:46,401 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-06-24T00:00:00.000Z_2016-06-25T00:00:00.000Z_2016-11-08T10:14:09.400Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,401 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-06-24T00:00:00.000Z_2016-06-25T00:00:00.000Z_2016-11-08T10:14:09.400Z].
2016-11-08T15:53:46,402 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-15T00:00:00.000Z_2016-10-16T00:00:00.000Z_2016-11-08T09:24:10.238Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,402 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-15T00:00:00.000Z_2016-10-16T00:00:00.000Z_2016-11-08T09:24:10.238Z].
2016-11-08T15:53:46,402 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-28T00:00:00.000Z_2016-09-29T00:00:00.000Z_2016-11-08T09:18:58.416Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,403 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-28T00:00:00.000Z_2016-09-29T00:00:00.000Z_2016-11-08T09:18:58.416Z].
2016-11-08T15:53:46,403 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-28T00:00:00.000Z_2016-07-29T00:00:00.000Z_2016-11-08T11:02:41.680Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,404 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-28T00:00:00.000Z_2016-07-29T00:00:00.000Z_2016-11-08T11:02:41.680Z].
2016-11-08T15:53:46,404 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-23T00:00:00.000Z_2016-09-24T00:00:00.000Z_2016-11-08T09:18:13.896Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,404 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-23T00:00:00.000Z_2016-09-24T00:00:00.000Z_2016-11-08T09:18:13.896Z].
2016-11-08T15:53:46,405 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-01T00:00:00.000Z_2016-07-02T00:00:00.000Z_2016-11-08T10:31:06.628Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,405 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-01T00:00:00.000Z_2016-07-02T00:00:00.000Z_2016-11-08T10:31:06.628Z].
2016-11-08T15:53:46,406 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-06-26T00:00:00.000Z_2016-06-27T00:00:00.000Z_2016-11-08T10:18:03.604Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,406 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-06-26T00:00:00.000Z_2016-06-27T00:00:00.000Z_2016-11-08T10:18:03.604Z].
2016-11-08T15:53:46,406 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-13T00:00:00.000Z_2016-08-14T00:00:00.000Z_2016-11-08T09:09:02.675Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,407 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-13T00:00:00.000Z_2016-08-14T00:00:00.000Z_2016-11-08T09:09:02.675Z].
2016-11-08T15:53:46,407 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-04T00:00:00.000Z_2016-09-05T00:00:00.000Z_2016-11-08T09:13:34.045Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,407 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-04T00:00:00.000Z_2016-09-05T00:00:00.000Z_2016-11-08T09:13:34.045Z].
2016-11-08T15:53:46,408 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-28T00:00:00.000Z_2016-10-29T00:00:00.000Z_2016-11-08T09:27:24.584Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,408 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-28T00:00:00.000Z_2016-10-29T00:00:00.000Z_2016-11-08T09:27:24.584Z].
2016-11-08T15:53:46,408 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-11-05T00:00:00.000Z_2016-11-06T00:00:00.000Z_2016-11-05T00:21:25.466Z_2] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,408 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-11-05T00:00:00.000Z_2016-11-06T00:00:00.000Z_2016-11-05T00:21:25.466Z_2].
2016-11-08T15:53:46,409 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-01T00:00:00.000Z_2016-10-02T00:00:00.000Z_2016-11-08T09:19:48.092Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,409 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-01T00:00:00.000Z_2016-10-02T00:00:00.000Z_2016-11-08T09:19:48.092Z].
2016-11-08T15:53:46,411 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-26T00:00:00.000Z_2016-09-27T00:00:00.000Z_2016-11-08T09:18:32.598Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,411 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-26T00:00:00.000Z_2016-09-27T00:00:00.000Z_2016-11-08T09:18:32.598Z].
2016-11-08T15:53:46,412 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-11T00:00:00.000Z_2016-07-12T00:00:00.000Z_2016-11-08T10:39:29.662Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,412 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-11T00:00:00.000Z_2016-07-12T00:00:00.000Z_2016-11-08T10:39:29.662Z].
2016-11-08T15:53:46,412 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-08T00:00:00.000Z_2016-09-09T00:00:00.000Z_2016-11-08T09:14:37.991Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,413 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-08T00:00:00.000Z_2016-09-09T00:00:00.000Z_2016-11-08T09:14:37.991Z].
2016-11-08T15:53:46,413 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-22T00:00:00.000Z_2016-07-23T00:00:00.000Z_2016-11-08T10:54:15.948Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,413 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-22T00:00:00.000Z_2016-07-23T00:00:00.000Z_2016-11-08T10:54:15.948Z].
2016-11-08T15:53:46,414 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-07T00:00:00.000Z_2016-09-08T00:00:00.000Z_2016-11-08T09:14:14.866Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,414 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-07T00:00:00.000Z_2016-09-08T00:00:00.000Z_2016-11-08T09:14:14.866Z].
2016-11-08T15:53:46,414 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-17T00:00:00.000Z_2016-07-18T00:00:00.000Z_2016-11-08T10:47:38.203Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,414 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-17T00:00:00.000Z_2016-07-18T00:00:00.000Z_2016-11-08T10:47:38.203Z].
2016-11-08T15:53:46,415 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-09-14T00:00:00.000Z_2016-09-15T00:00:00.000Z_2016-11-08T09:15:42.735Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,415 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-09-14T00:00:00.000Z_2016-09-15T00:00:00.000Z_2016-11-08T09:15:42.735Z].
2016-11-08T15:53:46,415 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-04T00:00:00.000Z_2016-07-05T00:00:00.000Z_2016-11-08T10:33:00.954Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,415 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-04T00:00:00.000Z_2016-07-05T00:00:00.000Z_2016-11-08T10:33:00.954Z].
2016-11-08T15:53:46,415 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-11-07T00:00:00.000Z_2016-11-08T00:00:00.000Z_2016-11-07T00:21:24.892Z_2] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,416 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-11-07T00:00:00.000Z_2016-11-08T00:00:00.000Z_2016-11-07T00:21:24.892Z_2].
2016-11-08T15:53:46,416 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-02T00:00:00.000Z_2016-07-03T00:00:00.000Z_2016-11-08T10:32:29.081Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,416 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-02T00:00:00.000Z_2016-07-03T00:00:00.000Z_2016-11-08T10:32:29.081Z].
2016-11-08T15:53:46,416 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-16T00:00:00.000Z_2016-10-17T00:00:00.000Z_2016-11-08T09:24:13.499Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,417 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-16T00:00:00.000Z_2016-10-17T00:00:00.000Z_2016-11-08T09:24:13.499Z].
2016-11-08T15:53:46,417 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-20T00:00:00.000Z_2016-07-21T00:00:00.000Z_2016-11-08T10:51:41.400Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,417 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-20T00:00:00.000Z_2016-07-21T00:00:00.000Z_2016-11-08T10:51:41.400Z].
2016-11-08T15:53:46,417 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-15T00:00:00.000Z_2016-08-16T00:00:00.000Z_2016-11-08T09:09:20.968Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,417 INFO [appenderator_persist_0] io.druid.curator.announcement.Announcer - unannouncing [/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T07:53:53.051Z_d56585d5d51c4d59a66a0b23cf1dee480]
2016-11-08T15:53:46,496 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-15T00:00:00.000Z_2016-08-16T00:00:00.000Z_2016-11-08T09:09:20.968Z].
2016-11-08T15:53:46,496 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-05T00:00:00.000Z_2016-10-06T00:00:00.000Z_2016-11-08T09:21:09.985Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,496 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-05T00:00:00.000Z_2016-10-06T00:00:00.000Z_2016-11-08T09:21:09.985Z].
2016-11-08T15:53:46,497 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-16T00:00:00.000Z_2016-07-17T00:00:00.000Z_2016-11-08T10:45:49.778Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,497 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-16T00:00:00.000Z_2016-07-17T00:00:00.000Z_2016-11-08T10:45:49.778Z].
2016-11-08T15:53:46,497 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-08-03T00:00:00.000Z_2016-08-04T00:00:00.000Z_2016-11-08T11:11:42.942Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,497 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-08-03T00:00:00.000Z_2016-08-04T00:00:00.000Z_2016-11-08T11:11:42.942Z].
2016-11-08T15:53:46,498 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-06-25T00:00:00.000Z_2016-06-26T00:00:00.000Z_2016-11-08T10:19:25.906Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,498 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-06-25T00:00:00.000Z_2016-06-26T00:00:00.000Z_2016-11-08T10:19:25.906Z].
2016-11-08T15:53:46,498 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-10-09T00:00:00.000Z_2016-10-10T00:00:00.000Z_2016-11-08T09:22:24.187Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,498 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-10-09T00:00:00.000Z_2016-10-10T00:00:00.000Z_2016-11-08T09:22:24.187Z].
2016-11-08T15:53:46,498 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,498 INFO [appenderator_persist_0] io.druid.curator.announcement.Announcer - unannouncing [/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T09:19:15.567Z_e3a98fbc33d74f43bf5b257cab4b23f61]
2016-11-08T15:53:46,505 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z].
2016-11-08T15:53:46,506 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-06T00:00:00.000Z_2016-07-07T00:00:00.000Z_2016-11-08T10:35:03.291Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,506 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-06T00:00:00.000Z_2016-07-07T00:00:00.000Z_2016-11-08T10:35:03.291Z].
2016-11-08T15:53:46,506 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-31T00:00:00.000Z_2016-08-01T00:00:00.000Z_2016-11-08T11:07:08.339Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,506 INFO [appenderator_persist_0] io.druid.curator.announcement.Announcer - unannouncing [/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T10:27:47.949Z_c2c1b23dc9c24e0c9a2b9640b73a697b2]
2016-11-08T15:53:46,515 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-31T00:00:00.000Z_2016-08-01T00:00:00.000Z_2016-11-08T11:07:08.339Z].
2016-11-08T15:53:46,516 INFO [task-runner-0-priority-0] io.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider - Unregistering chat handler[index_kafka_binnedSessions_3aa45483b4be612_iehphlgk]
2016-11-08T15:53:46,518 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Uncaught Throwable while running task[KafkaIndexTask{id=index_kafka_binnedSessions_3aa45483b4be612_iehphlgk, type=index_kafka, dataSource=binnedSessions}]
java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[?:1.8.0_74]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[?:1.8.0_74]
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93) ~[?:?]
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71) ~[?:?]
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) ~[?:?]
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) ~[?:?]
at org.apache.kafka.common.network.Selector.poll(Selector.java:286) ~[?:?]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) ~[?:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320) ~[?:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213) ~[?:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193) ~[?:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:908) ~[?:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:853) ~[?:?]
at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:385) ~[?:?]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
2016-11-08T15:53:46,530 ERROR [main] io.druid.cli.CliPeon - Error when starting up. Failing.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
at io.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:211) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]
at io.druid.cli.CliPeon.run(CliPeon.java:287) [druid-services-0.9.2-rc2.jar:0.9.2-rc2]
at io.druid.cli.Main.main(Main.java:106) [druid-services-0.9.2-rc2.jar:0.9.2-rc2]
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
at io.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:208) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]
... 2 more
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) ~[?:1.8.0_74]
at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) ~[?:1.8.0_74]
at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:93) ~[?:?]
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71) ~[?:?]
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:153) ~[?:?]
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:134) ~[?:?]
at org.apache.kafka.common.network.Selector.poll(Selector.java:286) ~[?:?]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:256) ~[?:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:320) ~[?:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:213) ~[?:?]
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:193) ~[?:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:908) ~[?:?]
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:853) ~[?:?]
at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:385) ~[?:?]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_74]
2016-11-08T15:53:46,542 INFO [Thread-44] io.druid.cli.CliPeon - Running shutdown hook
2016-11-08T15:53:46,543 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.server.coordination.AbstractDataSegmentAnnouncer.stop()] on object[io.druid.server.coordination.BatchDataSegmentAnnouncer@615db358].
2016-11-08T15:53:46,543 INFO [Thread-44] io.druid.server.coordination.AbstractDataSegmentAnnouncer - Stopping class io.druid.server.coordination.BatchDataSegmentAnnouncer with config[io.druid.server.initialization.ZkPathsConfig@22e2266d]
2016-11-08T15:53:46,543 INFO [Thread-44] io.druid.curator.announcement.Announcer - unannouncing [/druid/announcements/data-8ce3acae.node.rsa:8100]
2016-11-08T15:53:46,548 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.server.listener.announcer.ListenerResourceAnnouncer.stop()] on object[io.druid.query.lookup.LookupResourceListenerAnnouncer@33ec2c0c].
2016-11-08T15:53:46,548 INFO [Thread-44] io.druid.curator.announcement.Announcer - unannouncing [/druid/listeners/lookups/__default/data-8ce3acae.node.rsa:8100]
2016-11-08T15:53:46,551 INFO [Thread-44] io.druid.server.listener.announcer.ListenerResourceAnnouncer - Unannouncing start time on [/druid/listeners/lookups/__default/data-8ce3acae.node.rsa:8100]
2016-11-08T15:53:46,551 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.query.lookup.LookupReferencesManager.stop()] on object[io.druid.query.lookup.LookupReferencesManager@7c588adc].
2016-11-08T15:53:46,551 INFO [Thread-44] io.druid.query.lookup.LookupReferencesManager - Stopping lookup factory references manager
2016-11-08T15:53:46,709 INFO [Thread-44] org.eclipse.jetty.server.ServerConnector - Stopped ServerConnector@3e5beab5{HTTP/1.1}{0.0.0.0:8100}
2016-11-08T15:53:46,715 INFO [Thread-44] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@611d0763{/,null,UNAVAILABLE}
2016-11-08T15:53:46,722 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.worker.executor.ExecutorLifecycle.stop() throws java.lang.Exception] on object[io.druid.indexing.worker.executor.ExecutorLifecycle@14168e1].
2016-11-08T15:53:46,723 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.indexing.overlord.ThreadPoolTaskRunner.stop()] on object[io.druid.indexing.overlord.ThreadPoolTaskRunner@24fa9403].
2016-11-08T15:53:46,723 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@65629ac6].
2016-11-08T15:53:46,770 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.announcement.Announcer.stop()] on object[io.druid.curator.announcement.Announcer@4c19d42e].
2016-11-08T15:53:46,770 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on object[io.druid.curator.discovery.ServerDiscoverySelector@7b25ab31].
2016-11-08T15:53:46,770 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.http.client.NettyHttpClient.stop()] on object[com.metamx.http.client.NettyHttpClient@5b000fe6].
2016-11-08T15:53:46,839 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.storage.hdfs.HdfsStorageAuthentication.stop()] on object[io.druid.storage.hdfs.HdfsStorageAuthentication@377874b4].
2016-11-08T15:53:46,839 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.metrics.MonitorScheduler.stop()] on object[com.metamx.metrics.MonitorScheduler@53940e2f].
2016-11-08T15:53:46,839 INFO [Thread-44] io.druid.curator.CuratorModule - Stopping Curator
2016-11-08T15:53:46,840 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2016-11-08T15:53:46,850 INFO [Thread-44] org.apache.zookeeper.ZooKeeper - Session: 0x15842ed2230001a closed
2016-11-08T15:53:46,850 INFO [Curator-ServiceProvider-0-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x15842ed2230001a
2016-11-08T15:53:46,850 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void com.metamx.emitter.service.ServiceEmitter.close() throws java.io.IOException] on object[com.metamx.emitter.service.ServiceEmitter@1dcedc93].
2016-11-08T15:53:46,850 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner.stop()] on object[io.druid.initialization.Log4jShutterDownerModule$Log4jShutterDowner@7edb6fca].
David Lim
8 nov 2016, 21:13:538/11/16
a Druid User
Hey Linbo,
The intermediate segments spilled to disk when maxRowsInMemory is exceeded don't get loaded back into heap memory for a query (they're memory mapped, just like on historicals), but depending on what type of query you issue, the query will have overhead memory requirements and may do intermediate materialization of results that could cause an OOM. Some suggestions:
1) Is there a particular reason you need such a long taskDuration (30 hours)? If you can get away with handing off segments more frequently, it would help.
2) Otherwise, simply increasing the peon heap memory (leaving maxRowsInMemory as is) should help.
The intermediate segments spilled to disk when maxRowsInMemory is exceeded don't get loaded back into heap memory for a query (they're memory mapped, just like on historicals), but depending on what type of query you issue, the query will have overhead memory requirements and may do intermediate materialization of results that could cause an OOM. Some suggestions:
1) Is there a particular reason you need such a long taskDuration (30 hours)? If you can get away with handing off segments more frequently, it would help.
2) Otherwise, simply increasing the peon heap memory (leaving maxRowsInMemory as is) should help.
Linbo Jin
8 nov 2016, 21:37:128/11/16
a Druid User
Hi David,
Thanks for your reply.
1. We choose taskDuration (30 hours) because we have daily segment granularity and we think we should make taskDuration longer than 24h (otherwise it will create multiple vertical segments for same day and we have not implemented merge tasks).
2. After indexing task starting, we started to migrate old data (couple months) and it took 6h. And then if we query data, it will be OOM. So only if we set taskDuration < 6h, will it help. And in this case, indexing task will create 4 vertical segments for each day data. We concern the query performance.
3. Just FYI, query we issued:
select line chart for whole range data on pivot +
{ "queryType": "topN", "dataSource": "binnedSessions", "dimension": "apMac", "threshold": 10, "metric": "count", "granularity": "all", "filter": { "type": "selector", "dimension": "zoneName", "value": "KTW1M Nera Putrajaya (MESH)"}, "aggregations": [{"type":"count", "name":"count"}], "intervals": ["2016-06-20T00:00:00.000Z/2016-11-08T00:00:00.000Z"] }
{ "queryType": "timeseries", "dataSource": "binnedSessions", "granularity": "all", "filter": { "type": "selector", "dimension": "zoneName", "value": "KTW1M Nera Putrajaya (MESH)"}, "aggregations": [ {"type": "count", "name": "totalCount"}, {"type": "longSum", "fieldName":"traffic", "name": "traffic"} ], "intervals": ["2016-06-20T00:00:00.000Z/2016-11-08T00:00:00.000Z"] }...
Linbo Jin
8 nov 2016, 22:05:348/11/16
a Druid User
Hi David,
Update, unfortunately, the restarted indexing task hang after java.lang.OutOfMemoryError: GC overhead limit exceeded and it stop indexing any data.
Will supervisor kill it and restart new one, it already hang for 3 hours?
log:
2016-11-09T00:31:33,201 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T15:58:19.706Z_b2f1ecf788794aca850920a52d1709671]
2016-11-09T00:31:33,202 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-11-02T00:00:00.000Z_2016-11-03T00:00:00.000Z_2016-11-08T09:29:06.077Z].
2016-11-09T00:31:33,202 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-06T00:00:00.000Z_2016-07-07T00:00:00.000Z_2016-11-08T10:35:03.291Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T16:04:31.284Z_6bbd570da7ca414796f33fb2b4c3c5a02]
2016-11-09T00:31:33,202 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-06T00:00:00.000Z_2016-07-07T00:00:00.000Z_2016-11-08T10:35:03.291Z].
2016-11-09T00:31:33,203 INFO [appenderator_persist_0] io.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[binnedSessions_2016-07-31T00:00:00.000Z_2016-08-01T00:00:00.000Z_2016-11-08T11:07:08.339Z] at path[/druid/segments/data-8ce3acae.node.rsa:8100/data-8ce3acae.node.rsa:8100_indexer-executor__default_tier_2016-11-08T16:04:31.284Z_6bbd570da7ca414796f33fb2b4c3c5a02]
2016-11-09T00:31:33,203 INFO [appenderator_persist_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Removing sink for segment[binnedSessions_2016-07-31T00:00:00.000Z_2016-08-01T00:00:00.000Z_2016-11-08T11:07:08.339Z].
2016-11-09T00:31:33,202 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,203 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,203 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,203 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,203 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,203 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,203 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,203 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 ERROR [task-runner-0-priority-0] com.google.common.util.concurrent.Futures$CombinedFuture - input future failed.
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,204 WARN [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Unable to abandon existing segments during close()
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
at io.druid.segment.realtime.appenderator.AppenderatorImpl.close(AppenderatorImpl.java:600) [druid-server-0.9.2-rc2.jar:0.9.2-rc2]
at io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:544) [druid-kafka-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]
at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_74]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_74]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,206 INFO [task-runner-0-priority-0] io.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider - Unregistering chat handler[index_kafka_binnedSessions_3aa45483b4be612_nafbmbgb]
2016-11-09T00:31:33,208 ERROR [task-runner-0-priority-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Uncaught Throwable while running task[KafkaIndexTask{id=index_kafka_binnedSessions_3aa45483b4be612_nafbmbgb, type=index_kafka, dataSource=binnedSessions}]
java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,208 ERROR [main] io.druid.cli.CliPeon - Error when starting up. Failing.
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]
at io.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:211) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]
at io.druid.cli.CliPeon.run(CliPeon.java:287) [druid-services-0.9.2-rc2.jar:0.9.2-rc2]
at io.druid.cli.Main.main(Main.java:106) [druid-services-0.9.2-rc2.jar:0.9.2-rc2]
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:?]
at io.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:208) ~[druid-indexing-service-0.9.2-rc2.jar:0.9.2-rc2]
... 2 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2016-11-09T00:31:33,226 INFO [Thread-44] io.druid.cli.CliPeon - Running shutdown hook
2016-11-09T00:31:33,226 INFO [Thread-44] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking stop method[public void io.druid.server.coordination.AbstractDataSegmentAnnouncer.stop()] on object[io.druid.server.coordination.BatchDataSegmentAnnouncer@3b4ef59f].
2016-11-09T00:31:33,226 INFO [Thread-44] io.druid.server.coordination.AbstractDataSegmentAnnouncer - Stopping class io.druid.server.coordination.BatchDataSegmentAnnouncer with config[io.druid.server.initialization.ZkPathsConfig@22e2266d]
2016-11-09T00:31:33,226 INFO [Thread-44] io.druid.curator.announcement.Announcer - unannouncing [/druid/announcements/data-8ce3acae.node.rsa:8100]
On Wednesday, November 9, 2016 at 10:13:53 AM UTC+8, David Lim wrote:
...
David Lim
9 nov 2016, 2:35:499/11/16
a Druid User
Hey Linbo,
The supervisor won't kill the task until it tries communicating with it and it fails to respond, which generally won't happen until the end of the taskDuration. If the task is just sitting there garbage collecting all day, you can manually kill it with an HTTP POST to http://<OVERLORD_IP>:<port>/druid/indexer/v1/task/{taskId}/shutdown. Keep in mind that this will result in the supervisor recreating the task which will likely OOM again.
It looks like your peons are under higher than normal memory pressure because of the sheer number of segments they're building as you work through the backlog from the last few months. If possible, I'd suggest increasing the heap size for the peons at least temporarily. 1G heap for peons is a fairly memory constrained configuration.
How large (in MB) are your daily segments?
The supervisor won't kill the task until it tries communicating with it and it fails to respond, which generally won't happen until the end of the taskDuration. If the task is just sitting there garbage collecting all day, you can manually kill it with an HTTP POST to http://<OVERLORD_IP>:<port>/druid/indexer/v1/task/{taskId}/shutdown. Keep in mind that this will result in the supervisor recreating the task which will likely OOM again.
It looks like your peons are under higher than normal memory pressure because of the sheer number of segments they're building as you work through the backlog from the last few months. If possible, I'd suggest increasing the heap size for the peons at least temporarily. 1G heap for peons is a fairly memory constrained configuration.
How large (in MB) are your daily segments?
...
Linbo Jin
9 nov 2016, 2:55:479/11/16
a Druid User
Hi, David,
Yes, you are right. I killed the task and the new task OOM again.
1. Our daily segment is around 13MB, "2 shards (12.3 MB) for 2016-11-06T00 to 2016-11-07T00"
2. We decided to reduce taskDuration into 2h as you suggested and test it. It means there will be at least 12 vertical segments for each day, do you think it will affect query performance?
Thanks for your helps.
Best wishes.
By Linbo
On Wednesday, November 9, 2016 at 3:35:49 PM UTC+8, David Lim wrote:
Hey Linbo,
David Lim
9 nov 2016, 3:36:129/11/16
a Druid User
Ah okay, 13MB is still on the small side for segments. Probably a good segment granularity for your data would be monthly, once you start looking into running merge tasks.
Yes, having many small segments will have an affect on query performance, but how significantly would depend on what type of queries you issue. I would probably first focus on getting realtime ingestion stable and then look into setting up a merge task workflow to generate monthly segments.
Yes, having many small segments will have an affect on query performance, but how significantly would depend on what type of queries you issue. I would probably first focus on getting realtime ingestion stable and then look into setting up a merge task workflow to generate monthly segments.
Responder a todos
Responder al autor
Reenviar
0 mensajes nuevos