insane data transfer out from s3

148 views

Skip to first unread message

Prashant Deva

unread,

Apr 23, 2016, 8:07:58 PM4/23/16

to Druid User

We have 4 historical nodes with about 60gb data each.

We see that this month datatransfer out from s3 for the druid bucket has been over 1,700 GB this month so far (and its only 23rd)!

Using druid 0.8.3.

Is there a reason why druid is transferring so much data from s3?

What is it transferring anyway?

Prashant Deva

unread,

Apr 23, 2016, 8:11:39 PM4/23/16

to Druid User

investigating this further, in one of the historicals which has a ton of 'loading segments' msgs in the logs, I see a ton of these exceptions:

java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-04-20T20:00:00.000Z_2016-04-20T21:00:00.000Z_2016-04-20T20:00:00.000Z at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?] at io.druid.server.coordination.BaseZkCoordinator$1$1.execute(BaseZkCoordinator.java:139) ~[druid-server-0.8.3.jar:0.8.3] at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:186) ~[druid-server-0.8.3.jar:0.8.3] at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) ~[druid-server-0.8.3.jar:0.8.3] at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:115) [druid-server-0.8.3.jar:0.8.3] at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:518) [curator-recipes-2.9.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:512) [curator-recipes-2.9.1.jar:?] at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.9.1.jar:?] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?] at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.9.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:509) [curator-recipes-2.9.1.jar:?] at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.9.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:766) [curator-recipes-2.9.1.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77] Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-04-20T20:00:00.000Z_2016-04-20T21:00:00.000Z_2016-04-20T20:00:00.000Z at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) ~[zookeeper-3.4.6.jar:3.4.6-1569965] at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:242) ~[curator-framework-2.9.1.jar:?] at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:236) ~[curator-framework-2.9.1.jar:?] at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.9.1.jar:?] at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:232) ~[curator-framework-2.9.1.jar:?] at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:217) ~[curator-framework-2.9.1.jar:?] at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:41) ~[curator-framework-2.9.1.jar:?] at io.druid.server.coordination.BaseZkCoordinator$1$1.execute(BaseZkCoordinator.java:126) ~[druid-server-0.8.3.jar:0.8.3] ... 18 more

Prashant Deva

unread,

Apr 23, 2016, 8:57:34 PM4/23/16

to Druid User

While restarting the historical node seems to have fixed the exception, looking at the logs of all the historical nodes, I see tons of these msgs:

Segment [%s] size is different than expected size.

What does this mean?

I am ingesting through realtime nodes.

On Saturday, April 23, 2016 at 5:11:39 PM UTC-7, Prashant Deva wrote:

investigating this further, in one of the historicals which has a ton of 'loading segments' msgs in the logs, I see a ton of these exceptions:

java.lang.RuntimeException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-04-20T20:00:00.000Z_2016-04-20T21:00:00.000Z_2016-04-20T20:00:00.000Z at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?] at io.druid.server.coordination.BaseZkCoordinator$1$1.execute(BaseZkCoordinator.java:139) ~[druid-server-0.8.3.jar:0.8.3] at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:186) ~[druid-server-0.8.3.jar:0.8.3] at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) ~[druid-server-0.8.3.jar:0.8.3] at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:115) [druid-server-0.8.3.jar:0.8.3] at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:518) [curator-recipes-2.9.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:512) [curator-recipes-2.9.1.jar:?] at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.9.1.jar:?] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?] at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.9.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:509) [curator-recipes-2.9.1.jar:?] at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.9.1.jar:?] at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:766) [curator-recipes-2.9.1.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511

...

Prashant Deva

unread,

Apr 23, 2016, 9:11:20 PM4/23/16

to Druid User

Actually it seems even after rebooting i continue getting this exception:

2016-04-24 01:07:30,680 ERROR i.d.s.c.ZkCoordinator [ZkCoordinator-0] Failed to delete zNode[/druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z], but ignoring exception.

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z

at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) ~[zookeeper-3.4.6.jar:3.4.6-1569965]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:242) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:236) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:232) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:217) ~[curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:41) ~[curator-framework-2.9.1.jar:?]

at io.druid.server.coordination.BaseZkCoordinator$1$1.execute(BaseZkCoordinator.java:133) [druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.ZkCoordinator.addSegment(ZkCoordinator.java:186) [druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.SegmentChangeRequestLoad.go(SegmentChangeRequestLoad.java:42) [druid-server-0.8.3.jar:0.8.3]

at io.druid.server.coordination.BaseZkCoordinator$1.childEvent(BaseZkCoordinator.java:115) [druid-server-0.8.3.jar:0.8.3]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:518) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:512) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) [curator-framework-2.9.1.jar:?]

at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [guava-16.0.1.jar:?]

at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [curator-framework-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:509) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-2.9.1.jar:?]

at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:766) [curator-recipes-2.9.1.jar:?]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_77]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_77]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]

2016-04-24 01:07:30,694 ERROR i.d.s.c.ZkCoordinator [ZkCoordinator-0] Exception while removing zNode[/druidprod/base/loadQueue/10.0.0.33:8080/dripstat_2016-03-09T15:00:00.000Z_2016-03-09T16:00:00.000Z_2016-03-09T15:00:00.000Z]