Exception in middle manager logs

[Deepak]

Hey Guys,

I'm observing few exceptions in the middle manager logs.  The events seem to be getting indexed and I can see them in the druid overlord UI / pivot UI, also the handoff to historical nodes seem to be happening successfully. 

I've also seen the middle manager service shutdown itself after successfully indexing events for some time. The exceptions I found in the logs are below. Any idea why this is happening ?

[ERROR] 2015-11-23 19:05:07.550 [Thread-29] Lifecycle$AnnotationBasedHandler - Exception when stopping met

hod[public void io.druid.curator.discovery.ServerDiscoverySelector.stop() throws java.io.IOException] on o

bject[io.druid.curator.discovery.ServerDiscoverySelector@37a0ec3c]

java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0

_45]

        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]

        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.stop(Lifecycle.java:343) [java-uti

l-0.27.0.jar:?]

        at com.metamx.common.lifecycle.Lifecycle.stop(Lifecycle.java:267) [java-util-0.27.0.jar:?]

        at io.druid.cli.CliPeon$2.run(CliPeon.java:220) [druid-services-0.8.1-iap2.jar:0.8.1-iap2]

        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]

Caused by: java.lang.IllegalStateException: Already closed or has not been started

        at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[guava-16.0.1.jar:?]

        at org.apache.curator.x.discovery.details.ServiceCacheImpl.close(ServiceCacheImpl.java:91) ~[curat

or-x-discovery-2.8.0.jar:?]

[ERROR] 2015-11-23 18:57:07.173 [Thread-29] ServiceDiscoveryImpl - Could not unregister instance: stream-a

udit-datasource-49-0000-0000

java.lang.IllegalStateException: instance must be started before calling this method

        at com.google.common.base.Preconditions.checkState(Preconditions.java:176) ~[guava-16.0.1.jar:?]

        at org.apache.curator.framework.imps.CuratorFrameworkImpl.delete(CuratorFrameworkImpl.java:347) ~[

curator-framework-2.8.0.jar:?]

        at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalUnregisterService(ServiceDi

scoveryImpl.java:505) ~[curator-x-discovery-2.8.0.jar:?]

        at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.close(ServiceDiscoveryImpl.java:155

) [curator-x-discovery-2.8.0.jar:?]

        at io.druid.curator.discovery.DiscoveryModule$5.stop(DiscoveryModule.java:222) [druid-server-0.8.1

-iap2.jar:0.8.1-iap2]

        at com.metamx.common.lifecycle.Lifecycle.stop(Lifecycle.java:267) [java-util-0.27.0.jar:?]

        at io.druid.cli.CliPeon$2.run(CliPeon.java:220) [druid-services-0.8.1-iap2.jar:0.8.1-iap2]

        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]

I'm using imply-1.0.0 package and have started the services using the imply supervisor scripts.

Thanks,

Deepak

[Fangjin Yang]

Hi Deepak, we'll take a look and update this thread.