Getting exception in coordinator and historicals

[ss...@twitter.com]

2017-04-21T18:54:25,177 ERROR [LookupCoordinatorManager--3] io.druid.server.lookup.cache.LookupCoordinatorManager - Error submitting to [http://smf1-eug-24-sr1.prod.twitter.com:31887/druid/listen/v1/lookups]: {class=io.druid.server.lookup.cache.LookupCoordinatorManager, exceptionType=class java.io.IOException, exceptionMessage=Bad update request to [http://smf1-eug-24-sr1.prod.xxx.com:31887/druid/listen/v1/lookups] : [500] : [Internal Server Error]  Response: [:)

��error�Unexpected token (END_OBJECT), expected FIELD_NAME: missing property 'type' that is to contain type id  (for class io.druid.query.lookup.LookupExtractorFactory)

 at [Source: N/A; line: -1, column: -1]��]}

java.io.IOException: Bad update request to [http://smf1-eug-24-sr1.prod.twitter.com:31887/druid/listen/v1/lookups] : [500] : [Internal Server Error]  Response: [:)

��error�Unexpected token (END_OBJECT), expected FIELD_NAME: missing property 'type' that is to contain type id  (for class io.druid.query.lookup.LookupExtractorFactory)

 at [Source: N/A; line: -1, column: -1]��]

at io.druid.server.lookup.cache.LookupCoordinatorManager.updateAllOnHost(LookupCoordinatorManager.java:228) ~[druid-server-0.9.2.jar:0.9.2]

at io.druid.server.lookup.cache.LookupCoordinatorManager$5.run(LookupCoordinatorManager.java:371) [druid-server-0.9.2.jar:0.9.2]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_121]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_121]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_121]

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

We are running on 0.9.2 and seeing this on both historicals and coordinator. This started happening more recently.

[Himanshu Gupta]

My guess is that you've given a lookup spec to the coordinator which does not have "type" . 

Can you do , 

"GET /druid/coordinator/v1/lookups" to get all tiers and then "GET /druid/coordinator/v1/lookups/{tier}" for all tiers and verify that all the specs are correct ?

-- Himanshu

[Mallik Bathula]

Thx Himanshu, I do have a similar issue. The following is my coordinator status 

curl http://dal-appblx085-06.prod.walmart.com:8081/druid/coordinator/v1/lookups/__default/

["city_name"]

curl http://dal-appblx085-06.prod.walmart.com:8081/druid/coordinator/v1/lookups/__default/city_name/

{"version":"v0","lookupExtractorFactory":{"type":"cachedNamespace","extractionNamespace":{"type":"uri","uri":"file:/app/gec/edw/druid/lookup/city.csv","namespaceParseSpec":{"format":"csv","columns":["key","value"]},"pollPeriod":"PT5H"},"firstCacheTimeout":0,"injective":true}}