isWhitelist=false in listfiltered dimension spec query causing exception

64 views

Skip to first unread message

Ben Vogan

unread,

Sep 22, 2016, 12:02:57 AM9/22/16

to druid...@googlegroups.com

Hi all,

I was trying to play around with a multi-valued dimension when I encountered this problem.

What I am trying to accomplish is a drill-down query mechanism. In the UI I would present a normal group by query result at the top level, the user could then "drill-down" on a particular label (e.g. Android). The resulting query would perform a group by query with a filter of Android, showing the count of all of the other tags (i.e. not showing Android). My understanding is that this could be accomplished with a standard filter for Android, and then a listFiltered dimension spec with Android blacklisted. I could keep drilling down by adding more and more tags to the filter (using AND), and the listFiltered blacklist.

Setting isWhitelist to true makes the below query not return an error, although obviously not with the behavior I am looking for:

{

"filter": {

"type": "selector",

"dimension": "labels",

"value": "Android"

"dimensions": [

{

"type" : "listFiltered",

"delegate" : {

"type": "default",

"dimension": "labels",

"outputName": "labels"

"values" : ["Android"],

"isWhitelist" : "false"

}

"aggregations": [

{

"type": "count",

"name": "count"

}

"intervals": "2016-09-15T00:00:00+00:00/2016-09-22T02:52:05+00:00",

"limitSpec": {

"limit": 50000,

"type": "default",

"columns": [

{

"direction": "descending",

"dimension": "count"

}

]

"granularity": {

"timeZone": "UTC",

"type": "period",

"period": "P1D"

"postAggregations": [],

"queryType": "groupBy",

"dataSource": "sor_desk_com_cases"

}

Broker Logs:

2016-09-22T03:35:41,920 INFO [qtp1111100957-198[groupBy_sor_desk_com_cases_5d12877c-91c2-4175-a017-2431340ba74c]] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://druid-data001.internal.shopkick.com:8103

2016-09-22T03:35:41,943 INFO [HttpClient-Netty-Worker-79] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-09-22T03:35:41.943Z","service":"druid/broker","host":"druid-query002.internal.shopkick.com:8082","metric":"query/node/ttfb","value":28,"dataSource":"sor_desk_com_cases","duration":"PT528725S","hasFilters":"true","id":"5d12877c-91c2-4175-a017-2431340ba74c","interval":["2016-09-16T00:00:00.000Z/2016-09-22T02:52:05.000Z"],"numComplexMetrics":"0","numDimensions":"1","numMetrics":"1","server":"druid-data001.internal.shopkick.com:8083","type":"groupBy","version":"0.9.1.1"}]

2016-09-22T03:35:41,943 INFO [HttpClient-Netty-Worker-79] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-09-22T03:35:41.943Z","service":"druid/broker","host":"druid-query002.internal.shopkick.com:8082","metric":"query/node/time","value":28,"dataSource":"sor_desk_com_cases","duration":"PT528725S","hasFilters":"true","id":"5d12877c-91c2-4175-a017-2431340ba74c","interval":["2016-09-16T00:00:00.000Z/2016-09-22T02:52:05.000Z"],"numComplexMetrics":"0","numDimensions":"1","numMetrics":"1","server":"druid-data001.internal.shopkick.com:8083","type":"groupBy","version":"0.9.1.1"}]

2016-09-22T03:35:41,943 INFO [HttpClient-Netty-Worker-79] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-09-22T03:35:41.943Z","service":"druid/broker","host":"druid-query002.internal.shopkick.com:8082","metric":"query/node/bytes","value":14,"dataSource":"sor_desk_com_cases","duration":"PT528725S","hasFilters":"true","id":"5d12877c-91c2-4175-a017-2431340ba74c","interval":["2016-09-16T00:00:00.000Z/2016-09-22T02:52:05.000Z"],"numComplexMetrics":"0","numDimensions":"1","numMetrics":"1","server":"druid-data001.internal.shopkick.com:8083","type":"groupBy","version":"0.9.1.1"}]

2016-09-22T03:35:41,944 INFO [qtp1111100957-198[groupBy_sor_desk_com_cases_5d12877c-91c2-4175-a017-2431340ba74c]] io.druid.server.QueryResource - Unknown exception [5d12877c-91c2-4175-a017-2431340ba74c]

2016-09-22T03:35:41,944 INFO [qtp1111100957-198[groupBy_sor_desk_com_cases_5d12877c-91c2-4175-a017-2431340ba74c]] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-09-22T03:35:41.944Z","service":"druid/broker","host":"druid-query002.internal.shopkick.com:8082","metric":"query/time","value":33,"context":"{\"queryId\":\"5d12877c-91c2-4175-a017-2431340ba74c\",\"timeout\":300000}","dataSource":"sor_desk_com_cases","duration":"PT615125S","hasFilters":"true","id":"5d12877c-91c2-4175-a017-2431340ba74c","interval":["2016-09-15T00:00:00.000Z/2016-09-22T02:52:05.000Z"],"remoteAddress":"0:0:0:0:0:0:0:1","success":"false","type":"groupBy","version":"0.9.1.1"}]

2016-09-22T03:35:41,954 INFO [HttpClient-Netty-Worker-48] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-09-22T03:35:41.954Z","service":"druid/broker","host":"druid-query002.internal.shopkick.com:8082","metric":"query/node/ttfb","value":31,"dataSource":"sor_desk_com_cases","duration":"PT615125S","hasFilters":"true","id":"5d12877c-91c2-4175-a017-2431340ba74c","interval":["2016-09-15T00:00:00.000Z/2016-09-22T02:52:05.000Z"],"numComplexMetrics":"0","numDimensions":"1","numMetrics":"1","server":"druid-data002.internal.shopkick.com:8083","type":"groupBy","version":"0.9.1.1"}]

2016-09-22T03:35:41,954 INFO [HttpClient-Netty-Worker-48] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-09-22T03:35:41.954Z","service":"druid/broker","host":"druid-query002.internal.shopkick.com:8082","metric":"query/node/time","value":31,"dataSource":"sor_desk_com_cases","duration":"PT615125S","hasFilters":"true","id":"5d12877c-91c2-4175-a017-2431340ba74c","interval":["2016-09-15T00:00:00.000Z/2016-09-22T02:52:05.000Z"],"numComplexMetrics":"0","numDimensions":"1","numMetrics":"1","server":"druid-data002.internal.shopkick.com:8083","type":"groupBy","version":"0.9.1.1"}]

2016-09-22T03:35:41,954 INFO [HttpClient-Netty-Worker-48] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-09-22T03:35:41.954Z","service":"druid/broker","host":"druid-query002.internal.shopkick.com:8082","metric":"query/node/bytes","value":14,"dataSource":"sor_desk_com_cases","duration":"PT615125S","hasFilters":"true","id":"5d12877c-91c2-4175-a017-2431340ba74c","interval":["2016-09-15T00:00:00.000Z/2016-09-22T02:52:05.000Z"],"numComplexMetrics":"0","numDimensions":"1","numMetrics":"1","server":"druid-data002.internal.shopkick.com:8083","type":"groupBy","version":"0.9.1.1"}]

2016-09-22T03:35:41,969 INFO [HttpClient-Netty-Worker-112] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-09-22T03:35:41.968Z","service":"druid/broker","host":"druid-query002.internal.shopkick.com:8082","metric":"query/node/ttfb","value":52,"dataSource":"sor_desk_com_cases","duration":"PT10325S","hasFilters":"true","id":"5d12877c-91c2-4175-a017-2431340ba74c","interval":["2016-09-22T00:00:00.000Z/2016-09-22T02:52:05.000Z"],"numComplexMetrics":"0","numDimensions":"1","numMetrics":"1","server":"druid-data001.internal.shopkick.com:8103","type":"groupBy","version":"0.9.1.1"}]

2016-09-22T03:35:41,969 INFO [HttpClient-Netty-Worker-112] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-09-22T03:35:41.969Z","service":"druid/broker","host":"druid-query002.internal.shopkick.com:8082","metric":"query/node/time","value":53,"dataSource":"sor_desk_com_cases","duration":"PT10325S","hasFilters":"true","id":"5d12877c-91c2-4175-a017-2431340ba74c","interval":["2016-09-22T00:00:00.000Z/2016-09-22T02:52:05.000Z"],"numComplexMetrics":"0","numDimensions":"1","numMetrics":"1","server":"druid-data001.internal.shopkick.com:8103","type":"groupBy","version":"0.9.1.1"}]

Historical Logs

2016-09-22T03:30:02,127 INFO [groupBy_sor_desk_com_cases_[2016-09-22T00:00:00.000Z/2016-09-22T02:52:05.000Z]] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2016-09-22T03:30:02.127Z","service":"druid/historical","host":"druid-data001.internal.shopkick.com:8083","metric":"query/wait/time","value":2,"dataSource":"sor_desk_com_cases","duration":"PT10325S","hasFilters":"true","id":"f4a0967f-95f1-4ffe-89a2-4c7920d7e33f","interval":["2016-09-22T00:00:00.000Z/2016-09-22T02:52:05.000Z"],"numComplexMetrics":"0","numDimensions":"1","numMetrics":"1","segment":"sor_desk_com_cases_2016-09-22T00:00:00.000Z_2016-09-23T00:00:00.000Z_2016-09-22T00:30:03.311Z_2","status":"failed","type":"groupBy","version":"0.9.1.1"}]

2016-09-22T03:30:02,125 ERROR [processing-1] io.druid.query.GroupByMergedQueryRunner - Exception with one of the sequences!

java.lang.ArrayIndexOutOfBoundsException: 1

at io.druid.query.dimension.ListFilteredDimensionSpec.decorate(ListFilteredDimensionSpec.java:96) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at io.druid.segment.QueryableIndexStorageAdapter$CursorSequenceBuilder$1$1.makeDimensionSelector(QueryableIndexStorageAdapter.java:374) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at io.druid.query.groupby.GroupByQueryEngine$RowIterator.<init>(GroupByQueryEngine.java:328) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at io.druid.query.groupby.GroupByQueryEngine$1$1.make(GroupByQueryEngine.java:121) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at io.druid.query.groupby.GroupByQueryEngine$1$1.make(GroupByQueryEngine.java:117) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at com.metamx.common.guava.BaseSequence.toYielder(BaseSequence.java:78) ~[java-util-0.27.9.jar:?]

at com.metamx.common.guava.ConcatSequence.makeYielder(ConcatSequence.java:93) ~[java-util-0.27.9.jar:?]

at com.metamx.common.guava.ConcatSequence.toYielder(ConcatSequence.java:72) ~[java-util-0.27.9.jar:?]

at com.metamx.common.guava.ResourceClosingSequence.toYielder(ResourceClosingSequence.java:41) ~[java-util-0.27.9.jar:?]

at com.metamx.common.guava.YieldingSequenceBase.accumulate(YieldingSequenceBase.java:34) ~[java-util-0.27.9.jar:?]

at io.druid.query.MetricsEmittingQueryRunner$1.accumulate(MetricsEmittingQueryRunner.java:118) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at com.metamx.common.guava.MappedSequence.accumulate(MappedSequence.java:40) ~[java-util-0.27.9.jar:?]

at com.metamx.common.guava.Sequences$1.accumulate(Sequences.java:90) ~[java-util-0.27.9.jar:?]

at io.druid.query.MetricsEmittingQueryRunner$1.accumulate(MetricsEmittingQueryRunner.java:118) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at io.druid.query.spec.SpecificSegmentQueryRunner$2$1.call(SpecificSegmentQueryRunner.java:87) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at io.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:171) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at io.druid.query.spec.SpecificSegmentQueryRunner.access$400(SpecificSegmentQueryRunner.java:41) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at io.druid.query.spec.SpecificSegmentQueryRunner$2.doItNamed(SpecificSegmentQueryRunner.java:162) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at io.druid.query.spec.SpecificSegmentQueryRunner$2.accumulate(SpecificSegmentQueryRunner.java:80) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at io.druid.query.CPUTimeMetricQueryRunner$1.accumulate(CPUTimeMetricQueryRunner.java:81) ~[druid-processing-0.9.1.1.jar:0.9.1.1]

at com.metamx.common.guava.Sequences$1.accumulate(Sequences.java:90) ~[java-util-0.27.9.jar:?]

at io.druid.query.GroupByMergedQueryRunner$1$1.call(GroupByMergedQueryRunner.java:127) [druid-processing-0.9.1.1.jar:0.9.1.1]

at io.druid.query.GroupByMergedQueryRunner$1$1.call(GroupByMergedQueryRunner.java:117) [druid-processing-0.9.1.1.jar:0.9.1.1]

at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_101]

at io.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:271) [druid-processing-0.9.1.1.jar:0.9.1.1]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]

at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]

2016-09-22T03:30:02,125 ERROR [processing-0] io.druid.query.GroupByMergedQueryRunner - Exception with one of the sequences!

java.lang.ArrayIndexOutOfBoundsException: 5