Exceptions when using variance aggregator

[G. Y Narayan]

I am using Druid version 0.9.2 and am running the Druid components on a single host and using the following page http://druid.io/docs/latest/tutorials/quickstart.html to setup the nodes and test the variance and approxHistogramFold aggregators and stddev and quantile post aggregator. When I run the variance aggregator, it throws an exception. I am using the tranquility server and the example that comes with druid distribution to test ingestion. Ingestion happens without any issues, segments are formed and handed off the historical. I have the variance and approxHistogramFold aggregators defined in the ingestion spec (server.json file used by tranquility server). Can somebody help me with this?

Server.json (snippet):

          "metricsSpec" : [

            {

              "type" : "count",

              "name" : "count"

            },

            {

              "name" : "value_sum",

              "type" : "doubleSum",

              "fieldName" : "value"

            },

            {

              "fieldName" : "value",

              "name" : "value_min",

              "type" : "doubleMin"

            },

            {

              "type" : "doubleMax",

              "name" : "value_max",

              "fieldName" : "value"

            },

            {

              "type" : "variance",

              "name" : "value_var",

              "fieldName" : "value"

            },

            {

              "type" : "approxHistogram",

              "name" : "value_tiles",

              "fieldName" : "value"

            }

The task log from coordinator console:

{"task":"index_realtime_metrics_2017-03-28T07:31:00.000Z_0_0","payload":{"id":"index_realtime_metrics_2017-03-28T07:31:00.000Z_0_0","resource":{"availabilityGroup":"metrics-2017-03-28T07:31:00.000Z-0000","requiredCapacity":1},"spec":{"dataSchema":{"dataSource":"metrics","parser":{"type":"map","parseSpec":{"format":"json","timestampSpec":{"column":"timestamp","format":"millis","missingValue":null},"dimensionsSpec":{"dimensionExclusions":["count","timestamp","value_min","rowCount","value_tiles","value_max","value","value_var","value_sum"],"spatialDimensions":[]}}},"metricsSpec":[{"type":"count","name":"count"},{"type":"doubleSum","name":"value_sum","fieldName":"value"},{"type":"doubleMin","name":"value_min","fieldName":"value"},{"type":"doubleMax","name":"value_max","fieldName":"value"},{"type":"variance","name":"value_var","fieldName":"value","estimator":null,"inputType":"float"},{"type":"approxHistogram","name":"value_tiles","fieldName":"value","resolution":50,"numBuckets":7,"lowerLimit":"-Infinity","upperLimit":"Infinity"}],"granularitySpec":{"type":"uniform","segmentGranularity":"MINUTE","queryGranularity":{"type":"none"},"rollup":true,"intervals":null}},"ioConfig":{"type":"realtime","firehose":{"type":"clipped","delegate":{"type":"timed","delegate":{"type":"receiver","serviceName":"firehose:druid:overlord:metrics-091-0000-0000","bufferSize":100000},"shutoffTime":"2017-03-28T07:47:00.000Z"},"interval":"2017-03-28T07:31:00.000Z/2017-03-28T07:32:00.000Z"},"firehoseV2":null},"tuningConfig":{"type":"realtime","maxRowsInMemory":100000,"intermediatePersistPeriod":"PT10M","windowPeriod":"PT10M","basePersistDirectory":"/x/home/DRUID/var/tmp/1490686263720-0","versioningPolicy":{"type":"intervalStart"},"rejectionPolicy":{"type":"none"},"maxPendingPersists":0,"shardSpec":{"type":"linear","partitionNum":0},"indexSpec":{"bitmap":{"type":"concise"},"dimensionCompression":"lz4","metricCompression":"lz4","longEncoding":"longs"},"buildV9Directly":false,"persistThreadPriority":0,"mergeThreadPriority":0,"reportParseExceptions":false,"handoffConditionTimeout":0}},"context":null,"groupId":"index_realtime_metrics","dataSource":"metrics"}}

Query:

{

  "queryType": "timeseries",

  "dataSource": "metrics",

  "granularity": "all",

  "aggregations": [

    {

      "type": "count",

      "name": "count"

    },

    {

      "type": "doubleSum",

      "name": "TotalCount",

      "fieldName": "value_sum"

    },

    {

      "type": "approxHistogramFold",

      "name": "tiles_var",

      "fieldName": "value_tiles"

    },

    {

      "type": "variance",

      "name": "variance_var",

      "fieldName": "value_var"

    }

  ],

  "postAggregations": [

    {

      "type": "quantile",

      "name": "varp",

      "fieldName": "tiles_var",

      "probability": ".95"

    }

  ],

  "intervals": [

    "2017-03-27T07:00:00.000Z\/2017-03-28T00:00:00.000Z"

  ]

}

exception for the query:

curl -L -H'Content-Type: application/json' -XPOST --data-binary @ts3.q http://localhost:8082/druid/v2/?pretty

{

  "error" : "Unknown exception",

  "errorMessage" : "com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in START_OBJECT\n at [Source: java.io.SequenceInputStream@7fa5895e; line: -1, column: 87] (through reference chain: java.util.ArrayList[0])",

  "errorClass" : "java.lang.RuntimeException",

  "host" : null

}

error log on server:

017-03-28T08:28:57,812 ERROR [qtp538375433-152[timeseries_metrics_ea8e9e8a-da32-44a2-bd51-645d98c9e053]] io.druid.server.QueryResource - Exception handling request: {class=io.druid.server.QueryResource, exceptionType=class java.lang.RuntimeException, exceptionMessage=com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in START_OBJECT

 at [Source: java.io.SequenceInputStream@7fa5895e; line: -1, column: 87] (through reference chain: java.util.ArrayList[0]), exception=java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in START_OBJECT

 at [Source: java.io.SequenceInputStream@7fa5895e; line: -1, column: 87] (through reference chain: java.util.ArrayList[0]), query=TimeseriesQuery{dataSource='metrics', querySegmentSpec=LegacySegmentSpec{intervals=[2017-03-27T07:00:00.000Z/2017-03-28T00:00:00.000Z]}, descending=false, dimFilter=null, granularity='AllGranularity', aggregatorSpecs=[CountAggregatorFactory{name='count'}, DoubleSumAggregatorFactory{fieldName='value_sum', name='TotalCount'}, ApproximateHistogramFoldingAggregatorFactory{name='tiles_var', fieldName='value_tiles', resolution=50, numBuckets=7, lowerLimit=-Infinity, upperLimit=Infinity}, VarianceAggregatorFactory{fieldName='value_var', name='variance_var', isVariancePop='false', inputType='float'}], postAggregatorSpecs=[QuantilePostAggregator{probability=0.95, fieldName='tiles_var'}], context={queryId=ea8e9e8a-da32-44a2-bd51-645d98c9e053, timeout=300000}}, peer=127.0.0.1}

java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in START_OBJECT

 at [Source: java.io.SequenceInputStream@7fa5895e; line: -1, column: 87] (through reference chain: java.util.ArrayList[0])

  at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]

  at io.druid.client.DirectDruidClient$JsonParserIterator.next(DirectDruidClient.java:463) ~[druid-server-0.9.2.jar:0.9.2]

  at com.metamx.common.guava.BaseSequence.accumulate(BaseSequence.java:67) ~[java-util-0.27.10.jar:?]

  at com.metamx.common.guava.MappedSequence.accumulate(MappedSequence.java:40) ~[java-util-0.27.10.jar:?]

  at com.metamx.common.guava.MergeSequence.toYielder(MergeSequence.java:59) ~[java-util-0.27.10.jar:?]

  at com.metamx.common.guava.MergeSequence$2.accumulate(MergeSequence.java:66) ~[java-util-0.27.10.jar:?]

  at com.metamx.common.guava.MergeSequence$2.accumulate(MergeSequence.java:62) ~[java-util-0.27.10.jar:?]

  at com.metamx.common.guava.BaseSequence.accumulate(BaseSequence.java:67) ~[java-util-0.27.10.jar:?]

  at com.metamx.common.guava.MergeSequence.toYielder(MergeSequence.java:59) ~[java-util-0.27.10.jar:?]

  at com.metamx.common.guava.LazySequence.toYielder(LazySequence.java:43) ~[java-util-0.27.10.jar:?]

  at io.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:106) ~[druid-processing-0.9.2.jar:0.9.2]

  at io.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78) ~[druid-common-0.9.2.jar:0.9.2]

  at com.metamx.common.guava.MappedSequence.toYielder(MappedSequence.java:46) ~[java-util-0.27.10.jar:?]

  at io.druid.query.CPUTimeMetricQueryRunner$1.toYielder(CPUTimeMetricQueryRunner.java:93) ~[druid-processing-0.9.2.jar:0.9.2]

  at com.metamx.common.guava.Sequences$1.toYielder(Sequences.java:98) ~[java-util-0.27.10.jar:?]

  at io.druid.server.QueryResource.doPost(QueryResource.java:223) [druid-server-0.9.2.jar:0.9.2]

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]

  at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]

  at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [jersey-server-1.19.jar:1.19]

  at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [jersey-server-1.19.jar:1.19]

  at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [jersey-server-1.19.jar:1.19]

  at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [jersey-server-1.19.jar:1.19]

  at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [jersey-server-1.19.jar:1.19]

  at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19]

  at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [jersey-server-1.19.jar:1.19]

  at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19]

  at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19]

  at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19]

  at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19]

  at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19]

  at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19]

  at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19]

  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]

  at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) [guice-servlet-4.1.0.jar:?]

  at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) [guice-servlet-4.1.0.jar:?]

  at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) [guice-servlet-4.1.0.jar:?]

  at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-4.1.0.jar:?]

  at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [guice-servlet-4.1.0.jar:?]

  at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [guice-servlet-4.1.0.jar:?]

  at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) [guice-servlet-4.1.0.jar:?]

  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) [jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.server.Server.handle(Server.java:497) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]

  at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]

Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input in START_OBJECT

 at [Source: java.io.SequenceInputStream@7fa5895e; line: -1, column: 87] (through reference chain: java.util.ArrayList[0])

  at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:210) ~[jackson-databind-2.4.6.jar:2.4.6]

  at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:189) ~[jackson-databind-2.4.6.jar:2.4.6]

Thanks

-Narayan