Problem with Nested JSON (flattenSpec) in Tranquility 0.8.2

Saeed Shariati

unread,

Jul 5, 2016, 9:50:18 AM7/5/16

to Druid Development

Dear professions!

I'm trying to parse Nested JSON file in Tranquility 0.8.2 server and can't get any result. I tested the system in different ways and didn't get any result. Even by misspelling "flattenSpec" I won't get any result and this field will Ignore simply.

Does Imply 1.3.0 already support flattenSpec?

I highly appreciate if you could give me some advises where to look.

Here is my configuration file:

{
  "dataSources" : {
    "temp" : {
      "spec" : {
        "dataSchema" : {
          "dataSource" : "temp",
          "parser" : {
            "type" : "string",
            "parseSpec": {
              "format": "json",
              "flattenSpec": {
                "useFieldDiscovery": true,
                "fields": [
                  {
                    "type": "path",
                    "name": "sensor",
                    "expr": "$.data.sensor"
                  }                  
                ]
              },
              "dimensionsSpec" : {
               "dimensions" : ["sensor"],
               "dimensionsExclusions": ["timestamp","temperature"]
              },
              "timestampSpec" : {
               "format" : "auto",
               "column" : "timestamp"
              }
            }
          },
          "granularitySpec" : {
            "type" : "uniform",
            "segmentGranularity" : "hour",
            "queryGranularity" : "none"
          },
          "metricsSpec" : [
            {"name": "Count", "type": "count"},
            {"name": "TempMin", "type": "doubleMin", "fieldName": "temperature"},
            {"name": "TempMax", "type": "doubleMax", "fieldName": "temperature"}
          ]
        },
        "ioConfig" : {
          "type" : "realtime"
        },
        "tuningConfig" : {
          "type" : "realtime",
          "maxRowsInMemory" : "100000",
          "intermediatePersistPeriod" : "PT10M",
          "windowPeriod" : "PT10M"
        }
      },
      "properties" : {
        "task.partitions" : "1",
        "task.replicants" : "1"
      }
    }
  },
  "properties" : {
    "zookeeper.connect" : "localhost",
    "druid.discovery.curator.path" : "/druid/discovery",
    "druid.selectors.indexing.serviceName" : "druid/overlord",
    "http.port" : "8200",
    "http.threads" : "40"
  }
}

And here is the data:

{"timestamp": "2016-07-03T09:11:42Z", "data":{"sensor": "sensor4"}, "tag":"tag5", "temperature":25}

After all, I get null value for sensor field.

Any clue would be highly appreciated!

Regards,

Saeed

Gian Merlino

unread,

Jul 5, 2016, 12:03:51 PM7/5/16

to druid-de...@googlegroups.com

Hey Saeed,

Are you using the "text/plain" API or the direct object "application/json" API? (see https://github.com/druid-io/tranquility/blob/master/docs/server.md#http-api). The flattenSpec, and actually all other properties of the parser other than time format and dimensions, are only respected for the "text/plain" API. The "application/json" API bypasses the parser.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/91dd8219-a51f-4989-8e53-fd4b9bc47885%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Saeed Shariati

unread,

Jul 5, 2016, 3:00:29 PM7/5/16

to druid-de...@googlegroups.com

Hi Gian!

Wow! Thanks you! It was great help!

I don't know how I missed that.

Thanks again for your time.

Cheers,

Saeed

You received this message because you are subscribed to a topic in the Google Groups "Druid Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-development/k5A1FDn7tRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-developm...@googlegroups.com.

To post to this group, send email to druid-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/CACZNdYCH%2Bq20VcCoTMjiSaeOCJ%3DCgnkb4fu9T%3DZTk0rmJ97kfA%40mail.gmail.com.

Reply all

Reply to author

Forward