Trouble with nested aggregations

[Josh Hemann]

Hi,

I am looking for examples of doing nested aggregations in elastich, as I am currently getting a 400 (bad request) response using my current approach. My documents are tweets, along with a timestamp and a sentiment score. Here is how the namespace looks:

 

(ns tweets.core
  (:require [clojurewerkz.elastisch.rest :as es-rest]
            [clojurewerkz.elastisch.rest.document :as es-doc]
            [clojurewerkz.elastisch.query :as es-q]
            [clojurewerkz.elastisch.rest.response :as es-rsp]
            [clojurewerkz.elastisch.aggregation :as es-agg]
            [clojure.pprint :as pp]))

(clojure.core/refer 'clojure.core)

This code works as expected:

(defn get-some-tweets
  [kwd]
  (let [conn (es-rest/connect local-es-host)
        res  (es-doc/search conn indices mapping
               :query (es-q/filtered :query (es-q/query-string :query kwd)
                        :filter (es-q/range :timestamp :from "2015-11-19"
                                                       :to "2015-11-24")))]
    (pp/pprint (take 5 res))))

And I can use a date_histogram to get weekly counts of tweets like this:

(defn counts-by-week
  [kwd]
  (let [conn (es-rest/connect local-es-host)
        res  (es-doc/search conn indices mapping
               :query (es-q/query-string :query kwd)
               :aggregations {:weekly-data (es-agg/date-histogram :timestamp "week")})]
    (pp/pprint (get-in res [:aggregations]))))

Here is the nested aggregation that is not working. I am trying to bucket tweets by week, and then get sentiment stats within each weekly bucket:

(defn avg-weekly-sentiment
  [kwd]
  (let [conn (es-rest/connect local-es-host)
        res  (es-doc/search conn indices mapping
               :query (es-q/query-string :query kwd)
               :aggregations {:weekly-data (es-agg/date-histogram :timestamp "week")
                              :aggregations {:avg_sentiment (es-agg/avg "sentiment")}})]
    (pp/pprint (get-in res [:aggregations]))))

I have played with facets too without success. I am having trouble mapping the ES documentation to elastich and hoping someone has an example or sees an obvious issue with my attempt. I am fairly new to Clojure too...

Thanks,

Josh

[Michael Klishin]

2016-02-02 17:29 GMT+03:00 Josh Hemann <josh....@gmail.com>:

I am having trouble mapping the ES documentation to elastich and hoping someone has an example or sees an obvious issue with my attempt. I am fairly new to Clojure too...

With the REST client, all data structures you pass are converted to JSON as is,

so there really isn't much to mapping ES docs to Elastisch.

Native client has to do a lot more work and nested aggregations are not supported there.

--

MK

http://github.com/michaelklishin
http://twitter.com/michaelklishin

[Josh Hemann]

Hi Michael,

You may be on to something regarding connection over HTTP vs native client, but I am confused by your answer. I am in fact using the HTTP connection and not native, and nested aggregations were added to the native client in elastich 2.2.0. I assumed this was supported via HTTP, but I'll try the native client.

Thanks,

Josh

[Josh Hemann]

OK, just closing out this thread... Michael's pointer towards native vs HTTP client was the issue. By switching to the native client I was able to calculate stats-by-week. For the updated code, please see the Stack Overflow question I started (and ended up answering).

Thanks,

Josh