Scroll Error, need help

[Francis Hitchens]

Hi,

Not much activity on the group but let's try any way.

I was using this snippet of code I found in the tests to scroll through a lot of query results...

(defn fetch-scroll-results

  [scroll-id results]

  (let [scroll-response (esd/scroll conn scroll-id {:scroll "1m"})

        hits            (hits-from scroll-response)]

    (println "here")

    (if (seq hits)

      (recur (:_scroll_id scroll-response) (concat results hits))

      (concat results hits))))

But it blows up with this response.

    "{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Failed to parse request body\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Failed to parse request body\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Unrecognized token 'DnF1ZXJ5VGhlbkZldGNoWgAAAAAAAA4dFnYxemYxQW9aUlhTYmotVFZtSEpsWEEAAAAAACCadRZ0REJTdElGSVF2bWJQZ3NLeGwzbXB3AAAAAAAADh4WdjF6ZjFBb1pSWFNiai1UVm1ISmxYQQAAAAAAABgXFlo2V24yUnhTU2VTYTFrbnp0Ql9LZGcAAAAAAAAYFhZaNlduMlJ4U1NlU2Exa256dEJfS2RnAAAAAAAADh8WdjF6ZjFBb1pSWFNi...': was expecting ('true', 'false' or 'null')\\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@1ddaa960; line: 1, column: 257]\"}},\"status\":400}",

It looks like I am using Elastic 5.6.13 (it is part of OpenShift). 

{
 "name" : "logging-es-data-master-vamp431b",
 "cluster_name" : "logging-es",
 "cluster_uuid" : "HEk06iYDQYSO2d3Xc6moRg",
 "version" : {
   "number" : "5.6.13",
   "build_hash" : "921f2bd",
   "build_date" : "2018-11-16T16:58:29.974Z",
   "build_snapshot" : false,
   "lucene_version" : "6.6.1"
 },
 "tagline" : "You Know, for Search"
}

Any ideas on what is wrong?

This is the query I am using and it brings back a great big scroll_id and the first result.

(esd/search conn "project.*" "" {:query (q/bool {:must [{:match_all {}} {:exists {:field :mdc.X-B3-TraceId}}]})

                                           :search_type "query_then_fetch"

                                           :scroll "1m"

                                           :size 1} )

[Francis Hitchens]

Hi,

Not to worry, I think I found the issue. I used the Leiningen dependency mentioned in the Elastisearch Get Started documentation,

[clojurewerkz/elastisch "2.2.1"]

Not the github page

[clojurewerkz/elastisch "3.0.1"]

And I guess the 2.2.1 version just posted the scroll_id in the body without any JSON DSL around it whereas the 3.0.1 adds the DSL and everything would be fine.

So I upgraded my project to use "3.0.1" but it still did not work.

Upon examining the rest/document.clj in the clojars release I find it contains this old implementation of scroll which just POST's the scroll Id in the body of the request.

(defn scroll
 "Performs a scroll query, fetching the next page of results from a
 query given a scroll id"
 ([^Connection conn scroll-id] (scroll conn scroll-id nil))
 ([^Connection conn scroll-id opts]
  (let [qk [:search_type :scroll :routing :preference]
        qp   (select-keys opts qk)
        body scroll-id]
    (rest/post-string conn (rest/scroll-url conn)
                      {:body body
                       :query-params qp}))))

Instead of what is on the Master branch in github which POST's the request wrapped in JSON DSL which Elastic 5.x expects:

(defn scroll
	"Performs a scroll query, fetching the next page of results from a
	query given a scroll id"
	([^Connection conn scroll-id] (scroll conn scroll-id nil))
	([^Connection conn scroll-id opts]
	(let [qk (cond-> [:search_type :routing :preference]
	(not scroll-id) (conj :scroll))
	qp (select-keys opts qk)
	body (if scroll-id
	{:scroll (:scroll opts)
	:scroll_id scroll-id})]
	(rest/post conn (rest/scroll-url conn)
	{:body body
	:query-params qp}))))

Oh woe is me, what do I do now?

regards, Francis.

[khan379]

Hi Francis,

what did you end up doing to fix this ? 

we upgraded ealsticsearch from 2.4 to 6.1 and i updated the elastisch from [clojurewerkz/elastisch "3.0.0-BETA"] to [org.clojars.sergeyminaev/elastisch "6.0.0"]. also tried [clojurewerkz/elastisch "3.0.1"]. i am getting exactly same error as yours. 

here is my error,

[1;31mclojure.lang.ExceptionInfo [m: [3mclj-http: status 400 [m

                      [1mbody [m: "{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Failed to parse request body\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Failed to parse request body\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Unrecognized token 'DnF1ZXJ5VGhlbkZldGNoIgAAAAAAAAMyFm1TcEdhN3k5UnB5Y0wzaWpydnhENFEAAAAAAAAC2BZqa2hGa0YxVVJxU0JOeS02N1ZLUGx3AAAAAAAAA1UWNXJzQmN4dUZSaktQVnNjdlRNOWgyZwAAAAAAAAKbFjd6MzhSa2ZsVGdLU2p2Q0RpZDkySkEAAAAAAAACqhZCQVhxYWpRclNNT3VqUXlhWU1SZE1nAAAAAAAAA0AWS0Z2VzFGQVlSTGFZ...': was expecting ('true', 'false' or 'null')\\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@4ed32bfb; line: 1, column: 257]\"}},\"status\":400}"

                  [1mheaders [m: {"Server" "nginx/1.13.3",

                            "Date" "Fri, 06 Sep 2019 22:43:09 GMT",

                            "Content-Type" "application/json; charset=UTF-8",

                            "Content-Length" "473",

                            "Connection" "close",

                            "access-control-allow-credentials" "true"}

    [1morig-content-encoding [m: "gzip"

              [1mrequest-time [m: 155

                    [1mstatus [m: 400

          [1mtrace-redirects [m: ["https://localhost:9200/_search/scroll"]