how do I find an IO exception triggering retries in clj-http?

[larry google groups]

So, thanks to Michael Klishin, Aaron Cohen, Frank Siebenlist and Craig
Brozefsky I am now able to correctly ping the Omniture API. But I am
getting a strange behavior from the clj-http library. It makes 4 calls
to the API server, even though the first call is successful.

When I look here:

https://github.com/dakrone/clj-http

I see it says:

;; Apache's http client automatically retries on IOExceptions, if you
;; would like to handle these retries yourself, you can specify a
;; :retry-handler.

So, since it re-tries, I should assume that it is encountering an
IOException. But I get a successful response on the first try, so what
would the IOException be?

Because clj-http uses Slingshot, I have wrapped it in a "try+ / catch
Object o" block. And I print the "o" to the terminal, and yet I am not
seeing anything in the terminal. So where is the IOException? How do I
find it?

This is the actual function I use to ping the Omniture API:

(defn omniture-call-api [url-with-queue-method api-payload headers]
(timbre/spy :debug " return value of omniture-call-api "
(try+
(http-client/post url-with-queue-method
{:body api-payload
:debug true
:debug-body true
:insecure true
:headers {"X-Api-Version" "2"
"X-WSSE" headers}
:content-type :json
:socket-timeout 4000
:conn-timeout 4000
:accept :json
:client-params
{"http.protocol.allow-circular-redirects" true
"http.useragent"
"clj-http"}})
(catch Object o (println (pp/pprint o))))))

If there is an IOException, why doesn't this line catch it?

(catch Object o (println (pp/pprint o))

I read here that "catch Object o" is the correct way to catch
everything, using Slingshot:

https://github.com/scgilardi/slingshot/issues/24

So why would I not see this error?

[Marc Boschma]

Might be a silly suggestion but are the first 3 connections leading to redirects (30x's) ???

> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[larry google groups]

> Might be a silly suggestion but are the first 3 connections leading to redirects (30x's) ???

Interesting idea. I will check. You are thinking that I should:

{"http.protocol.allow-circular-redirects" false

yes?

On Mar 5, 2:39 am, Marc Boschma <marc.bosc...@gmail.com> wrote:
> Might be a silly suggestion but are the first 3 connections leading to redirects (30x's) ???
>

[larry google groups]

As near as I can see, the 1st request is successful, and then the next
3 return with 401 errors, plus the message "The nonce has already been
used." Clearly, the 2nd, 3rd and 4th attempts should fail, because
they are re-using the nonce that was already used in the first
request, and a nonce can only be used once. So it makes sense that
they would get a 401 error, but it makes no sense that they would ever
get called at all, because why would the code re-try if there was no
IOException.

One of the requests returns with a 200 status:

{:trace-redirects ["https://api2.omniture.com/admin/1.3/rest/?
method=Report.QueueOvertime"], :request-time 1658, :status
200, :headers {"date" "Mon, 04 Mar 2013 21:23:44 GMT", "server"
"Omniture AWS/2.0.0", "vary" "Accept-Encoding", "content-encoding"
"gzip", "xserver" "www322", "content-length" "83", "content-type"
"application/json", "connection" "close"}, :body "{\"status\":\"queued
\",\"statusMsg\":\"Your report has been queued\",\"reportID\":xxx}"}

I assume this is the first request, because if it wasn't then it would
get the error "This nonce has already been used." And why would the
later requests get the 401 error "This nonce has already been used"
unless the first request had succeeded?

My understanding is the code will retry up to 4 times if it encounters
an IOException. But in my case it seems to be retrying even though
here was no IOException.

Maybe this is some kind of timing issue?






On Mar 5, 10:50 am, larry google groups <lawrencecloj...@gmail.com>

[Meikel Brandmeyer (kotarak)]

Hi,

silly question are you using the call in a sequence pipeline with pmap or mapcat or the like? Or does clj-http that under the covers?

Kind regards

Meikel

[larry google groups]

> silly question are you using the call in a sequence pipeline with pmap or
> mapcat or the like? Or does clj-http that under the covers?

Those are good questions. Until I get this working I have been doing
the simplest thing possible, which is simply starting the app, and
then having the app make the call to Omniture as one of the first
things it does on startup.

I will dig into clj-http to see what it is doing under the covers.

I am curious, do you have a suspicion about something? A theory?




On Mar 5, 11:17 am, "Meikel Brandmeyer (kotarak)" <m...@kotka.de>
wrote:

[larry google groups]

This is strange. An IOException is thrown, but it seems to do with
with the retry, which should never exist in the first place, unless an
IOException has been thrown:


clj-http has thrown this exception:

{:trace-redirects
["https://api2.omniture.com/admin/1.3/rest/?
method=Report.QueueOvertime"],

:request-time 885,
:status 401,
:headers
{"date" "Tue, 05 Mar 2013 19:14:46 GMT",
"server" "Omniture AWS/2.0.0",
"www-authenticate"
"WSSE realm=\"Omniture REST Api\", profile=\"UsernameToken\"",
"xserver" "www811",
"content-length" "90",

"content-type" "application/json",
"connection" "close"},
:body

"{\"error\":\"The nonce
(NGQ1ODc4NmU3M2QyY2I5MmIyOTIzOWFiN2Q4ODc1NjQ=) has already been used
\"}"}




On Mar 5, 12:45 pm, larry google groups <lawrencecloj...@gmail.com>
wrote:

[larry google groups]

Out of frustration, I have decided to surpress the re-tries. This
worries me -- I hate to bury a problem without understanding it. But
the retries are burying the actual queries. This page:

https://github.com/dakrone/clj-http

says:

;; Apache's http client automatically retries on IOExceptions, if you
;; would like to handle these retries yourself, you can specify a

;; :retry-handler. Return true to retry, false to stop trying:
(client/post "http://example.org" {:multipart [["title" "Foo"]
["Content/type" "text/
plain"]

["file" (clojure.java.io/file "/tmp/missing-file")]]
:retry-handler (fn [ex try-count
http-context]
(println "Got:"
ex)
(if (> try-count
4) false true))})

So I have re-written my function so that the re-tries are always
suppressed:

(defn omniture-call-api [url-with-queue-method api-payload headers]
(timbre/spy :debug " return value of omniture-call-api "
(try+
(http-client/post url-with-queue-method
{:body api-payload
:debug true
:debug-body true
:insecure true
:headers {"X-Api-Version" "2"
"X-WSSE" headers}
:content-type :json
:socket-timeout 4000
:conn-timeout 4000
:accept :json
:client-params

{"http.protocol.allow-circular-redirects" false
"http.useragent"
"clj-http"}
:retry-handler (fn [ex try-count
http-context]
false)
})
(catch Object o (catch-clj-http-exceptions o)))))


I will have to come back to this later, when I have more time, and try
to figure out what is going wrong.



On Mar 5, 2:19 pm, larry google groups <lawrencecloj...@gmail.com>

[larry google groups]

Strangely enough, the re-tries continue even when the retry handler
always returns false. I must be misunderstanding things at a
fundamental level.


On Mar 5, 2:33 pm, larry google groups <lawrencecloj...@gmail.com>

[larry google groups]

I look here:

https://github.com/dakrone/clj-http/issues/45

and I see the original reasoning for :trace-redirects

"add a key like :trace-redirects in the response containing the
ordered list of traversed uris. This would allow clients to know what
was the final uri fetched more easily than now (I think the only way
is to set :save-request? and check the url inside the req). It could
also be very useful for caching purpose (knowing all the uris which
are equivalent to the terminal one). "

The README says: ":trace-redirects will contain the chain of the
redirections followed."

But in my case, I don't think I'm getting redirected. My :trace-
redirects only has 1 URL, and it is the URL that I POSTed to.

[:trace-redirects ["https://api2.omniture.com/admin/1.3/rest/?
method=Report.QueueOvertime"]] [:request-time 1601][:status 200]
[:headers {"date" "Tue, 05 Mar 2013 19:57:24 GMT", "server" "Omniture

AWS/2.0.0", "vary" "Accept-Encoding", "content-encoding" "gzip",

"xserver" "www503", "content-length" "83", "content-type" "application/
json", "connection" "close"}][:body "{\"status\":\"queued\",\"statusMsg
\":\"Your report has been queued\",\"reportID\":83928267}"]


Maybe the POST gets redirected to something with the same URL? The
status I see is 200, not 301 or 302.







On Mar 5, 2:33 pm, larry google groups <lawrencecloj...@gmail.com>