How does clj-http work regarding https?

[larry google groups]

This might be a dumb "How does the Internet work" kind of question.

I have been asked to pull data from Omniture, using the Omniture API.
I thought this would take me an hour, but I've been working on this
for 3 days now. I keep getting authentication errors.

I became curious about exactly what code was sending to Omniture, so I
downloaded Charles, the network debugging tool:

http://www.charlesproxy.com/

I am using clj-http to make the POST. The post is suppose to be
https.

This is the thing that surprises me: If I use http then I can see all
the headers in Charles, and they all look correct. But if I use https
(which is what I need to use) then there are no headers that I can see
in Charles.

Is that because Charles does not want to show me a bunch of encrypted
garbage? Or is clj-http not adding in the headers with https? Maybe I
need a special setting to get clj-http to correctly send to https? (I
have not been able to find any such setting.)

This is the code where I use clj-http (here I call http-client):

(defn omniture-call-api [url-with-queue-method api-payload headers]
(timbre/spy :debug " return value of omniture-call-api "
(try+
(http-client/post url-with-queue-method
{:body api-payload
:headers {"X-Api-Version" "2"
"X-WSSE" headers}
:content-type :json
:socket-timeout 4000
:conn-timeout 4000
:accept :json
:client-params
{"http.protocol.allow-circular-redirects" false
"http.useragent"
"clj-http"}})
(catch Object o (println (pp/pprint o))))))

The url is:

https://api2.omniture.com/admin/1.3/rest/?method=Report.QueueRanked

At first I assumed this was a problem with Omniture's code. I have
asked several question on the Developer forum at Omniture, but no one
there could help me (Adobe has apparently cut most of the developer
support since Adobe bought Omniture in 2009). However, I am now
wondering if maybe I am doing this POST incorrectly.

I am puzzled by something else as well: I call this function once, yet
Charles shows 4 calls being made to Omniture, and my own code, when it
prints data to the terminal, seems to show many requests being made.
Why would that be?

Again, if I change the URL so it uses "http" then in Charles I can see
all the headers that suppose to be in this line of code:

"X-WSSE" headers

and the headers look correct (I posted them to the developer forums at
Omniture and the one guy from Omniture who gave it a look felt there
was nothing terribly amiss -- but he couldn't rule anything out.)

But If I change the URL to use https, then I see no headers in
Charles. Why is that?

---- lawrence

[larry google groups]

When I turn on:

:debug true
:debug-body true

I get all that follows. Which mostly looks right, I guess. But why
don't I see the headers in Charles?

I'll go back to assuming this is a problem with Omniture, but I would
be grateful if anyone could glance over this output and tell me if
anything looks amiss. I took out the username and password but
otherwise left in all the details.


{:scheme :https,
 :http-url
 "https://api2.omniture.com/admin/1.3/rest/?
method=Report.QueueOvertime",
 :conn-timeout 4000,
 :debug-body true,
 :request-method :post,
 :query-string "method=Report.QueueOvertime",
 :content-type :json,
 :uri "/admin/1.3/rest/",
 :server-name "api2.omniture.com",
 :headers
 {"accept-encoding" "gzip, deflate",
  "accept" "application/json",
  "content-type" "application/json",
  "x-api-version" "2",
  "x-wsse"
  "UsernameToken Username=\"xxx\", PasswordDigest=\"xxx\", Nonce=
\"Y2FiOGM3YjAxY2QyY2YwMTdmZTY2YTIzNzNjNzU2OWI=\", Created=
\"2013-02-23T16:40:14Z"},
 :socket-timeout 4000,
 :debug true,
 :body-type org.apache.http.entity.StringEntity,
 :server-port nil,
 :character-encoding "UTF-8",
 :client-params
 {"http.useragent" "clj-http",
  "http.protocol.allow-circular-redirects" false},
 :body
 "{\"reportDescription\" : {
                          \"reportSuiteID\" : \"timeoutny\",
                          \"dateFrom\" : \"2013-02-01\",
                          \"dateTo\" : \"2013-02-12\",
                          \"dateGranularity\" : \"hourly\",
                          \"metrics\" : [
                                         {
                                          \"id\" : \"pageViews\"
                                          },
                                         {
                                          \"id\" : \"visits\"
                                          }
                                         ],
                          \"validate\":\"true\"
                          }
                  }",
 :user-info nil}

HttpRequest:
{:requestLine
 #<BasicRequestLine POST https://api2.omniture.com/admin/1.3/rest/?method=Report.QueueOvertime
HTTP/1.1>,
 :protocolVersion #<HttpVersion HTTP/1.1>,
 :params
 #<BasicHttpParams org.apache.http.params.BasicHttpParams@47b0081>,
 :method "POST",
 :entity #<StringEntity org.apache.http.entity.StringEntity@117eb1bd>,
 :class org.apache.http.client.methods.HttpPost,
 :allHeaders
 [#<BasicHeader Connection: close>,
  #<BasicHeader accept-encoding: gzip, deflate>,
  #<BasicHeader accept: application/json>,
  #<BasicHeader content-type: application/json>,
  #<BasicHeader x-api-version: 2>,
  #<BasicHeader x-wsse: UsernameToken Username="xxx",
PasswordDigest="xxx",
Nonce="Y2FiOGM3YjAxY2QyY2YwMTdmZTY2YTIzNzNjNzU2OWI=",
Created="2013-02-23T16:40:14Z>],
 :aborted false,
 :URI
 #<URI https://api2.omniture.com/admin/1.3/rest/?method=Report.QueueOvertime>}



Sadly, Omniture doesn't offer any support service, so I am not able to
contact them directly and get any help from them. (They have support
contracts that cost many thousands of dollars, but my company would
not be interested in that kind of expense.)








{:trace-redirects
 ["https://api2.omniture.com/admin/1.3/rest/?
method=Report.QueueOvertime"],
 :request-time 1064,
 :status 401,
 :headers
 {"date" "Sat, 23 Feb 2013 21:40:15 GMT",
  "server" "Omniture AWS/2.0.0",
  "www-authenticate"
  "WSSE realm=\"Omniture REST Api\", profile=\"UsernameToken\"",
  "xserver" "www336",
  "content-length" "46",
  "content-type" "application/json",
  "connection" "close"},
 :body "{\"error\":\"Unable to validate authentication.\"}"}
nil
Request: org.apache.http.entity.StringEntity



On Feb 23, 4:18 pm, larry google groups <lawrencecloj...@gmail.com>
wrote:

[Jonah Benton]

Try adding

:insecure? true

to the map. Charles dynamically generates a cert pretending to be the target host when acting as an ssl proxy, and clj-http probably has to be told to accept it.

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[larry google groups]

> Try adding
>
> :insecure? true
>
> to the map. Charles dynamically generates a cert pretending to be the
> target host when acting as an ssl proxy, and clj-http probably has to be
> told to accept it.

Okay, I've done so, but I don't think I understand what you are
telling me. Are you saying that clj-http won't send its POST if it
feels the cert is false?

Charles has consistently said, in each report, "SSL Proxying not
enabled for this host: enable in Proxy Settings, SSL locations"

And I've made the change you suggested, but I still see that message.

[larry google groups]

Any idea why a single call to clj-http/post causes 4 transactions to
appear in Charles?



On Feb 23, 5:47 pm, larry google groups <lawrencecloj...@gmail.com>

[Jonah Benton]

Ok- so in Charles, you'll need to do that, tell it to ssl proxy the domain

api2.omniture.com

Described in a little more detail here:

http://www.charlesproxy.com/documentation/proxying/ssl-proxying/

[Jonah Benton]

Very likely it's the automatic retry logic:

;; Apache's http client automatically retries on IOExceptions, if you
;; would like to handle these retries yourself, you can specify a
;; :retry-handler. Return true to retry, false to stop trying:
(client/post "http://example.org" {:multipart [["title" "Foo"]
                                               ["Content/type" "text/plain"]
                                               ["file" (clojure.java.io/file "/tmp/missing-file")]]
                                   :retry-handler (fn [ex try-count http-context]
                                                    (println "Got:" ex)
                                                    (if (> try-count 4) false true))})

from

https://github.com/dakrone/clj-http

[larry google groups]

> Very likely it's the automatic retry logic:

Thank you for that. One mystery solved. I wonder why it retries? The
ping reaches Omniture, I get back a 401 error. Maybe it retries on any
4xx error?

[larry google groups]

>
> Described in a little more detail here:
>
> http://www.charlesproxy.com/documentation/proxying/ssl-proxying/
>

Thank you, that is a huge help.

I am finding it is a real headache to use several new technologies,
all at once.

[larry google groups]

>;; Apache's http client automatically retries on IOExceptions,

I guess clj-http swallows the exception? It seems strange that it
doesn't bubble up to my code. But then I don't know Java so I probably
miss some of the etiquette about when a library should catch its own
exceptions.



On Feb 23, 6:52 pm, larry google groups <lawrencecloj...@gmail.com>

[Jonah Benton]

It won't retry on http-level errors; those semantics are up to the application. It should only retry connection-layer problems. So there must have been something funny happening either between the client and Charles, or between Charles and Omniture.

[Jonah Benton]

If you cut Charles out of the picture and just send your payload directly to Omniture over https, how does Omniture respond?

[Chris Jeris]

When you say you're getting "authentication errors", are you getting authentication errors from the Omniture API, or are you getting SSL certificate authentication errors?  If it's the latter, the problem may be that your runtime doesn't have a truststore containing a CA certificate that authenticates the SSL certificate Omniture is presenting.  Check your javax.net.ssl.trustStore property.

peace, C

--
Chris Jeris
cje...@brightcove.com
freenode/twitter/github: ystael