[groovy-user] HTTPBuilder and basic authentication.

2,793 views
Skip to first unread message

Erik Husby

unread,
Oct 26, 2010, 4:20:44 PM10/26/10
to us...@groovy.codehaus.org
I am trying to make use of the Groovy HTTPBuilder to handle some REST services which require basic authentication.

I can successfully use curl --basic to get a valid response.

When I follow the documentation for Groovy HTTPBuilder, http://groovy.codehaus.org/modules/http-builder/doc/auth.html, and specify the same username/password, the request fails.

The code looks like this (just like the documentation says)

         restClient = new RESTClient("http://prodinfobuild.broadinstitute.org:8085/rest/api/latest/")
         restClient.auth.basic 'mhusby', 'bamboo'
         def response = restClient.get(path:'project')


I am using Groovy 1.7.2, HTTPBuilder 0.5.1 (the latest).

What am I overlooking?


The log output shows:

DEBUG - Receiving response: HTTP/1.1 401 Unauthorized
DEBUG - << HTTP/1.1 401 Unauthorized
DEBUG - << WWW-Authenticate: OAuth realm="http%3A%2F%2Fprodinfobuild.broadinstitute.org%3A8085"
DEBUG - << Cache-Control: no-transform
DEBUG - << Content-Type: application/xml
DEBUG - << Content-Length: 174
DEBUG - << Server: Jetty(6.1.15)
DEBUG - Connection can be kept alive indefinitely
DEBUG - Target requested authentication
DEBUG - Authentication schemes in the order of preference: [ntlm, digest, basic]
DEBUG - Challenge for ntlm authentication scheme not available
DEBUG - Challenge for digest authentication scheme not available
DEBUG - Challenge for basic authentication scheme not available
WARN - Authentication error: Unable to respond to any of these challenges: {oauth=WWW-Authenticate: OAuth realm="http%3A%2F%2Fprodinfobuild.broadinstitute.org%3A8085"}
DEBUG - Response code: 401; found handler: org.codehaus.groovy.runtime.MethodClosure@32830122
DEBUG - Parsing response as: application/xml
WARN - Could not find charset in response
DEBUG - << "<?xml version="1.0" encoding="UTF-8" standalone="yes"?><status><status-code>401</status-code><message>Client must be authenticated to access this resource.</message></status>"
DEBUG - Releasing connection org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter@4ab9b8d0
DEBUG - Parsed data to instance of: class groovy.util.slurpersupport.NodeChild

groovyx.net.http.HttpResponseException: Unauthorized
at groovyx.net.http.RESTClient.defaultFailureHandler(RESTClient.java:240)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:923)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:276)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:492)
at groovyx.net.http.RESTClient.get(RESTClient.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.broadinstitute.prodinfo.releng.BambooInfo.getProjects(BambooInfo.groovy:30)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl$GetBeanMethodMetaProperty.getProperty(MetaClassImpl.java:3460)
at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.getProperty(GetEffectivePogoPropertySite.java:84)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSite.java:241)
at org.broadinstitute.prodinfo.releng.BambooInfoTest.GetProjectsFromBamboo(BambooInfoTest.groovy:22)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)
at org.junit.internal.runners.MethodRoadie.runTestMethod(MethodRoadie.java:98)
at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:79)
at org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:87)
at org.junit.internal.runners.MethodRoadie.runTest(MethodRoadie.java:77)
at org.junit.internal.runners.MethodRoadie.run(MethodRoadie.java:42)
at org.junit.internal.runners.JUnit4ClassRunner.invokeTestMethod(JUnit4ClassRunner.java:88)
at org.junit.internal.runners.JUnit4ClassRunner.runMethods(JUnit4ClassRunner.java:51)
at org.junit.internal.runners.JUnit4ClassRunner$1.run(JUnit4ClassRunner.java:44)
at org.junit.internal.runners.ClassRoadie.runUnprotected(ClassRoadie.java:27)
at org.junit.internal.runners.ClassRoadie.runProtected(ClassRoadie.java:37)
at org.junit.internal.runners.JUnit4ClassRunner.run(JUnit4ClassRunner.java:42)
at org.junit.runner.JUnitCore.run(JUnitCore.java:130)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)



---

Erik Husby

Senior Software Engineer I

Broad Institute

Rm. 2139, 320 Charles St, Cambridge, MA 02141-2023

mobile: 781.354.6669, office: 617.714.8443

email: mhu...@broadinstitute.org Jabber:mhusby@.broadinstitute.org







Thom Nichols

unread,
Oct 26, 2010, 5:53:10 PM10/26/10
to user
Looks right so far, but you left out the critical part of debug info in your email -- the request!  Can you send the log output of the preceding request?  That should help troubleshooting.

Thanks.
-Tom

Erik Husby

unread,
Oct 27, 2010, 9:38:43 AM10/27/10
to us...@groovy.codehaus.org
Here is the full trace

2010-10-27 09:37:03,307 DEBUG  RESTClient - GET http://prodinfobuild.broadinstitute.org:8085/rest/api/latest/project
2010-10-27 09:37:03,548 DEBUG  SingleClientConnManager - Get connection for route HttpRoute[{}->http://prodinfobuild.broadinstitute.org:8085]
2010-10-27 09:37:03,596 DEBUG  RequestAddCookies - CookieSpec selected: best-match
2010-10-27 09:37:03,607 DEBUG  DefaultHttpClient - Attempt 1 to execute request
2010-10-27 09:37:03,608 DEBUG  DefaultClientConnection - Sending request: GET /rest/api/latest/project HTTP/1.1
2010-10-27 09:37:03,608 DEBUG  wire - >> "GET /rest/api/latest/project HTTP/1.1[EOL]"
2010-10-27 09:37:03,610 DEBUG  wire - >> "Accept: application/json, application/javascript, text/javascript[EOL]"
2010-10-27 09:37:03,610 DEBUG  wire - >> "Host: prodinfobuild.broadinstitute.org:8085[EOL]"
2010-10-27 09:37:03,610 DEBUG  wire - >> "Connection: Keep-Alive[EOL]"
2010-10-27 09:37:03,611 DEBUG  wire - >> "Accept-Encoding: gzip,deflate[EOL]"
2010-10-27 09:37:03,611 DEBUG  wire - >> "[EOL]"
2010-10-27 09:37:03,611 DEBUG  headers - >> GET /rest/api/latest/project HTTP/1.1
2010-10-27 09:37:03,611 DEBUG  headers - >> Accept: application/json, application/javascript, text/javascript
2010-10-27 09:37:03,612 DEBUG  headers - >> Host: prodinfobuild.broadinstitute.org:8085
2010-10-27 09:37:03,612 DEBUG  headers - >> Connection: Keep-Alive
2010-10-27 09:37:03,612 DEBUG  headers - >> Accept-Encoding: gzip,deflate
2010-10-27 09:37:03,616 DEBUG  wire - << "HTTP/1.1 401 Unauthorized[EOL]"
2010-10-27 09:37:03,618 DEBUG  wire - << "WWW-Authenticate: OAuth realm="http%3A%2F%2Fprodinfobuild.broadinstitute.org%3A8085"[EOL]"
2010-10-27 09:37:03,619 DEBUG  wire - << "Cache-Control: no-transform[EOL]"
2010-10-27 09:37:03,619 DEBUG  wire - << "Content-Type: application/json[EOL]"
2010-10-27 09:37:03,619 DEBUG  wire - << "Transfer-Encoding: chunked[EOL]"
2010-10-27 09:37:03,619 DEBUG  wire - << "Server: Jetty(6.1.15)[EOL]"
2010-10-27 09:37:03,620 DEBUG  wire - << "[EOL]"
2010-10-27 09:37:03,620 DEBUG  DefaultClientConnection - Receiving response: HTTP/1.1 401 Unauthorized
2010-10-27 09:37:03,621 DEBUG  headers - << HTTP/1.1 401 Unauthorized
2010-10-27 09:37:03,621 DEBUG  headers - << WWW-Authenticate: OAuth realm="http%3A%2F%2Fprodinfobuild.broadinstitute.org%3A8085"
2010-10-27 09:37:03,621 DEBUG  headers - << Cache-Control: no-transform
2010-10-27 09:37:03,622 DEBUG  headers - << Content-Type: application/json
2010-10-27 09:37:03,622 DEBUG  headers - << Transfer-Encoding: chunked
2010-10-27 09:37:03,622 DEBUG  headers - << Server: Jetty(6.1.15)
2010-10-27 09:37:03,628 DEBUG  DefaultHttpClient - Connection can be kept alive indefinitely
2010-10-27 09:37:03,629 DEBUG  DefaultHttpClient - Target requested authentication
2010-10-27 09:37:03,629 DEBUG  DefaultTargetAuthenticationHandler - Authentication schemes in the order of preference: [ntlm, digest, basic]
2010-10-27 09:37:03,630 DEBUG  DefaultTargetAuthenticationHandler - Challenge for ntlm authentication scheme not available
2010-10-27 09:37:03,630 DEBUG  DefaultTargetAuthenticationHandler - Challenge for digest authentication scheme not available
2010-10-27 09:37:03,630 DEBUG  DefaultTargetAuthenticationHandler - Challenge for basic authentication scheme not available
2010-10-27 09:37:03,631 WARN   DefaultHttpClient - Authentication error: Unable to respond to any of these challenges: {oauth=WWW-Authenticate: OAuth realm="http%3A%2F%2Fprodinfobuild.broadinstitute.org%3A8085"}
2010-10-27 09:37:03,634 DEBUG  RESTClient - Response code: 401; found handler: org.codehaus.groovy.runtime.MethodClosure@756f70a4
2010-10-27 09:37:03,634 DEBUG  RESTClient - Parsing response as: application/json
2010-10-27 09:37:03,675 WARN   ParserRegistry - Could not find charset in response
2010-10-27 09:37:03,676 DEBUG  wire - << "57[EOL]"
2010-10-27 09:37:03,676 DEBUG  wire - << "{"status-code":"401","message":"Client must be authenticated to access this resource."}"
2010-10-27 09:37:03,676 DEBUG  wire - << "[\r]"
2010-10-27 09:37:03,676 DEBUG  wire - << "[\n]"
2010-10-27 09:37:03,677 DEBUG  wire - << "0[EOL]"
2010-10-27 09:37:03,677 DEBUG  wire - << "[EOL]"
2010-10-27 09:37:03,677 DEBUG  SingleClientConnManager - Releasing connection org.apache.http.impl.conn.SingleClientConnManager$ConnAdapter@4cd4544
2010-10-27 09:37:03,706 DEBUG  RESTClient - Parsed data to instance of: class net.sf.json.JSONObject

groovyx.net.http.HttpResponseException: Unauthorized
at groovyx.net.http.RESTClient.defaultFailureHandler(RESTClient.java:240)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1058)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:923)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)
at groovy.lang.Closure.call(Closure.java:276)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:492)
at groovyx.net.http.RESTClient.get(RESTClient.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:40)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:117)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
at org.broadinstitute.prodinfo.releng.BambooInfo.getProjects(BambooInfo.groovy:34)
Process finished with exit code 255

Thom Nichols

unread,
Oct 27, 2010, 9:50:30 AM10/27/10
to user
Whups, actually I should have figured it out after the first email....  The server is asking for OAuth authentication, not basic!

2010-10-27 09:37:03,621 DEBUG  headers - << WWW-Authenticate: OAuth realm="http%3A%2F%2Fprodinfobuild.broadinstitute.org%3A8085"

The way HttpClient works is it waits for a 401 response (the WWW-Authenticate challenge) and _then_ sends the credentials if it has them.  The purpose is to not send auth information unless the server asks for it.

See: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html#d4e950

If the server truly supports basic auth it _should_ indicate so with a second WWW-Authenticate: Basic header in the same response.  But HTTPBuilder isn't sending your Basic auth credentials because the server isn't asking for them.  Your options then are either to supply OAuth credentials or implement preemptive basic auth as described in the above link.  You do it on the Apache HttpClient that HTTPBuilder uses, by using the HTTPBuilder.getClient() property.

-Thom

Erik Husby

unread,
Oct 27, 2010, 10:33:04 AM10/27/10
to us...@groovy.codehaus.org

On Oct 27, 2010, at 9:50 AM, Thom Nichols wrote:

Whups, actually I should have figured it out after the first email....  The server is asking for OAuth authentication, not basic!

2010-10-27 09:37:03,621 DEBUG  headers - << WWW-Authenticate: OAuth realm="http%3A%2F%2Fprodinfobuild.broadinstitute.org%3A8085"

The way HttpClient works is it waits for a 401 response (the WWW-Authenticate challenge) and _then_ sends the credentials if it has them.  The purpose is to not send auth information unless the server asks for it.

See: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/authentication.html#d4e950

If the server truly supports basic auth it _should_ indicate so with a second WWW-Authenticate: Basic header in the same response.  But HTTPBuilder isn't sending your Basic auth credentials because the server isn't asking for them.  Your options then are either to supply OAuth credentials or implement preemptive basic auth as described in the above link.  You do it on the Apache HttpClient that HTTPBuilder uses, by using the HTTPBuilder.getClient() property.

-Thom




Thanks for the info, will try that. Is that what Curl is doing when I use the --basic option?

If I do

* About to connect() to prodinfobuild.broadinstitute.org port 8085 (#0)
*   Trying 69.173.75.0... connected
* Connected to prodinfobuild.broadinstitute.org (69.173.75.0) port 8085 (#0)
> GET /rest/api/latest/project HTTP/1.1
> User-Agent: curl/7.16.4 (i386-apple-darwin9.0) libcurl/7.16.4 OpenSSL/0.9.7l zlib/1.2.3
> Accept: */*
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: OAuth realm="http%3A%2F%2Fprodinfobuild.broadinstitute.org%3A8085"
< Cache-Control: no-transform
< Content-Type: application/xml
< Content-Length: 174
< Server: Jetty(6.1.15)
* Connection #0 to host prodinfobuild.broadinstitute.org left intact
* Closing connection #0
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><status><status-code>401</status-code><message>Client must be authenticated to access this resource.</message></status>cm096-46c ~ 10:26:40 $ 

I can see that Bamboo is sending the OAuth request..

If I do 
* About to connect() to prodinfobuild.broadinstitute.org port 8085 (#0)
*   Trying 69.173.75.0... connected
* Connected to prodinfobuild.broadinstitute.org (69.173.75.0) port 8085 (#0)
* Server auth using Basic with user 'releng'
> GET /rest/api/latest/project HTTP/1.1
> Authorization: Basic cmVsZW5nOnJlbGVuZw==
> User-Agent: curl/7.16.4 (i386-apple-darwin9.0) libcurl/7.16.4 OpenSSL/0.9.7l zlib/1.2.3
> Accept: */*
< HTTP/1.1 200 OK
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: JSESSIONID=1m2l9fcgi5daq;Path=/;HttpOnly
< Content-Type: application/xml
< Content-Length: 4096
< Server: Jetty(6.1.15)
* Connection #0 to host prodinfobuild.broadinstitute.org left intact
* Closing connection #0

No WWW-Authenticate comes back.

Perhaps you should consider adding preemptive authorization to HTTPBuilder as a future option.

Thom Nichols

unread,
Oct 27, 2010, 6:36:17 PM10/27/10
to user
So the problem is, roughly, that the server is claiming to want OAuth credentials when it really wants Basic credentials (or maybe it will accept both.)  But the point is, the server isn't asking for Basic credentials. 

Curl is being 'preemptive' -- that is, if you give the '--user' option, curl will send Basic auth credentials no matter what.  This is different from what HttpClient does by default (again, the reasoning behind the behavior is explained here.) 

So your options stand at either (1) implementing preemptive basic authentication, or (2) supplying OAth credentials instead of Basic.  Or maybe (3) filing a bug with Bamboo indicating that they are missing a WWW-Authenticate challenge for Basic credentials in 401 responses.

-Thom

Erik Husby

unread,
Oct 28, 2010, 10:01:52 AM10/28/10
to us...@groovy.codehaus.org
On Oct 27, 2010, at 6:36 PM, Thom Nichols wrote:

So the problem is, roughly, that the server is claiming to want OAuth credentials when it really wants Basic credentials (or maybe it will accept both.)  But the point is, the server isn't asking for Basic credentials. 

Curl is being 'preemptive' -- that is, if you give the '--user' option, curl will send Basic auth credentials no matter what.  This is different from what HttpClient does by default (again, the reasoning behind the behavior is explained here.) 
Actually, Curl only behaves preemptively when one uses the --basic option.


So your options stand at either (1) implementing preemptive basic authentication, or (2) supplying OAth credentials instead of Basic.  Or maybe (3) filing a bug with Bamboo indicating that they are missing a WWW-Authenticate challenge for Basic credentials in 401 responses.

For my purposes, I will implement 1 and consider 3.

Thanks for your help.
Reply all
Reply to author
Forward
0 new messages