[groovy-user] Multiple concurrent requests with HTTPBuilder

[nabblee]

I am trying to send multiple requests through HTTPBuilder concurrently.
Should I be able to do that, or do I have to manage my own pool? Here's
what I see when I attempt to make a new request while a long-running request
is pending:

<Apr 27, 2009 8:35:34 PM EDT> <Warning>
<org.apache.http.impl.conn.SingleClientConnManager> <BEA-000000> <Invalid
use of SingleClientConnManager: connection still allocated.
Make sure to release the connection before allocating another one.>
java.lang.IllegalStateException: Adapter is detached.
at
org.apache.http.impl.conn.AbstractPooledConnAdapter.assertAttached(Ab
stractPooledConnAdapter.java:90)
at
org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPool
edConnAdapter.java:118)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultReq
uestDirector.java:432)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpCl
ient.java:555)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpCl
ient.java:487)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpCl
ient.java:465)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:432)
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:405)
at groovyx.net.http.HTTPBuilder.request(HTTPBuilder.java:358)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMet
hodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
at
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMeta
MethodSite.java:52)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSi
teArray.java:43)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCa
llSite.java:116)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCa
llSite.java:132)

I'm probably misusing it. Is there a way to accomplish what I'm after?

Thanks!

--Lee
--
View this message in context: http://www.nabble.com/Multiple-concurrent-requests-with-HTTPBuilder-tp23268224p23268224.html
Sent from the groovy - user mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email

[Tom Nichols]

Use AsyncHTTPBuilder. The response closure is executed asynchronously
and the request methods return a Future that can be used to determine
the state of the request and pass any information from the response
closure back to the main thread. See
http://groovy.codehaus.org/modules/http-builder/doc/async.html

[Marc Guillemot]

an alternative could perhaps be to use a
MultiThreadedHttpConnectionManager for the underlying HttpClient.

Cheers,
Marc.
--
Web: http://www.efficient-webtesting.com
Blog: http://mguillem.wordpress.com

[Marc Guillemot]

an alternative could perhaps be to use a
MultiThreadedHttpConnectionManager for the underlying HttpClient.

Cheers,
Marc.
--
Web: http://www.efficient-webtesting.com
Blog: http://mguillem.wordpress.com

[Marc Guillemot]

an alternative could perhaps be to use a
MultiThreadedHttpConnectionManager for the underlying HttpClient.

Cheers,
Marc.
--
Web: http://www.efficient-webtesting.com
Blog: http://mguillem.wordpress.com

[Tom Nichols]

That's exactly what AsyncHttpBuilder does. But there's a little more
to it than that - it also wraps a ThreadPoolExecutor so that requests
return a Future that can be used to retrieve the result.

[nabblee]

I'm having a lot of trouble making AsyncHTTPBuilder work.

When I was using HTTPBuilder, my requests looked like this (from one of the
examples):

http.URL = 'https://...'
def httpResult = http.request(POST,TEXT) { req ->
send URLENC, [sSessionID:binding.sessionid, lOrderID:id]

response.success = { resp, reader ->
assert resp.statusLine.statusCode == 200
def xml = new XmlSlurper().parse(reader)
...
}

response.'404' = { resp ->
println "Retrieval failed: $resp"
}

I'm trying to turn these into AsyncHTTPBuilder requests, but I keep getting
a NullPointerException that I don't understand. Here's what I've got:

def httpResult = http.post( path:'/',
body:[sUserName:username, sPassword:password],
contentType:URLENC) {resp,html ->
}

while(!httpResult.done) {
Thread.sleep(1000)
}

def xml = httpResult.post()
// do something with response

The stacktrace originates at the very first line (def httpResult = ...), but
here is the rest of it:

Exception in thread "Thread-1" java.lang.NullPointerException
at groovyx.net.http.URIBuilder.convertToURI(URIBuilder.java:85)
at
groovyx.net.http.HTTPBuilder$SendDelegate.setPropertiesFromMap(HTTPBuilder.java:831)
at groovyx.net.http.HTTPBuilder.post(HTTPBuilder.java:323)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoCachedMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:229)
at
org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:52)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:43)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)

I need to use a different URL for each request, so I tried to put in a base
uri when instantiating AsyncHTTPBuilder:

http = new AsyncHTTPBuilder(
poolSize : 20,
uri : baseUri,
contentType : URLENC )

I've tried using path:'/xyz' in the actual call. I've tried uri, url,
uri.path, etc. I have yet to find the magic incantation that works.

All I want to do is allow multiple HTTP POSTs to occur in parallel. I can't
figure out the proper syntax for a post, and I'm not sure why I'm getting an
NPE. Nothing seems to be null, but the stacktrace indicates a problem with
the URI.

Thanks,
Lee

--
View this message in context: http://www.nabble.com/Multiple-concurrent-requests-with-HTTPBuilder-tp23268224p23360431.html

[nabblee]

Okay, my bad. I was running version 0.4. I just upgraded to the latest
0.5-SNAPSHOT (http-builder-0.5.0-20090502.024916-2-all.zip). So, I'm not
getting the same NPE problem as before, but I'm not out of the woods,
either.

http = new AsyncHTTPBuilder(
poolSize : 20,

uri : 'https://oxbranch.optionsxpress.com/',
contentType : URLENC )

def httpResult = http.post(
path:'https://oxbranch.optionsxpress.com/accountservice/account.asmx/GetOxSessionWithSource',

body:[sUserName:username, sPassword:password],
contentType:URLENC) {resp,html ->

println "resp = ${resp.dump()}"
println "html = $html"
}

while(!httpResult.done) {
println "Outside login..."
Thread.sleep(1000)
}

println httpResult.dump()
def xml = httpResult.get()
println "httpResult.get() returns $xml"

Here's the output:
Outside login...
Outside login...
Outside login...
Outside login...
resp = <groovyx.net.http.HttpResponseDecorator@ef4504 headers=null
responseBase=org.apache.http.message.BasicHttpResponse@9c176c
responseData=null>
html = [:]
<java.util.concurrent.FutureTask@82d603
sync=java.util.concurrent.FutureTask$Sync@1b09282[State = 2, empty queue]>
httpResult.get() returns null

Why does get() return null? Am I supposed to be doing something else to
retrieve the response?

Thanks again!

--
View this message in context: http://www.nabble.com/Multiple-concurrent-requests-with-HTTPBuilder-tp23268224p23360779.html

[Tom Nichols]

Since you are defining your own response handler closure, needs to
return whatever you want Future.get() to return. If you do not pass a
closure as the last argument to post(), it will use the default
response handler which does just this.

So your example should look like this:
def httpResult = http.post(...) { resp, html ->

println "resp = ${resp.dump()}"
println "html = $html"

return html

[nabblee]

Thanks for your response, Tom, but it's still not working. :-( I'm getting
an empty map back...

def httpResult = http.post(

uri:'https://oxbranch.optionsxpress.com/accountservice/account.asmx/GetOxSessionWithSource',
body:[sUserName:username, sPassword:password, sSource:binding.sourceid,
sSessionID:''],
contentType:URLENC
) { resp, xml ->

println "resp = ${resp.dump()}"

println "xml = $xml"
return xml
}

while(!httpResult.done) {
println "Outside login..."
Thread.sleep(1000)
}

println "httpResult: ${httpResult.dump()}"

def xml = httpResult.get()
println "httpResult.get() returns $xml"

Output:

Outside login...
Outside login...
Outside login...
Outside login...

resp = <groovyx.net.http.HttpResponseDecorator@9c176c headers=null
responseBase=org.apache.http.message.BasicHttpResponse@1c5ddd3
responseData=null>
xml = [:]
httpResult: <java.util.concurrent.FutureTask@82d603

sync=java.util.concurrent.FutureTask$Sync@1b09282[State = 2, empty queue]>

httpResult.get() returns [:]

Maybe I'm not structuring the arguments to post() correctly? It takes about
the amount of time I expect, but I don't see a usable reply.

Thanks for any insights,
Lee

--
View this message in context: http://www.nabble.com/Multiple-concurrent-requests-with-HTTPBuilder-tp23268224p23361021.html

[nabblee]

Okay, I've got it now. The problem was the contentType. It needed to be
XML, since that's what comes back. I had URLENC, because that's what I'm
sending to them, but I guess maybe that's implied with a POST? Anyway, I
guess that's why I was getting an empty map back, since it had no success
finding key/value pairs in the XML response. Once I changed it to XML, I
started seeing the contents I expected.

Not to look a gift horse in the mouth, because I do appreciate HTTPBuilder,
but it really needs more docs in order to be understood. The few examples
that are provided don't give enough data points to allow extrapolation. And
POST examples are always more useful than GET examples. ;-)

Sincere thanks for the library, which I make lots of use of.

Cheers,
Lee
--
View this message in context: http://www.nabble.com/Multiple-concurrent-requests-with-HTTPBuilder-tp23268224p23361126.html

[nabblee]

I do have another question about not passing a closure to post(). When I try
that, I get an error:

def httpResult = http.post(

uri:'https://oxbranch.optionsxpress.com/accountservice/account.asmx/GetOxSessionWithSource',
body:[sUserName:username, sPassword:password, sSource:binding.sourceid,
sSessionID:''],

contentType:XML
)

May 3, 2009 7:48:39 PM groovyx.net.http.AsyncHTTPBuilder$1 call
SEVERE: Exception thrown from request delegate:
groovyx.net.http.HTTPBuilder$RequestConfigDelegate@13a0212
java.lang.IllegalArgumentException: Response closure must accept one or two
parameters
at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:459)
at
groovyx.net.http.AsyncHTTPBuilder.doRequestSuper(AsyncHTTPBuilder.java:119)
at groovyx.net.http.AsyncHTTPBuilder.access$000(AsyncHTTPBuilder.java:55)
at groovyx.net.http.AsyncHTTPBuilder$1.call(AsyncHTTPBuilder.java:103)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

So I seem to have no choice but to create a closure that simply returns the
second parameter. Why is that different from what you said would be the
default?

Thanks,
Lee

Tom Nichols wrote:
>
> Since you are defining your own response handler closure, needs to
> return whatever you want Future.get() to return. If you do not pass a
> closure as the last argument to post(), it will use the default
> response handler which does just this.
>

--
View this message in context: http://www.nabble.com/Multiple-concurrent-requests-with-HTTPBuilder-tp23268224p23361185.html

[Tom Nichols]

Yes, in this case the post method assumes the request body is
url-encoded. I suppose I should have mentioned it in the
documentation example, but it is mentioned in the JavaDoc:
http://groovy.codehaus.org/modules/http-builder/apidocs/groovyx/net/http/HTTPBuilder.html#post(java.util.Map,%20groovy.lang.Closure)

If you use RESTClient, it does not automatically assume a url-encoded
request, since you could i.e. be POSTing XML data. HTTPBuilder.post
is meant as a convenience method since HTTP form POSTs are a very
common task. It's the only method that explicitly assumes a different
request content type. I suppose I should have made that clear in the
website documentation. Thanks for pointing that out.

Normally, in any case where contentType is specified, it defines the
request _and_ response contentType. If they're different, there's a
'requestContentType' that you can specify as well. The 'post'
method's javadoc also links to this method:
http://groovy.codehaus.org/modules/http-builder/apidocs/groovyx/net/http/HTTPBuilder.RequestConfigDelegate.html#setPropertiesFromMap(java.util.Map)
which defines all named parameters for all of the convenience methods
used in HTTPBuilder and child classes (RESTClient and
AsyncHTTPBuilder).

FYI, if you were to do the same using the request() method, you would
do it like so in order to define both the request and response
content-type:
http.request( 'https://oxbranch.optionsxpress.com/accountservice/account.asmx/GetOxSessionWithSource',
XML ) {
send URLENC, [sUserName:username, sPassword:password,
sSource:binding.sourceid, sSessionID:'']
response.success = { resp, xml ->

println "resp = ${resp.dump()}"
println "xml = $xml"
return xml
}
}

I tend to prefer this form as it's easier to read: "I'm requesting
this URL to send me XML, and sending a url-encoded map as the body.
Then on a successful response do (xyz)...

'send' is defined here:
http://groovy.codehaus.org/modules/http-builder/apidocs/groovyx/net/http/HTTPBuilder.RequestConfigDelegate.html#send(java.lang.Object,%20java.lang.Object)
and it's what defines the extra methods in the request closure's scope.

Hope this helps explain things.
-Tom

[Tom Nichols]

This appears to be a bug related to my use of MethodClosure from a
sub-class. I'm looking into it now. But yes, a work-around is to
define your own response handler for now.

[David Rosenstark]

I am using HttpBuilder to download files -- mp3, etc.

I get the following warning:
WARNING: Cannot find parser for content-type: audio/mpeg -- using
default parser.

The default parser is fine for me since I am just saving the file
locally.
Perhaps this should be an INFO log and not WARNING?

[Tom Nichols]

Well, that's just it -- there is no 'default parser' -- you'll
actually get null for the second argument in your response handler
closure. I suppose I could make a default parser that just passes the
InputStream to the response handler. But in your case, you could do
something like this:

def mp3Dir = new File( 'mp3s/' )
mp3Dir.mkdirs()

def http = new HTTPBuilder( 'http://www.somesite.com' )
http.handler.'audio/mpeg' = { resp ->
def filename =
resp.getFirstHeader('Content-Disposition').elements[0].getParameterByName('filename').value
def destFile = new File( mp3Dir, filename )
destFile << resp.entity.content
}

Then just call:
http.get( path : 'dir/song.mp3' )

and it will save the file directly to your folder or throw an
exception if the request failed. Note that the above code is
untested.

I'm reluctant to change the log level for that message, because I
imagine most people will expect the automatic parsing, and I suspect
I'd start getting a lot of "why am I not getting any parsed data"
questions if that log message was hidden.

-Tom

[David Rosenstark]

Thanks, that seems pretty simple. All I have to do is move my current code.

My point was that if I have a default parser in my code, like this:
http.get(path: strUrl) {resp, reader ->
.....
}
Then, there is no reason for the warning since I am handling all cases the same.
In my case, the code is generic -- it can read any binary data and write to the file so why define a specific handler?

I need to look to confirm that the " destFile << resp.entity.content" really works. I think when I started this project I tried that and in the end had to read the data using eachByte so it would be read correctly (binary).

[David Rosenstark]

Tom, sorry but one more question about your response.

My response closure takes 2 parameters currently -- the resp and the reader, so I have defined my own catch all parser.
Is there a way to tell the class that this parser is the one to use for all content types?

-----Original Message-----
From: Tom Nichols [mailto:tmni...@gmail.com]
Sent: Wednesday, May 06, 2009 4:01 PM
To: us...@groovy.codehaus.org

[Tom Nichols]

I realized in my previous code example I have a
http.handler.'audio/mpeg' -- which should be http.parser.'audio/mpeg'
= { ... }

I just remembered what will actually do what you want --
http.contentType = ContentType.BINARY

This will cause all responses to be parsed using
http://groovy.codehaus.org/modules/http-builder/apidocs/groovyx/net/http/ParserRegistry.html#parseStream(org.apache.http.HttpResponse)
which really just passes the underlying
HttpResponse.getEntity().getContent() as the second parameter to the
response closure. And the stream will automatically be closed after
your response closure returns.

2009/5/6 David Rosenstark <David.Ro...@idt.net>:

[David Rosenstark]

Two questions:
1. When am I setting the contentType after creating my HttpBuilder?
2. All this returns is an outputstream which does not seem that different from my current code

http.get(path: strUrl) {resp, reader ->

reader.eachByte {
outFile.write(it)
}
outFile.close()
}

(By the way, I had decided to suppress the log for this class alone in log4j config to solve the issue)

[Tom Nichols]

You know, I was tired this morning when I read your initial question
and if I had woken up a little bit first I would have realized that
you're already doing what I was trying to suggest. I also forgot that
you do in fact get an InputStream passed to the response handler by a
default parser (the parseStream method); I was thinking the parsed
data would be null if it couldn't find a suitable parser. (BTW -
looking at your code example, just remember it is a stream, not a
reader - reader suggests a conversion from bytes to characters based
on a certain character set).

The reason I suggested passing a BINARY contentType parameter is
because this would force the HTTPBuilder to use
ParserRegistry.parseStream, which is what the default parser does
anyway (but you wouldn't get that warning).

Anyway, if all I've done is serve to confuse you, I realized it is
probably worthwhile to be able to manipulate the default parser
behavior, so that's what I've done in the ParserRegistry class. And
assuming one did set a default parser, it would be annoying to see a
warning message for an expected behavior. So I will in fact change
that log level.

But after all that, and reading your emails a couple more times, it
looks like you already had the solution; changing the log level for
ParserRegistry was probably the easiest thing to do. The one thing
I'd suggest is that iterating over each byte and writing a byte at a
time will probably be slow, but if it works for you then great :)

-Tom

2009/5/6 David Rosenstark <David.Ro...@idt.net>:

[David Rosenstark]

If what you are saying is correct, that it is a stream, why does the code work as a reader?
And what method can I use to iterate over it in groovy -- I know of eachByte and eachLine, what would be my alternative?
As for efficiency, the main problem was output and by using a bufferedwriter, performance improved. If I could read blocks in groovy that would be great and not have to lose my each... Closure solution.

Thanks,
David

-Tom

>>> - To unsubscribe from this list, please visit:

[Tom Nichols]

Hi David,

Sorry I never replied to your latest email; It sounds like you've
found a suitable solution by now.

If you still have some questions, I'd be happy to help. It would
probably be helpful to me if you send a code example too.

-Tom

2009/5/7 David Rosenstark <David.Ro...@idt.net>: