I would say your solution of reusing HttpClient is probably not good
considering this snippet from the Spray docs:
"As you can see from this API the spray-can HttpClient works on the
basis of individual connections. There is no higher-level support for
automatic connection pooling and such, since this is considered the
responsibility of the next-higher application layer."
I looked at the potion of the code where you had issues and it seems
strange it would be causing problems. That HttpConnection needs to be
mutable since Conn instances are reused. But I don't see how two
conns would interfere with each other (though I've only briefly
checked out the code). Would you be able to post a snippet of code
where you are actually sending the request and handling the response?
It also sounds like you are prematurely closing your HttpConduit. If
you are getting a NPE because there is no self then that sounds like
you are sending multiple requests through a single conduit but closing
it once you get the first response. Seems like what you really want
to do is something like:
1. Get a new host from wherever they're coming from
2. Declare a new HttpConduit
3. For each relative URI for that host call sendReceive and collect
all results as a list of Futures (with onComplete callbacks to send
results where needed)
4. Use Akka's sequence or traverse to turn your List[Future[Result]]
into a Future[List[Result]] and attach a callback to that to clean up
your conduit since that will guarantee all responses have come back
and you won't orphan any requests
You can skip #3 if you are OK with processing all the Futures only
after they're all complete but I'm not sure that's ideal for your
situation. If this is a long running process that will be calling the
same host over and over then you may not even need to worry about
cleaning up your conduits until the container shuts down. Like I said
I've not used Spray client much so I've used some assumptions. If you
have any more code you can share that may help diagnose the problem.
Chris
sorry for not getting back to you earlier on this but I still haven't completely resurfaced after the holidays (however, reaction times should be back to normal from now on).
Also: thanks for digging into spray-client as deeply as you did and reporting the issues you have encountered. I'm sure we'll be able to fix any problems with your spray-client use case.
As there appeared to be some confusion about what the exact responsibilities of the spray-can HttpClient and the spray-client HttpConduit are here is some more detail on this:
The spray-can HttpClient is responsible for the low-level reception and rendering of HTTP messages. Its interface allows you to open a connection to a host, use it for one or more request/response cycles and close it eventually. It does not provide any logic on a "super-connection" level, i.e. it does not distribute requests across a bunch of connections or somehow group connections by host. However, a single HttpClient should be able to handle thousands of concurrent connections, which is why there is no point in creating more than one per JVM instance.
The spray-client HttpConduit provides a layer on top of the HttpClient. It manages a bunch of connections for 1 host. So here, you create one HttpConduit per remote host and work with it until you no longer need to talk to that host.
The problem of several HttpConduits getting into each others way like you are describing is certainly a big issue that needs to be addressed.
Would you be able to distill a small test example exhibiting the behavior you were seeing?
Cheers,
Mathias
---
mat...@spray.cc
http://www.spray.cc
very good!
Thanks for the in-depth analysis.
I have created two new issues for spray-client to track progress on the problems you found:
https://github.com/spray/spray/issues/72 and https://github.com/spray/spray/issues/73
Cheers,
Mathias
---
mat...@spray.cc
http://www.spray.cc