HTTP clients in clojure

[Eric Tschetter]

Last I checked the various clojure libraries, it seemed like noone has
publicized a set of wrappers/clojure-native implementation of an http
client. I'm wonder if such a thing exists, or has everyone basically
just rolled their own wrapper on top of their favorite Java HTTP
client library?

--Eric Tschetter

[Raoul Duke]

> client.  I'm wonder if such a thing exists, or has everyone basically
> just rolled their own wrapper on top of their favorite Java HTTP
> client library?

i dunno. but, for possible cribbing, there's a Scala wrapper around
HttpClient, which supposedly makes it all Suck Less.

e.g.

http://technically.us/code/x/pour-some-sugar-on-httpclient/

[Chris Dean]

Eric Tschetter <eche...@gmail.com> writes:
> Last I checked the various clojure libraries, it seemed like noone has
> publicized a set of wrappers/clojure-native implementation of an http
> client.

There is (slurp* url) and (reader url) in clojure.contrib.duck-streams

(count (slurp* "http://google.com")) => 4675

Cheers,
Chris Dean

[Brian Doyle]

I wrote clj-web-crawler which wraps the Apache commons client library and made it suck
a little bit less.   I haven't tested it out with the Clojure 1.0 release just yet, but I'll do that
tonight.

http://github.com/heyZeus/clj-web-crawler/tree/master

[Shawn Hoover]

I was recently tipped off to this nascent project: http://github.com/technomancy/clojure-http-client/tree/master

[Phil Hagelberg]

Eric Tschetter <eche...@gmail.com> writes:

Dan Larkin started one a little while ago. I took it and ran with it.

http://github.com/technomancy/clojure-http-client

It consists of two different namespaces, clojure.http.client and
clojure.http.resourcefully. The client code is lower-level and works
pretty close to the underlying API, while resourcefully is higher-level
and is designed for interacting with REST-based APIs. It handles
persisting cookies and it will raise exceptions on requests that fail
rather than just giving a return code. (Still working on that last bit.)

Documentation is sparse, but I think it's pretty straightforward. It
uses the JDK's HTTPURLConnection internally. Some folks have pointed me
towards the Apache Commons HTTP Client, but I haven't seen much that it
offers over what's built-in to the JDK apart from connection pooling, so
I've been content with what I've got.

Dan has suggested that once it's better-tested and stabilized it could
be submitted to Clojure Contrib. I think that might be a good idea.

Let me know if you find it useful. I would love to get some comments.

-Phil

[Stuart Sierra]

On May 6, 8:34 pm, Eric Tschetter <eched...@gmail.com> wrote:
>  I'm wonder if such a thing exists, or has everyone basically
> just rolled their own wrapper on top of their favorite Java HTTP
> client library?

I just use the Apache Commons HTTP client.
-SS

[Richard Newman]

I wrote my own wrapper around the Apache Commons HTTP client,
approximately mirroring AllegroServe's HTTP client. I often find
myself wanting to react to the HTTP response code and response without
the burden of exception handling… after all, a non-200 response is
hardly "exceptional", if an exception was the best way to announce a
code and body, surely we'd use it for success responses, too…

At some point I'll see if I can the OK to publish the source.

[André Thieme]

Depending on what you want to do htmlunit may be interesting:
http://htmlunit.sourceforge.net/
It can do a whole lot more than just doing http requests.

[Perry Trolard]

> Let me know if you find it useful. I would love to get some comments.
> -Phil

Hi Phil,

I'm trying out clojure-http-client, & thus far I like the idea of it
quite a bit: a simple but clever wrapper on built-in JDK APIs, so
provides convenience w/o the burden of external jars.

Quickly, I ran into problems with the resourcefully/with-cookies
macro. Making two changes to resourcefully.clj & altering the
signature from the example made it work (but I didn't test too
rigorously).

Re: altering the signature, the example use of with-cookies in the
file is

(with-cookies (get ...) (post ...) ...)

but the macro expects the first argument to be a (possibly nil?)
cookies map, like this

(with-cookies {} (get ...) (post ...)) OR (with-cookies nil
(get ...) (post ...))

Re: changes to resourcefully, here's the diff:

diff --git a/src/clojure/http/resourcefully.clj b/src/clojure/http/
resourcefully.clj
index 0376df8..358c157 100644
--- a/src/clojure/http/resourcefully.clj
+++ b/src/clojure/http/resourcefully.clj
@@ -44,7 +44,7 @@
map. Cookies will be saved if inside with-cookies block.")
[u# & [headers# body#]]
(let [response# (save-cookies (client/request u# ~(str method)
- headers# *cookies*
body#))]
+ headers#
@*cookies* body#))]
(if (error? response#)
(throw (java.io.IOException. (error-message response#)))
response#))))
@@ -59,5 +59,5 @@ map. Cookies will be saved if inside with-cookies
block.")
"Perform body with *cookies* bound to cookie-map. Responses that
set
cookies will have them saved in the *cookies* ref."
[cookie-map & body]
- `(binding [*cookies* (ref (or cookie-map {}))]
+ `(binding [*cookies* (ref (or ~cookie-map {}))]
~@body))

The gist is that (1), in with-cookies, the parameter "cookie-map" must
be unquoted & thus evaluated to the value passed in -- w/o the unquote
cookie-map gets namespace-qualified to the var resourcefully/cookie-
map, which doesn't exist. (2) in the define-method macro, the
*cookies* ref must be dereferenced when passed to client/request,
because client/request expects a map, not a ref.

Thanks for the lib. I for one would appreciate its inclusion in
contrib once it gets a little more testing.

Best,
Perry

[Phil Hagelberg]

Perry Trolard <tro...@gmail.com> writes:

>> Let me know if you find it useful. I would love to get some comments.
>> -Phil
>
> Hi Phil,
>
> I'm trying out clojure-http-client, & thus far I like the idea of it
> quite a bit: a simple but clever wrapper on built-in JDK APIs, so
> provides convenience w/o the burden of external jars.
>
> Quickly, I ran into problems with the resourcefully/with-cookies
> macro. Making two changes to resourcefully.clj & altering the
> signature from the example made it work (but I didn't test too
> rigorously).

Thanks! I haven't used the cookie functionality much, as I suppose is
obvious. =)

I've committed your changes.

-Phil