Parallel http requests

[chinmoy debnath]

I am a newbie in clojure. I need to send multiple http requests in parallel and need to have a call back when response for each request come back. What will be the idiomatic way of doing it in clojure?
Thanks in advacne

[Mikera]

I'd suggest taking a look at the http-kit client library:

http://http-kit.org/client.html

[Timothy Baldridge]

http-kit is great, and here's an example of using it with core.async to manage the callbacks: https://github.com/halgari/clojure-conj-2013-core.async-examples/blob/master/src/clojure_conj_talk/core.clj#L361

Timothy

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Jan Herich]

Maybe it's not exactly what you need, but i did similar thing once - i needed to scrape many linked html resources to extract tree data structure, each request/parse operation took considerable time - around 2 seconds - i was using clj-http/enlive combo (which is actualy Apache HttpClient/ TagSoup under the hood). Therefore i needed to parallelize this operations as much as possible and clojure pmap function provides exactly that (with futures and threads under the hood). Many times i stumbled upon people complaining that use of the pmap actually sloves down rather then speeds the code, but this is because the coordination overhead for managing threads is substantional, so it's probably not useful for tasks where the actual mapping function runs very fast, but in my case, the speedup in comparison to normal map function was massive - here is the code i'm talking about.

Dňa utorok, 31. decembra 2013 9:46:53 UTC+1 chinmoy debnath napísal(-a):

[Glen Mailer]

I can't speak for whether this is a truly idiomatic way of doing this, but this is a macro i've been using for this case.

(defmacro plet [bindings & body]
  "Just like let, but evaluates bindings in parallel"
  (let [bindings (partition-all 2 bindings)
        destructs (map first bindings)
        expressions (map second bindings)
        vars (repeatedly (count bindings) gensym)]
    `(let [~@(mapcat (fn [var expr] `[~var (future ~expr)]) vars expressions)
           ~@(mapcat (fn [destr var] `[~destr (deref ~var)]) destructs vars)]
       ~@body)))

This gives me a similar construct to let, but with each binding evaluated in parallel using a future. For small sets i think this is less overhead than pmap - note that unlike the traditional let, this will not handle multiple bindings for the same variable properly.

Usage would then be

(plet [a (http/get "url1")
       b (http/get "url2")]
  (vec a b))

Or similar

Cheers
Glen

On Tuesday, 31 December 2013 08:46:53 UTC, chinmoy debnath wrote: