Attempt At Futures

[Chris White]

Spoiler alert: I'm really really new to this language so don't expect quality code

In an attempt to see how futures work I'm trying to make code that does the following:

1. Take a list of sites
   
2. Loop through the sites and retrieve the HEAD content via a future for each individual site
   
3. In the main loop once the futures have been created print the HEAD content as each future completes

Now I have this code which I'm getting stuck at:

(ns cjr-http-test.core
  (:require [clj-http.client :as client]))

(defn get-head-response-for-sites
  [sites]
  (map (fn [site] (future (client/head site))) sites))

(doseq [head-data (get-head-response-for-sites '("http://www.google.com" "http://www.yahoo.com" "http://www.bing.com"))]
  (println (deref head-data)))

(shutdown-agents)

(Please note I know that the list of sites I'd normally expect to be something from a DB/text file. I'm just trying to get it working without adding extra things to think about)

So get-head-response-for-sites is where I'm trying to do #2. It gets me a list of futures that I can use. Where  I'm having trouble is that the current println line, which is where I'm trying to deal with #3, blocks due to deref-ing so it's basically not really all that different than if I did it non threading.

What I (think) I need is something that keeps looping through all the futures, checking their status, and println’ing the result when something has come back. This will be repeated until all the futures are done. The reason I want this is that for example if the first site takes 3 minutes to respond I want the other two sites to print their HEAD content as soon as it’s retrieved. Here's what I'm trying to figure out in order of importance:

1. How do I get a constant loop through the futures, println'ing the HEAD content as they finish, until all futures are finished?
2. Is there a better way to structure this?
3. Is there something in Clojure/contrib that's better suited for this?
4. Is there a 3rd party library better suited for this?

Thanks for any and all response. Once again I apologize for the not so pro code but some code is better than nothing I hope.

- Chris White ( @cwgem )

[Kevin Downey]

The thing to remember is map is lazy, so you are lazily (on demand)
creating a bunch of futures. Then doseq walks through those futures,
demanding one at a time. You deref the future immediately after doseq
requested it from the lazy-seq and are blocking on its completion then
doseq can move on to demanding the next future be created.

This effectively creates a future and then waits on its completion
before creating the next future. So you end up with everything happening
in sequence.

On 04/08/2016 10:38 PM, Chris White wrote:
> Spoiler alert: I'm really really new to this language so don't expect
> quality code
>
> In an attempt to see how futures work I'm trying to make code that does
> the following:
>

> 1. Take a list of sites
> 2. Loop through the sites and retrieve the HEAD content via a future
> for each individual site
> 3. In the main loop once the futures have been created print the HEAD

> 1. How do I get a constant loop through the futures, println'ing the

> HEAD content as they finish, until all futures are finished?

> 2. Is there a better way to structure this?
> 3. Is there something in Clojure/contrib that's better suited for this?
> 4. Is there a 3rd party library better suited for this?

>
> Thanks for any and all response. Once again I apologize for the not so
> pro code but some code is better than nothing I hope.
>
>
> - Chris White ( @cwgem )
>

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to clojure+u...@googlegroups.com
> <mailto:clojure+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.


--
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

[Gary Verhaegen]

You could:

* Create all futures *without* deref'ing them, so they all start in parallel;
* Loop through the futures, asking them if they have finished, and print those that have (and remove them from the list)

But if you want to get each result as it comes back, it's probably a better fit for core.async than plain futures. I would suggest creating a channel with multiple producers and a single consumer, where each producer gets a site and the consumer prints the result. That way, you get the results as they come.

Here is a rough draft:

(ns cjr-http-test.core
  (:require [clj-http.client :as client]

            [clojure.core.async :as async :refer [<!! >!!]]))

(defn get-heads
  [sites]
  (mapv (fn [site] (future {:head (client/head site)
                            :url site}))
        sites))

(def sites ["http://www.google.com"
            "http://www.yahoo.com"
            "http://www.bing.com"])

(defn use-futures
  []
  (let [head-requests (get-heads sites)]
    (loop [to-check head-requests checked []]
      (cond (and (empty? checked) (empty? to-check))
            :finished

            (empty? to-check)
            (recur checked [])

            (realized? (first to-check))
            (do (-> to-check first deref :url println)
                (recur (rest to-check) checked))

            :else (recur (rest to-check) (cons (first to-check) checked))))))

(defn use-async
  []
  (let [ch (async/chan)
        producers (mapv (fn [site]
                          (doto (Thread. #(>!! ch {:head (client/head site)
                                                   :url site}))
                            (.start)))
                        sites)
        close-chan (doto (Thread. (fn []
                                    (mapv #(.join %) producers)
                                    (async/close! ch)))
                     (.start))]
    (loop [v (<!! ch)]
      (if (nil? v) :finished
        (do (-> v :url println)
            (recur (<!! ch)))))))

[Leonardo Borges]

<shameless plug>

Something like imminent might be useful here. In particular the section about combinators: https://github.com/leonardoborges/imminent#combinators

</shameless plug>

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.