doing a Google search from Clojure?

[Rich Morin]

I've been successfully using slurp and laser to harvest and pull
apart some web pages. However, I can't figure out how to use
Google Search from my code.

My first thought was to use the Google Search API, but after
a lot of frustration in trying to get and use an API key, I
gave up on that.

My next thought was to slurp in a page from the interactive
Google Search facility, using the URL from Advanced Search:

"http://www.google.com/search?hl=en&as_q=..."

However, this gives me a 403 nastygram:

IOException Server returned HTTP response code: 403 for URL:
https://www.google.com/search?hl=en&as_q=&as_epq=...
sun.net.www.protocol.http.HttpURLConnection.getInputStream
(HttpURLConnection.java:1436)

Has anyone here, by chance, been able to do this sort of thing?

-r

--
http://www.cfcl.com/rdm Rich Morin
http://www.cfcl.com/rdm/resume r...@cfcl.com
http://www.cfcl.com/rdm/weblog +1 650-873-7841

Software system design, development, and documentation

[Cedric Greevey]

Change your code to it spoofs a common browser user-agent, change your DHCP-assigned IP address, and try again. They're probably trying to obstruct bots from making overwhelming numbers of requests or something. As long as you don't flood them with requests at a higher rate than a human would generate by clicking, I don't see any ethical issue with circumventing their countermeasures, especially not if the search will be triggered by a user input to your application anyway.

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jonathan Fischer Friberg]

Found some info here:

http://stackoverflow.com/questions/3727662/how-can-you-search-google-programmatically-java-api

Jonathan

[juan.facorro]

Setting the user agent did the trick, at least in my case.

(ns google-search
  (:import [java.net URL URLEncoder]))
 
(def google-search-url "http://www.google.com/search?q=")
(def user-agent "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172")
 
(defn open-connection [url]
  (doto (.openConnection url)
        (.setRequestProperty "User-Agent" user-agent)))
 
(defn get-response [url]
  (let [conn (open-connection url)
        in   (.getInputStream conn)
        sb   (StringBuilder.)]
    (loop [c (.read in)]
      (if (neg? c)
        (str sb)
        (do 
          (.append sb (char c))
          (recur (.read in)))))))
 
(defn search [query]
  (let [url (URL. (str google-search-url (URLEncoder/encode query)))]
    (get-response url)))
    
(spit "response.html" (search "URLEncoder java 7"))

HIH,

Juan

[Jim - FooBar();]

On 22/03/13 15:00, juan.facorro wrote:

        (do 
          (.append sb (char c))

do you really need the 'do'?

Jim

[Juan Martín]

Yes, the do is necessary since the character needs to be appended to the StringBuilder and recur needs to be called after doing that.

I actually took the code from the clojure.core/slurp function :).

Cheers,

Juan

--

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/QwKmsLwLhjE/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Juan Facorro

[Jim - FooBar();]

ooops! I'm really sorry! my bad!

JIm

[Rich Morin]

On Mar 22, 2013, at 08:00, juan.facorro wrote:
> Setting the user agent did the trick, at least in my case.

Thanks! Using your code, I was able to bring in a page and
write it to a file. I was then able to confirm that it had
the expected content. FTW!

I still think this should be easier, but now I'm back on a
productive path.

[juan.facorro]

I gave the code another look and remembered that slurp can actually handle a bunch of types as input, so I just passed the InputStream from the connection and got the same results. Additionaly in the code I posted before, the get-response function was never closing the stream, which slurp does.

(ns google-search
  (:import [java.net URL URLEncoder]))
 
(def google-search-url "http://www.google.com/search?q=")
(def user-agent "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.172")
 
(defn open-connection [url]
  (doto (.openConnection url)
        (.setRequestProperty "User-Agent" user-agent)))
 
(defn get-response [url]
  (let [conn (open-connection url)

        sb   (StringBuilder.)]
    (slurp (.getInputStream conn))))
 
(defn search [query]
  (let [url (java.net.URL. (str google-search-url (URLEncoder/encode query)))]
    (get-response url)))
    
(spit "response.html" (search "clojure google"))

J

[Armando Blancas]

Rich, you may want to check out clojure-http-client.

https://github.com/technomancy/clojure-http-client

(require '[clj-http.client :as client])

(spit "result.html" (client/get "http://www.google.com/search?q=clojure"))

[Anthony Grimes]

clojure-http-client is more or less unmaintained. https://github.com/dakrone/clj-http is the canonical http client these days.

Lazybot has a plugin for doing this with the google ajax api, if that's helpful. No API key needed. https://github.com/flatland/lazybot/blob/develop/src/lazybot/plugins/google.clj

[Armando Blancas]

Thanks, Anthony; will use that one.