Unicode Web Scraping

[Pooya Razavian]

Hello,

Bit of a noob question. So I'm trying to scrape a few sites that require unicode support. So for example trying to get the title of this book returns jumbled characters:

(-> "http://www.brill.nl/publications/evliya-celebis-book-travels" java.net.URL. enlive/html-resource
  (en/select [:h1#page-title]) first :content)

And trying to scrape and Arabic site returns with ?????? all over the place.

(enlive/html-resource (java.net.URL. "http://www.aljazeera.net/portal"))

I'm not sure how I'm supposed to activate unicode support. Thanks for your help in advance!

[Christophe Grand]

Hello,

Enlive is not (and does not include) a full-featured HTTP agent.
When you pass a java.net.URL to a html-resource it call .getContent on it, get an InputStream an then assume UTF-8.
However if you know the actual encoding you can do :

(-> "http://www.brill.nl/publications/evliya-celebis-book-travels" java.net.URL.
  .getContent (java.io.InputStreamReader. "ENCODING GOES HERE")
  enlive/html-resource

  (en/select [:h1#page-title]) first :content)
 

Or use an agent library which will detect the correct encoding and pass the resulting Reader to html-resource.

hth,

Christophe

--
You received this message because you are subscribed to the Google Groups "Enlive" group.
To view this discussion on the web visit https://groups.google.com/d/msg/enlive-clj/-/hYIwARiaid0J.
To post to this group, send email to enliv...@googlegroups.com.
To unsubscribe from this group, send email to enlive-clj+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/enlive-clj?hl=en.

--
Professional: http://cgrand.net/ (fr)
On Clojure: http://clj-me.cgrand.net/ (en)

[Pooya Razavian]

Hi Chris,

Thanks for taking the time to reply. You're suggestion worked wonderfully. The original solution I emailed before actually did work. It was only jumbled on the Mac terminal, which someone over at Stackoverflow pointed out. Thanks!

On Saturday, May 19, 2012 10:28:52 AM UTC+1, Christophe Grand wrote:

Hello,

Enlive is not (and does not include) a full-featured HTTP agent.
When you pass a java.net.URL to a html-resource it call .getContent on it, get an InputStream an then assume UTF-8.
However if you know the actual encoding you can do :

(-> "http://www.brill.nl/publications/evliya-celebis-book-travels" java.net.URL.
  .getContent (java.io.InputStreamReader. "ENCODING GOES HERE")
  enlive/html-resource
  (en/select [:h1#page-title]) first :content)
 
Or use an agent library which will detect the correct encoding and pass the resulting Reader to html-resource.

hth,

Christophe

On Wed, May 16, 2012 at 8:25 PM, Pooya Razavian <poo...@gmail.com> wrote:

Hello,

Bit of a noob question. So I'm trying to scrape a few sites that require unicode support. So for example trying to get the title of this book returns jumbled characters:

(-> "http://www.brill.nl/publications/evliya-celebis-book-travels" java.net.URL. enlive/html-resource
  (en/select [:h1#page-title]) first :content)

And trying to scrape and Arabic site returns with ?????? all over the place.

(enlive/html-resource (java.net.URL. "http://www.aljazeera.net/portal"))

I'm not sure how I'm supposed to activate unicode support. Thanks for your help in advance!

--
You received this message because you are subscribed to the Google Groups "Enlive" group.
To view this discussion on the web visit https://groups.google.com/d/msg/enlive-clj/-/hYIwARiaid0J.
To post to this group, send email to enliv...@googlegroups.com.

To unsubscribe from this group, send email to enlive-clj+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/enlive-clj?hl=en.