Customized HTML Parsing

[B R]

Hi Jorge,

I need to customize the parsing of HTML wherein I modify some elements
in the HTML DOM, as soon as the page is fetched, before they can be
processed further. I tried adding the customized parse-html plugin to
the nutch-site.xml as

<property>
<name>plugin.includes</name>
<value>protocol-http|scoring-opic|parse-html</value>
</property>

However, it does not seem to work. Could you guide me as to
configuring the same ?

Thanks.

[Jorge Handl]

Try setting the fetcher.parse property in conf/nutch-site.xml to "true".
- Jorge

[B R]

I tried setting the fetcher.parse property in conf/nutch-site.xml to
"true" - that causes Hounder to download all content-types including
images and mp3 files but web pages stopped getting fetched.

I understand that Hounder uses the Nutch fetcher to fetch the data
into a segment directory. After the data is fetched, Hounder processes
the content - parses the HTML content, runs the modules, adds the
outllnks to the next PageDB, etc. I wish to modify the out-links
before they are added to the PageDB.

Would it be possible to do this using :
1) Modifying the fetched HTML content using a Nutch plugin before it
is written into the segment directory
OR
2) Modifying the out-links using a module configured in the
crawler.properties ?

Thanks.

On Aug 31, 10:13 pm, Jorge Handl <jha...@gmail.com> wrote:
> Try setting the fetcher.parse property in conf/nutch-site.xml to "true".
> - Jorge
>

[Jorge Handl]

Using the nutch html parser seems to mess with the outlinks, in my tests I don't get any outlink at all.
So I'd add the url editing code in the com.flaptor.util.parser.HtmlParser class.

- Jorge

[Jorge Handl]

After thinking a bit more about this, I realized you can write a
Crawler module that changes the links before they are added to the new
pagedb. I haven't tried it, but something like this should work:

for (Link link : fetchdocument.getLinks()) {
String url = link.getUrl();
// ... modify the url in some way
link.setUrl(url);
}

- Jorge

On Sep 2, 5:16 pm, Jorge Handl <jha...@gmail.com> wrote:
> Using the nutch html parser seems to mess with the outlinks, in my tests I
> don't get any outlink at all.
> So I'd add the url editing code in the

> com.flaptor.util.parser.HtmlParser<http://code.google.com/p/flaptor-util/source/checkout>class.
>
> - Jorge

[B R]

I modified the com.flaptor.util.parser.HtmlParser class and it worked
well. I'll try out your suggestion of implementing it as a Crawler
module.

However, I did end up having to extract some specific HTML tags while
parsing the content. Let me try moving this functionality also into a
crawler module, along with the modification of the outlinks.

Thanks.

[B R]

Hi Jorge,

Using a Crawler module for modifying outlinks as well as for
extracting HTML content worked very well.

Thanks.

[Jorge Handl]

I'm glad it worked. Is your module generic enough to contribute it to the project?

- Jorge