URL Normalization / Canonicalization

[Avi Hayun]

Hi,

When crawling, normalization of URLs is quite important as you don't want to crawl the same URL twice or more...

Wikipedia gives an explicit picture of how to do it

Upon looking at the source code of Nutch, Heritrix & crawler4j it seems that all of them have implemented their own URL Normalizer / Canonicalizer.

Do you think that a URL Normalizer is a good candidate for Crawler-Commons ?

[Julien Nioche]

Hi 

See https://github.com/crawler-commons/crawler-commons/issues/74

J.

--
You received this message because you are subscribed to the Google Groups "crawler-commons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crawler-commo...@googlegroups.com.
Visit this group at http://groups.google.com/group/crawler-commons.
For more options, visit https://groups.google.com/d/optout.

--

Open Source Solutions for Text Engineering

http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble

[Avi Hayun]

Thanks Julien,

Can you point me to the BasicUrlNormalizer class in github ? - I can't seem to find it...  (I get stuck here: https://github.com/apache/nutch/tree/trunk/src/java/org/apache/nutch/net )

[Julien Nioche]

Sure, it's here https://github.com/apache/nutch/tree/trunk/src/plugin/urlnormalizer-basic

[Avi Hayun]

Nice

I will add it to the issue as a candidate for an initial url normalizer

[Avi Hayun]

Looking at the thread in on storm crawler (https://github.com/DigitalPebble/storm-crawler/issues/120)

I will take it as a vote of +1 for adding the url normalizer from Ken and Julien

Which answers my initial question.

Any further conversations on this issue should be on issue #74