If you are wondering what Crawler-Commons is about :
Crawler-Commons is a set of reusable Java components that implement functionality common to any web crawler. These components benefit from collaboration among various existing web crawler projects and reduce duplication of effort.
The artefacts are available from Maven central, simply add the following to your project's POM file.
<dependency>
<groupId>com.github.crawler-commons</groupId>
<artifactId>crawler-commons</artifactId>
<version>0.8</version>
</dependency>
Thanks to all contributors and users and happy crawling!