Make LDSpider behave like a non linked data spider.

[Altruist]

Hello All

Is it possible for the LDSpider to behave like regular crawler which does not neccesarily looks for semantic links but follows regular hyperlinks and crawls based on the seedfile.

I know that this defeats the purpose of LDSpider but there are certain features of LDS that I need but I need it to function like a regular crawler.I am using the Java API.Any guidance would be greatly appreciated.

Thank You.

Thanks.

[Andreas Harth]

Hi,

you could set up Any23 which may have HTML handlers. You have to
make sure that the links get parsed out and fed back into the queue
(IIRC via the LinkHandler). Also, you might need to change the
Accept header and the Content-Type header check when downloading.

Good luck,
Andreas.