CSC326 Project

[Nazanin]

We are trying to test our search engine with multiple pages such as wikipedia and we get HTTP 403/400 error codes and the code is not capable of indexing the page as it can't open it. Should we be able to handle HTTP errors?

Thanks!

[Wesley May]

If a page that the crawler tries to access returns an HTTP error, I think it's fine to just ignore that link. if you can't access the page it won't be of any use anyways.

[Nazanin]

Thanks! That's right, the problem occurs when crawler tries to open up the page, I was actually able to hit the page on the browser.