Conifer undergoing heavy bot activity today

[Daniel Scott]

Earlier today Conifer had some periods of slowness and complete outages. It looks like the catalogue has both been unusually active today, mostly because of a new set of search engine "bots" that are requesting the page behind every link they can find for Conifer.

Well-behaved bots (such as those for the Google and Bing search engines) will notice that a system is responding slowly and will throttle their requests until system response time returns to normal.

However, either because the new bots are not well-behaved, or because there are a number of them operating at the same time, our server has been under an unusually heavy load and consequently slows down or stops responding entirely.

I've been looking at the server logs trying to determine if there are some URLs that are causing problems in particular, but at this point I don't have a short-term solution ready at hand to mitigate the situation. Blocking the bots would only address the symptom of the problem, not the cause, but if this continues I'll start building up a blacklist again...

--

Dan Scott
Associate Librarian & Chair, Library and Archives
Bibliothécaire agrégé & Directeur, Bibliothèque et archives
Laurentian University / Université Laurentienne
Phone: 705-675-1151 x3315