Limit crawler activity
27 views
Skip to first unread message
zfc
Oct 1, 2012, 8:20:40 AM10/1/12
to glos...@googlegroups.com
My glossword-powered dictionary website has been suspended by hosting company due to issue in CPU limit. I suspect this is due to extensive crawler activity on my website. For instance, Google index all my terms (index.php/term/) and lists (index.php/list/) to its search result. My question is, can I prevent crawlers from indexing the terms and lists by adding this to robots.txt?
Disallow: /index.php/list/
Disallow: /index.php/term/
Will it affect crawler activity on my homepage (index.php)?
Disallow: /index.php/list/
Disallow: /index.php/term/
Will it affect crawler activity on my homepage (index.php)?
Reply all
Reply to author
Forward
0 new messages