Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

twin robots hourly spidering

1 view
Skip to first unread message

jid...@jidanni.org

unread,
Jan 13, 2008, 10:49:46 PM1/13/08
to
What's the deal, the strategy as it were? Checking every hour independently etc. Apache:
61.135.166.102 - - [13/Jan/2008:16:04:31 -0800] "GET / HTTP/1.1" 200 522 "-" "Baiduspider+
220.181.38.169 - - [13/Jan/2008:16:07:25 -0800] "GET / HTTP/1.1" 200 522 "-" "Baiduspider+
61.135.166.102 - - [13/Jan/2008:17:04:24 -0800] "GET / HTTP/1.1" 200 522 "-" "Baiduspider+
220.181.38.169 - - [13/Jan/2008:17:04:52 -0800] "GET / HTTP/1.1" 200 522 "-" "Baiduspider+
61.135.166.102 - - [13/Jan/2008:18:04:25 -0800] "GET / HTTP/1.1" 200 522 "-" "Baiduspider+
220.181.38.169 - - [13/Jan/2008:18:07:28 -0800] "GET / HTTP/1.1" 200 522 "-" "Baiduspider+
0 new messages