A spider's view of Web 2.0 - Ajax and crawlability - Crawling, indexing, and ranking

Message from discussion A spider's view of Web 2.0 - Ajax and crawlability

cass-hacks

View profile

More options Nov 8 2007, 9:04 am

From: cass-hacks

Date: Thu, 08 Nov 2007 06:04:50 -0800

Local: Thurs, Nov 8 2007 9:04 am

Subject: Re: A spider's view of Web 2.0 - Ajax and crawlability

Forward | Print | View thread | Report this message | Find messages by this author

> I don't know about Google, but MSN does exactly that:

That's interesting!!

I've never seen anything like that, although I've never really looked
for sequences like that.

Cool, more access log diving!!! :-()

Thanks for the info!

That said, if snooping is what MSN is actually doing, it wouldn't
surprise me.

Anyhoo, back to the discussion of Flash sites and making them search
engine and accessible friendly. :-)

Craig

> Host: 65.55.2**.***

> /robots.txt
> Http Code: 200 Date: Oct 10 16:07:56 Http Version:HTTP/1.0 Size in
> Bytes: 196
> Referer: -
> Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)/binnengekomen_2005.html
> Http Code: 200 Date: Oct 10 16:08:19 Http Version: HTTP/1.0 Size in
> Bytes: 14723
> Referer: -
> Agent: msnbot/1.0 (+http://search.msn.com/msnbot.htm)

> Host: 65.55.1**.***

> /binnengekomen_2005.html
> Http Code: 200 Date: Oct 10 16:09:03 Http Version: HTTP/1.0 Size in
> Bytes: 14723
> Referer:http://search.live.com/results.aspx?q=dieren
> Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)
> /va.css
> Http Code: 200 Date: Oct 10 16:09:03 Http Version: HTTP/1.0 Size in
> Bytes: 4281
> Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2)

> On 7 nov, 15:34, cass-hacks wrote:

> > > Wouldn't it be
> > > better to sort this issue out by removing those sites first as an easy
> > > fix? Other than that great advice.

> > Detecting that type of cloaking likely has to be done manually, which
> > means a webmaster tools spam report might be helpful.

> > For a bot to detect it, the bot would have to lie and not say it is a
> > crawler. ;-)

Forward