Question to favicon crawler ...

ich

unread,

Jul 1, 2015, 2:46:26 AM7/1/15

to common...@googlegroups.com

I can see a request from 66.102.8.170 forwarded for 179.156.176.143 (using google favicon as browser).
Because i use FCrDNS this access has been blocked. Is google not using FCrDNS on each service ?

Greg Lindahl

unread,

Jul 3, 2015, 4:57:27 PM7/3/15

to common...@googlegroups.com

On Tue, Jun 30, 2015 at 11:46:25PM -0700, ich wrote:
> I can see a request from *66.102.8.170 forwarded for *179.156.176.143

> (using google favicon as browser).
> Because i use FCrDNS this access has been blocked. Is google not using
> FCrDNS on each service ?

I'm not really understanding the question... when you crawl, if you
want everything, it's extremely important to be permissive. You can't
trust anyone to reliably use FCrDNS.

It used to be the case that mailservers were pretty good at having
FCrDNS, but not recently. I've never seen any writeup for web servers
mention FCrDNS.

-- greg

ich

unread,

Jul 17, 2015, 11:07:18 AM7/17/15

to common...@googlegroups.com

As google mentions here: https://support.google.com/webmasters/answer/80553?hl=en a webmaster can do a DNS lookup to verify the bot. I use this in all cases to avoid "bad" crawlers from spidering or eating up bandwidth.

ich

unread,

Nov 2, 2015, 2:53:12 AM11/2/15

to Common Crawl

Hello Group !

I ask the question again. Today i got an request from 66.102.7.187 (which belongs to google) and with Google favicon as Browser. The RDNS resolves to google.com so this seems to be a valid access from google. Furthermore 58.140.204.125 has been detected as proxy addresse using the google service inside this access.

How is possible that someone can use a google service for this request or is it another service from google which can be used by everyone and which can impersonate as google favicon browser ?

Reply all

Reply to author

Forward