Is an APP ENGINE Application Attacking websites? - DOTS user agent / browser
378 views
Skip to first unread message
Viper
Aug 8, 2011, 1:57:34 AM8/8/11
to google-a...@googlegroups.com
Apparently this has been reported in several other places on Google
http://www.google.com/support/forum/p/Webmasters/thread?tid=6dd38e35376e723e&hl=en
http://www.google.com/support/forum/p/Google+Apps/thread?tid=0a4d894e858660f6&hl=en
http://www.google.com/support/forum/p/Webmasters/thread?tid=1ef5e9959bb406c1&hl=en
And no one seems to know what it is, but found a reference for one of the IP addresses having to do something with the Google App Engine
Has anyone noticed the new DOTS browser / spider / user agent browsing/scanning their site from Google IP Addresses? Here is an example of a log entry from one
Log File: X:\W3SVC1\ex110804.log
Log Entries For:74.125.64.84
Browser: dots
Referrer:
Started: 8/4/2011 5:20:26 PM
2011-08-04 21:20:26 GET /--------------------- 74.125.64.84 dots - - www.------- 403 6 0 5521 109 62
1 Log Entries
They seem to be coming from blocks
74.125.0.0 - 74.125.255.255
and a few other Google blocks.
The problem is that had to no enable the Plus (+1 button) on a site because apparently the Spider (it uses a different user agent) comes from the same blocks as this mysterious DOTS browser / spider / useragent.
Haven't seem to find an concrete information as to what the purpose of DOTS is and whether it should be trusted or not.
Anyone have any ideas?
http://www.google.com/support/forum/p/Webmasters/thread?tid=6dd38e35376e723e&hl=en
http://www.google.com/support/forum/p/Google+Apps/thread?tid=0a4d894e858660f6&hl=en
http://www.google.com/support/forum/p/Webmasters/thread?tid=1ef5e9959bb406c1&hl=en
And no one seems to know what it is, but found a reference for one of the IP addresses having to do something with the Google App Engine
Has anyone noticed the new DOTS browser / spider / user agent browsing/scanning their site from Google IP Addresses? Here is an example of a log entry from one
Log File: X:\W3SVC1\ex110804.log
Log Entries For:74.125.64.84
Browser: dots
Referrer:
Started: 8/4/2011 5:20:26 PM
2011-08-04 21:20:26 GET /--------------------- 74.125.64.84 dots - - www.------- 403 6 0 5521 109 62
1 Log Entries
They seem to be coming from blocks
74.125.0.0 - 74.125.255.255
and a few other Google blocks.
The problem is that had to no enable the Plus (+1 button) on a site because apparently the Spider (it uses a different user agent) comes from the same blocks as this mysterious DOTS browser / spider / useragent.
Haven't seem to find an concrete information as to what the purpose of DOTS is and whether it should be trusted or not.
Anyone have any ideas?
Greg Darke (Google)
Aug 8, 2011, 6:30:25 AM8/8/11
to google-a...@googlegroups.com, carolinas...@gmail.com
Do you have the full useragent?
App engine applications state that they are running on google app
engine in their user agent. They also state the application id that is
making the request. If the user agent does not contain google app
engine, then it is not an app engine application making the requests.
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/qMxZ0KLbzy4J.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
Message has been deleted
Viper
Aug 8, 2011, 11:38:04 AM8/8/11
to google-a...@googlegroups.com
This is all that comes through on the logs (a few examples - URLS have
been removed) As you can see, they are 403'd right now until it is
determined what their purpose is.
2011-08-08 00:00:25 GET /-------------------74.125.44.
2011-08-08 00:00:25 GET /-------------------74.125.44.
89 dots - - www.------------------- 403 6 0 5521 109 125
2011-08-08 15:14:54 GET /---------------------- 74.125.66.84 dots - - www.---------------- 403 6 0 5521 109 265
And here is googlebot for comparison
2011-08-08 00:00:00 GET /------------------------- 66.249.72.105 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - - www.----------------------- 200 0 0 15116 256 890
And the google web preview for comparison. As you see here, we have had to 403 the google web preview due to the fact that the DOTS crawler / bot / spider or ??? sometimes comes from this same IP block as well.
2011-08-08 15:22:05 GET /------------------------------ 209.85.226.88 Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/534.24+(KHTML,+like+Gecko;+Google+Web+Preview)+Chrome/11.0.696+Safari/534.24 - - www.------------------- 403 6 0 5521 338 375
And other Google Feedfetcher for comparison
2011-08-08 09:42:32 GET /rss.php------ 209.85.238.149 Feedfetcher-Google;+(+http://www.google.com/feedfetcher.html;+1+subscribers;+feed-id=--------------------------------------) - - www.------------------ 200 0 0 13069 334 1531
2011-08-08 15:14:54 GET /---------------------- 74.125.66.84 dots - - www.---------------- 403 6 0 5521 109 265
And here is googlebot for comparison
2011-08-08 00:00:00 GET /------------------------- 66.249.72.105 Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) - - www.----------------------- 200 0 0 15116 256 890
And the google web preview for comparison. As you see here, we have had to 403 the google web preview due to the fact that the DOTS crawler / bot / spider or ??? sometimes comes from this same IP block as well.
2011-08-08 15:22:05 GET /------------------------------ 209.85.226.88 Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/534.24+(KHTML,+like+Gecko;+Google+Web+Preview)+Chrome/11.0.696+Safari/534.24 - - www.------------------- 403 6 0 5521 338 375
And other Google Feedfetcher for comparison
2011-08-08 09:42:32 GET /rss.php------ 209.85.238.149 Feedfetcher-Google;+(+http://www.google.com/feedfetcher.html;+1+subscribers;+feed-id=--------------------------------------) - - www.------------------ 200 0 0 13069 334 1531
Brandon Wirtz
Aug 8, 2011, 3:03:01 PM8/8/11
to google-a...@googlegroups.com, carolinas...@gmail.com
Dots is a project in Google, and the guy working on it is a <insert a word
that starts with F and ends with tard>. Someone needs to give the guy a
sandbox that doesn't connect to the real world cause when the thing goes hay
wire it will crawl a site for 300k pages in a matter of hours.
that starts with F and ends with tard>. Someone needs to give the guy a
sandbox that doesn't connect to the real world cause when the thing goes hay
wire it will crawl a site for 300k pages in a matter of hours.
I think the project is related to maps somehow. It used to only "attack"
sites with addresses in the page code, or sites with a GeoTag in their RSS
feeds.
I think the guy didn't get the "No Evil" memo.
Viper
Aug 8, 2011, 5:28:06 PM8/8/11
to google-a...@googlegroups.com, carolinas...@gmail.com
Ah, finally some information. Ok, so now we know that this is a "not so good" Google project as this spider definitely doesn't behave itself - what can be done?
As it stands right now, 75.125 is completely blocked because this thing is hitting from IP's all across the range. The problem is, the block is affecting other things like Google Web Preview, etc. Also, is there a reason that a webmaster would want Google to crawl pages for the web preview (or not want) as it appears that the bot also doesn't necessarily honor robots.txt (as reported in other posts)
What else comes from 75.125.X.X Why would projects such as Google DOTS that have potential to cause these sort off issues be permitted to run from IP addresses where Google App Engine traffic and other Google Spider traffic originate?
As it stands right now, 75.125 is completely blocked because this thing is hitting from IP's all across the range. The problem is, the block is affecting other things like Google Web Preview, etc. Also, is there a reason that a webmaster would want Google to crawl pages for the web preview (or not want) as it appears that the bot also doesn't necessarily honor robots.txt (as reported in other posts)
What else comes from 75.125.X.X Why would projects such as Google DOTS that have potential to cause these sort off issues be permitted to run from IP addresses where Google App Engine traffic and other Google Spider traffic originate?
Brandon Wirtz
Aug 8, 2011, 9:13:46 PM8/8/11
to google-a...@googlegroups.com
I don’t have solution other than to block requests by agent.
And I really only provided this information so that someone at GOOG could go track him down and beat him (again).
From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of Viper
Sent: Monday, August 08, 2011 2:28 PM
To: google-a...@googlegroups.com
Cc: carolinas...@gmail.com
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/zQ0_X02n0BEJ.
Chris Wage
Aug 9, 2011, 3:14:38 AM8/9/11
to google-a...@googlegroups.com
I have been seeing this crap too.. around 1300 hits (~7MB of traffic) to the same URL so far today I've just blocked it via user-agent in .htaccess for now..
Sami Shalabi
Aug 9, 2011, 2:49:24 PM8/9/11
to google-a...@googlegroups.com
Hi Folks
Sorry about this. We are aware of the issue and have turning the service off until we have a fix out.
Best,
Sami
Sami Shalabi
Aug 9, 2011, 2:52:55 PM8/9/11
to google-a...@googlegroups.com
Hi Everyone:
First, we are sorry about this.
We are aware of the problem. The service has been turned off and we are working on a fix right now.
Best,
Sami
Viper
Aug 26, 2011, 12:52:25 PM8/26/11
to Google App Engine
Sami,
The service has not been turned off with the exception of only 'the 1
day" after which we made the original post here. Since then, the DOTS
spider/bot is hitting sites in full force. Fortunately we are
blocking it but others might not be able to do so.
Even though we are blocking it, it still tries over and over and over
and over and over and over again - non-stop.
What is it that you are trying to do with this project and why so many
problems/bad behavior of the bot?
The service has not been turned off with the exception of only 'the 1
day" after which we made the original post here. Since then, the DOTS
spider/bot is hitting sites in full force. Fortunately we are
blocking it but others might not be able to do so.
Even though we are blocking it, it still tries over and over and over
and over and over and over again - non-stop.
What is it that you are trying to do with this project and why so many
problems/bad behavior of the bot?
Viper
Aug 26, 2011, 12:53:35 PM8/26/11
to Google App Engine
Sami,
The service has not been turned off with the exception of only 'the 1
day" after which we made the original post here. Since then, the DOTS
spider/bot is hitting sites in full force. Fortunately we are
blocking it but others might not be able to do so.
Even though we are blocking it, it still tries over and over and over
and over and over and over again - non-stop.
What is it that you are trying to do with this project and why so many
problems?
The service has not been turned off with the exception of only 'the 1
day" after which we made the original post here. Since then, the DOTS
spider/bot is hitting sites in full force. Fortunately we are
blocking it but others might not be able to do so.
Even though we are blocking it, it still tries over and over and over
and over and over and over again - non-stop.
What is it that you are trying to do with this project and why so many
Viper
Aug 26, 2011, 12:53:48 PM8/26/11
to Google App Engine
Sami,
The service has not been turned off with the exception of only 'the 1
day" after which we made the original post here. Since then, the DOTS
spider/bot is hitting sites in full force. Fortunately we are
blocking it but others might not be able to do so.
Even though we are blocking it, it still tries over and over and over
and over and over and over again - non-stop.
What is it that you are trying to do with this project and why so many
problems?
On Aug 8, 1:57 am, Viper <carolinasviperc...@gmail.com> wrote:
The service has not been turned off with the exception of only 'the 1
day" after which we made the original post here. Since then, the DOTS
spider/bot is hitting sites in full force. Fortunately we are
blocking it but others might not be able to do so.
Even though we are blocking it, it still tries over and over and over
and over and over and over again - non-stop.
What is it that you are trying to do with this project and why so many
problems?
> Apparently this has been reported in several other places on Googlehttp://www.google.com/support/forum/p/Webmasters/thread?tid=6dd38e353...http://www.google.com/support/forum/p/Google+Apps/thread?tid=0a4d894e...http://www.google.com/support/forum/p/Webmasters/thread?tid=1ef5e9959...
Viper
Aug 31, 2011, 6:12:19 PM8/31/11
to Google App Engine
I wanted to post a followup for those following this thread
Well as I mentioned before, we moved away from blocking entire IP
blocks to just blocking user agents like DOTS which was effective
until today. Now it seems that from the IP's that DOTs is
originating, this project (or another) has resorted to completely
stripping the user agent in an attempt to bypass these blocks. I fail
to understand why this is occurring and why does Google tolerate
activity such as what is reported in the links above and as I have
described here? We have seen this activity thus far on
74.125.52.*
and
74.125.126.*
Now due to this, we have to resort back to IP blocking.......
Taking this along with the other issues, I guess it just makes things
a bit frustrating due to the fact there is a bit of a lack of
transparency as to what exactly is going on and why the cloak and
dagger with these bots/projects/etc from Google IP addresses.
> To view this discussion on the web visithttps://groups.google.com/d/msg/google-appengine/-/zQ0_X02n0BEJ.
Well as I mentioned before, we moved away from blocking entire IP
blocks to just blocking user agents like DOTS which was effective
until today. Now it seems that from the IP's that DOTs is
originating, this project (or another) has resorted to completely
stripping the user agent in an attempt to bypass these blocks. I fail
to understand why this is occurring and why does Google tolerate
activity such as what is reported in the links above and as I have
described here? We have seen this activity thus far on
74.125.52.*
and
74.125.126.*
Now due to this, we have to resort back to IP blocking.......
Taking this along with the other issues, I guess it just makes things
a bit frustrating due to the fact there is a bit of a lack of
transparency as to what exactly is going on and why the cloak and
dagger with these bots/projects/etc from Google IP addresses.
Viper
Aug 31, 2011, 6:12:53 PM8/31/11
to Google App Engine
I wanted to post a followup for those following this thread
Well as I mentioned before, we moved away from blocking entire IP
blocks to just blocking user agents like DOTS which was effective
until today. Now it seems that from the IP's that DOTs is
originating, this project (or another) has resorted to completely
stripping the user agent in an attempt to bypass these blocks. I fail
to understand why this is occurring and why does Google tolerate
activity such as what is reported in the links above and as I have
described here? We have seen this activity thus far on
74.125.52.*
and
74.125.126.*
Now due to this, we have to resort back to IP blocking.......
Taking this along with the other issues, I guess it just makes things
a bit frustrating due to the fact there is a bit of a lack of
transparency as to what exactly is going on and why the cloak and
dagger with these bots/projects/etc from Google IP addresses.
On Aug 8, 9:13 pm, "Brandon Wirtz" <drak...@digerat.com> wrote:
Well as I mentioned before, we moved away from blocking entire IP
blocks to just blocking user agents like DOTS which was effective
until today. Now it seems that from the IP's that DOTs is
originating, this project (or another) has resorted to completely
stripping the user agent in an attempt to bypass these blocks. I fail
to understand why this is occurring and why does Google tolerate
activity such as what is reported in the links above and as I have
described here? We have seen this activity thus far on
74.125.52.*
and
74.125.126.*
Now due to this, we have to resort back to IP blocking.......
Taking this along with the other issues, I guess it just makes things
a bit frustrating due to the fact there is a bit of a lack of
transparency as to what exactly is going on and why the cloak and
dagger with these bots/projects/etc from Google IP addresses.
On Aug 8, 9:13 pm, "Brandon Wirtz" <drak...@digerat.com> wrote:
Nick Johnson
Sep 2, 2011, 1:20:56 AM9/2/11
to google-a...@googlegroups.com
Hi Sami,
--
Nick Johnson, Developer Programs Engineer, App Engine
Have you seen the latest report regarding a bot from our IP range with no user agent at all crawling sites? Do you have any idea who/what this might be?
-Nick
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/CTlpt-mFwF4J.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
Nick Johnson, Developer Programs Engineer, App Engine
Reply all
Reply to author
Forward
0 new messages