It's come to our attention that some URLs are listed as 404s for some
sites in Webmaster Tools even though they were apparently crawled
correctly. In general, even if we were not able to crawl some URLs
correctly once or twice, this should not affect a site's crawling,
indexing or ranking in our search engine.
We're currently analyzing the situation and will give you more
information as soon as we have it.
Thanks for using Webmaster Tools and thank you for your ongoing
feedback!
> It's come to our attention that some URLs are listed as 404s for some
> sites in Webmaster Tools even though they were apparently crawled
> correctly. In general, even if we were not able to crawl some URLs
> correctly once or twice, this should not affect a site's crawling,
> indexing or ranking in our search engine.
> We're currently analyzing the situation and will give you more
> information as soon as we have it.
> Thanks for using Webmaster Tools and thank you for your ongoing
> feedback!
Thanks again, John.
If it will help in tracking the problem down, I'll be glad to share as
many of my daily server logs as needed, along with my .htaccess file
and anything else you folks think would be helpful in nailing it.
> It's come to our attention that some URLs are listed as 404s for some
> sites in Webmaster Tools even though they were apparently crawled
> correctly. In general, even if we were not able to crawl some URLs
> correctly once or twice, this should not affect a site's crawling,
> indexing or ranking in our search engine.
> We're currently analyzing the situation and will give you more
> information as soon as we have it.
> Thanks for using Webmaster Tools and thank you for your ongoing
> feedback!
Recently (last two months) I have had many 404 (Not found) URL errors
in webmaster tools even though almost all of the URL's resolve to
valid pages when viewed in a browser.
I have been using webmaster tools for over a year, however this has
only been occuring in the last two months. It started after a short
period of downtime but since that downtime there has been no other
server issues yet new 404 errors keep appearing in webmaster tools.
At the moment I have 65 of these errors, even though almost all of
these pages are displaying fine in a browser, and there has been no
recent server downtime either.
After investigating apache logs there were no 404 errors in my apache
access logs for those reported errors in sitemaps. take at these two
examples errors from webmaster tools.
> It's come to our attention that some URLs are listed as 404s for some
> sites in Webmaster Tools even though they were apparently crawled
> correctly. In general, even if we were not able to crawl some URLs
> correctly once or twice, this should not affect a site's crawling,
> indexing or ranking in our search engine.
> We're currently analyzing the situation and will give you more
> information as soon as we have it.
> Thanks for using Webmaster Tools and thank you for your ongoing
> feedback!
I am having a similar problem. I just moved my site from an all HTML
site to a Joomla CMS/PHP site. I have 301 redirects installed for
about 80% of my pages (loaded into access.conf) so I do have a lot of
traffic still coming. However, in webtools when I review the 404
files, many of the files of my new site are listed and these URL's
clearly work and are not 404. Webtools even lists my domain
www.fabulousfoods.com as 404. My site is roughly 4000 pages deep.
I don't get it.
Here is a list of some good URL"s webtools lists as 404
There are at least 1000 more in webtools 404 list that are valid.
Also, it does list my old URL's as 404. This I understand as they are
404 because they were moved. However, many of these that are showing
404 are in my 301 redirect list so they actually are not 404.
What can I do to fix this so I can find the actual 404's?
> Recently (last two months) I have had many 404 (Not found) URL errors
> in webmaster tools even though almost all of the URL's resolve to
> valid pages when viewed in a browser.
> I have been using webmaster tools for over a year, however this has
> only been occuring in the last two months. It started after a short
> period of downtime but since that downtime there has been no other
> server issues yet new 404 errors keep appearing in webmaster tools.
> At the moment I have 65 of these errors, even though almost all of
> these pages are displaying fine in a browser, and there has been no
> recent server downtime either.
> After investigating apache logs there were no 404 errors in my apache
> access logs for those reported errors in sitemaps. take at these two
> examples errors from webmaster tools.
> As you can see no 404 errors for those two pages on that date.
> Keep us updated on what google is doing about this issue as we are
> worried that it could effect our rankings.
> Thanks
> On Jul 30, 4:13 pm, JohnMu wrote:
> > Hi everyone!
> > It's come to our attention that some URLs are listed as 404s for some
> > sites in Webmaster Tools even though they were apparently crawled
> > correctly. In general, even if we were not able to crawl some URLs
> > correctly once or twice, this should not affect a site's crawling,
> > indexing or ranking in our search engine.
> > We're currently analyzing the situation and will give you more
> > information as soon as we have it.
> > Thanks for using Webmaster Tools and thank you for your ongoing
> > feedback!
> I am having a similar problem. I just moved my site from an all HTML
> site to a Joomla CMS/PHP site. I have 301 redirects installed for
> about 80% of my pages (loaded into access.conf) so I do have a lot of
> traffic still coming. However, in webtools when I review the 404
> files, many of the files of my new site are listed and these URL's
> clearly work and are not 404. Webtools even lists my domainwww.fabulousfoods.comas 404. My site is roughly 4000 pages deep.
> I don't get it.
> Here is a list of some good URL"s webtools lists as 404
> There are at least 1000 more in webtools 404 list that are valid.
> Also, it does list my old URL's as 404. This I understand as they are
> 404 because they were moved. However, many of these that are showing
> 404 are in my 301 redirect list so they actually are not 404.
> What can I do to fix this so I can find the actual 404's?
> Thanks, Mitch
> On Jul 31, 3:38 am, taxwebmaster wrote:
> > Thanks JohnMu,
> > Recently (last two months) I have had many 404 (Not found) URL errors
> > in webmaster tools even though almost all of the URL's resolve to
> > valid pages when viewed in a browser.
> > I have been using webmaster tools for over a year, however this has
> > only been occuring in the last two months. It started after a short
> > period of downtime but since that downtime there has been no other
> > server issues yet new 404 errors keep appearing in webmaster tools.
> > At the moment I have 65 of these errors, even though almost all of
> > these pages are displaying fine in a browser, and there has been no
> > recent server downtime either.
> > After investigating apache logs there were no 404 errors in my apache
> > access logs for those reported errors in sitemaps. take at these two
> > examples errors from webmaster tools.
> > As you can see no 404 errors for those two pages on that date.
> > Keep us updated on what google is doing about this issue as we are
> > worried that it could effect our rankings.
> > Thanks
> > On Jul 30, 4:13 pm, JohnMu wrote:
> > > Hi everyone!
> > > It's come to our attention that some URLs are listed as 404s for some
> > > sites in Webmaster Tools even though they were apparently crawled
> > > correctly. In general, even if we were not able to crawl some URLs
> > > correctly once or twice, this should not affect a site's crawling,
> > > indexing or ranking in our search engine.
> > > We're currently analyzing the situation and will give you more
> > > information as soon as we have it.
> > > Thanks for using Webmaster Tools and thank you for your ongoing
> > > feedback!
Please excuse my stupidity! I'm new here and don't understand about
90% of what is going on. Recently (since approx. 1 month ago?), I have
been getting LOTS of strange Google-based "signals" that something is
wrong. Like when I launch IE7, my sites fail with 404 errors about 30%
of the time. I see Google messages when this happens: "...your query
looks similar to automated requests from a computer virus or spyware
application..." Previously, I have never seen these kinda things.
Today I did a Google search and it returned one of my sites with a
disturbing annotation: "This site may harm your computer." as a link
to http://www.google.com/support/bin/answer.py? answer=45449&topic=360&hl=en&sa=X&oi=malwarewarninglink&resnum=6&ct=help.
This page says: "We're sorry, but the information you've requested
cannot be found." Duh!
Do your lawyers think such unwarranted messages are liability
exposures for your company? ;-)
OK, I figured out that I needed to use your "webmaster tools" and
tried both ways to get my site to "Verify". Nada. Apparently w/o being
verified, I can't submit a sitemap?
What the @#$%? I used to like Google, but this leaves a bitter tase in
my mouth...
Can anyone please tell a simple man what he's supposed to do??? In a
simple and comprehendable way???
> It's come to our attention that some URLs are listed as 404s for some
> sites in Webmaster Tools even though they were apparently crawled
> correctly. In general, even if we were not able to crawl some URLs
> correctly once or twice, this should not affect a site's crawling,
> indexing or ranking in our search engine.
> We're currently analyzing the situation and will give you more
> information as soon as we have it.
> Thanks for using Webmaster Tools and thank you for your ongoing
> feedback!
> Please excuse my stupidity! I'm new here and don't understand about
> 90% of what is going on. Recently (since approx. 1 month ago?), I have
> been getting LOTS of strange Google-based "signals" that something is
> wrong. Like when I launch IE7, my sites fail with 404 errors about 30%
> of the time. I see Google messages when this happens: "...your query
> looks similar to automated requests from a computer virus or spyware
> application..." Previously, I have never seen these kinda things.
> Today I did a Google search and it returned one of my sites with a
> disturbing annotation: "This site may harm your computer." as a link
> tohttp://www.google.com/support/bin/answer.py? > answer=45449&topic=360&hl=en&sa=X&oi=malwarewarninglink&resnum=6&ct=help.
> This page says: "We're sorry, but the information you've requested
> cannot be found." Duh!
> Do your lawyers think such unwarranted messages are liability
> exposures for your company? ;-)
> OK, I figured out that I needed to use your "webmaster tools" and
> tried both ways to get my site to "Verify". Nada. Apparently w/o being
> verified, I can't submit a sitemap?
> What the @#$%? I used to like Google, but this leaves a bitter tase in
> my mouth...
> Can anyone please tell a simple man what he's supposed to do??? In a
> simple and comprehendable way???
> TIA
> On Jul 30, 10:13 am, JohnMu wrote:
> > Hi everyone!
> > It's come to our attention that some URLs are listed as 404s for some
> > sites in Webmaster Tools even though they were apparently crawled
> > correctly. In general, even if we were not able to crawl some URLs
> > correctly once or twice, this should not affect a site's crawling,
> > indexing or ranking in our search engine.
> > We're currently analyzing the situation and will give you more
> > information as soon as we have it.
> > Thanks for using Webmaster Tools and thank you for your ongoing
> > feedback!
We've had the 404 error show up for our home page since July 23,
2008. I was a bit confused as it showed up as being crawled ok, yet
this error still appears.
> It's come to our attention that some URLs are listed as 404s for some
> sites in Webmaster Tools even though they were apparently crawled
> correctly. In general, even if we were not able to crawl some URLs
> correctly once or twice, this should not affect a site's crawling,
> indexing or ranking in our search engine.
> We're currently analyzing the situation and will give you more
> information as soon as we have it.
> Thanks for using Webmaster Tools and thank you for your ongoing
> feedback!
John wrote "even if we were not able to crawl some URLs
correctly once or twice, this should not affect a site's crawling,
indexing or ranking in our search engine"
Google WMT has shown 404 response to some 300+ urls on our site www.proz.com and has our home page 404 on multiple occasions (more than once or
twice) and our pages are dropping out of the index and pages are being
delivered in the serps lower and lower (which makes perfect sense - if
Googlebot thinks we are not accessible and 404 why would Google
deliver those urls to searchers - that would entail a bad user
experience for the web searcher). BUT if this 404 issue is related to
Googles mobile crawler i wish that someone would post something
concrete. Googlebot activity to our home page has also slowed down
(by more than 50%) - apparently Googlebot thinks we are 404 so often
that he/she no longer wants to even come back as frequently as they
used to.
Has anyone else with these problems tried blocking Googlebot mobile?
Has anyone done anything that has helpped? Our site's natural traffic
is plummeting with no end in site, no solution, no answers and nothing
we can find on our end that would indicate a 404 delivery of our home
page - much less repeatedly over the last few months.
My site is doing the same thing. I just moved from HTML to PHP and
the crawl shows over 9000 404 pages (up from 6000 a few days ago). I
can understand why some of the old HTML pages are 404. I downloaded
the 404 csv and noticed that all my new PHP pages are 404 also. The
bulk of the 404 are old search's that no longer exist.
Here's a sampleing of 10 URL's from the google list (there are 1000's
more like it). Click any of them, I dare you. You'll see a page and
not a 404 error. What's up with google showing 404 when the page is
200?
> John wrote "even if we were not able to crawl some URLs
> correctly once or twice, this should not affect a site's crawling,
> indexing or ranking in our search engine"
> Google WMT has shown 404 response to some 300+ urls on our sitewww.proz.com > and has our home page 404 on multiple occasions (more than once or
> twice) and our pages are dropping out of the index and pages are being
> delivered in the serps lower and lower (which makes perfect sense - if
> Googlebot thinks we are not accessible and 404 why would Google
> deliver those urls to searchers - that would entail a bad user
> experience for the web searcher). BUT if this 404 issue is related to
> Googles mobile crawler i wish that someone would post something
> concrete. Googlebot activity to our home page has also slowed down
> (by more than 50%) - apparently Googlebot thinks we are 404 so often
> that he/she no longer wants to even come back as frequently as they
> used to.
> Has anyone else with these problems tried blocking Googlebot mobile?
> Has anyone done anything that has helpped? Our site's natural traffic
> is plummeting with no end in site, no solution, no answers and nothing
> we can find on our end that would indicate a 404 delivery of our home
> page - much less repeatedly over the last few months.
> DrewZ
> On Aug 4, 9:02 am, Phil Payne wrote:
> > > We're currently analyzing the situation and will give you more
> > > information as soon as we have it.
> > I haven't seen one of these for a couple of days now, John.
> > Did someone fix something?
> > (Or did someone you were talking to turn bright red and rush out of
> > the room?)
I blocked Googlebot from indexing WML pages and blocked Googlebot-
Mobile from indexing HTML pages a few weeks ago. The bogus 404 errors
for existing HTML pages went away (hundreds of them) and have not
returned.
On the other hand, Googlebot-Mobile is still reporting bogus 404
errors for existing WML pages but I can live with that for now.
Oh, it also appears that Google fixed that WML POST bug I reported
weeks ago.
> John wrote "even if we were not able to crawl some URLs
> correctly once or twice, this should not affect a site's crawling,
> indexing or ranking in our search engine"
> Google WMT has shown 404 response to some 300+ urls on our sitewww.proz.com > and has our home page 404 on multiple occasions (more than once or
> twice) and our pages are dropping out of the index and pages are being
> delivered in the serps lower and lower (which makes perfect sense - if
> Googlebot thinks we are not accessible and 404 why would Google
> deliver those urls to searchers - that would entail a bad user
> experience for the web searcher). BUT if this 404 issue is related to
> Googles mobile crawler i wish that someone would post something
> concrete. Googlebot activity to our home page has also slowed down
> (by more than 50%) - apparently Googlebot thinks we are 404 so often
> that he/she no longer wants to even come back as frequently as they
> used to.
> Has anyone else with these problems tried blocking Googlebot mobile?
> Has anyone done anything that has helpped? Our site's natural traffic
> is plummeting with no end in site, no solution, no answers and nothing
> we can find on our end that would indicate a 404 delivery of our home
> page - much less repeatedly over the last few months.
> DrewZ
> On Aug 4, 9:02 am, Phil Payne wrote:
> > > We're currently analyzing the situation and will give you more
> > > information as soon as we have it.
> > I haven't seen one of these for a couple of days now, John.
> > Did someone fix something?
> > (Or did someone you were talking to turn bright red and rush out of
> > the room?)- Hide quoted text -
I think this is the message many of us have been looking for. You have
confirmed my suspicion that the major, if not the only, "culprit" in
this is Googlebot-Mobile.
As to John Mu's statement that this should not impact indexing, I do
know that the number of "404's" reported in my Sitemap URL Errors
report has dropped from about 355 to 91, but that the number of
indexed links has dropped from 643 to 576. It has done this before,
and has corrected itself after a while, but it is unnerving. (Of
course, as I have noted several times before, the reported 404's are
not actual 404's.)
Anyway, your confirmation regarding Googlebot-Mobile should be helpful
in getting this resolved. I am tempted also to block Googlebot-Mobile,
but also afraid that it might have unintended consequences, so I will
leave things as they are for now.
> I blocked Googlebot from indexing WML pages and blocked Googlebot-
> Mobile from indexing HTML pages a few weeks ago. The bogus 404 errors
> for existing HTML pages went away (hundreds of them) and have not
> returned.
> On the other hand, Googlebot-Mobile is still reporting bogus 404
> errors for existing WML pages but I can live with that for now.
> Oh, it also appears that Google fixed that WML POST bug I reported
> weeks ago.
> On Aug 4, 3:24 pm, DrewZ wrote:
> > John wrote "even if we were not able to crawl some URLs
> > correctly once or twice, this should not affect a site's crawling,
> > indexing or ranking in our search engine"
> > Google WMT has shown 404 response to some 300+ urls on our sitewww.proz.com > > and has our home page 404 on multiple occasions (more than once or
> > twice) and our pages are dropping out of the index and pages are being
> > delivered in the serps lower and lower (which makes perfect sense - if
> > Googlebot thinks we are not accessible and 404 why would Google
> > deliver those urls to searchers - that would entail a bad user
> > experience for the web searcher). BUT if this 404 issue is related to
> > Googles mobile crawler i wish that someone would post something
> > concrete. Googlebot activity to our home page has also slowed down
> > (by more than 50%) - apparently Googlebot thinks we are 404 so often
> > that he/she no longer wants to even come back as frequently as they
> > used to.
> > Has anyone else with these problems tried blocking Googlebot mobile?
> > Has anyone done anything that has helpped? Our site's natural traffic
> > is plummeting with no end in site, no solution, no answers and nothing
> > we can find on our end that would indicate a 404 delivery of our home
> > page - much less repeatedly over the last few months.
> > DrewZ
> > On Aug 4, 9:02 am, Phil Payne wrote:
> > > > We're currently analyzing the situation and will give you more
> > > > information as soon as we have it.
> > > I haven't seen one of these for a couple of days now, John.
> > > Did someone fix something?
> > > (Or did someone you were talking to turn bright red and rush out of
> > > the room?)- Hide quoted text -
DrewZ wrote:
> Google WMT has shown 404 response to some 300+ urls on our site www.proz.com > and has our home page 404 on multiple occasions ...
If the postulation is correct and Google is actually reporting as 404s
problems that it cannot otherwise describe, I suggest your DNS
structure is very likely to be the cause:
ERROR: One or more of your nameservers did not respond:
The ones that did not responded are:
216.246.59.66 216.218.254.148 72.52.2.1 66.225.199.10
I'd fix that before whingeing at Google - it's much more likely to be
the real cause.
Yes, if the pages are erroneously labeled as 404 errors and they have
little or no PR they will fall out of the index. This is why many
webmasters reported that their home pages were labeled as 404's in
GWT yet the home pages continued to be listed in the index.
I also believe Googlebot-Mobile to be the culprit. If it encounters a
301 it confuses it (for whatever reason) for a 404. It will stay a 404
until the original Googlebot comes back around and re-crawls the page.
> I think this is the message many of us have been looking for. You have
> confirmed my suspicion that the major, if not the only, "culprit" in
> this is Googlebot-Mobile.
> As to John Mu's statement that this should not impact indexing, I do
> know that the number of "404's" reported in my Sitemap URL Errors
> report has dropped from about 355 to 91, but that the number of
> indexed links has dropped from 643 to 576. It has done this before,
> and has corrected itself after a while, but it is unnerving. (Of
> course, as I have noted several times before, the reported 404's are
> not actual 404's.)
> Anyway, your confirmation regarding Googlebot-Mobile should be helpful
> in getting this resolved. I am tempted also to block Googlebot-Mobile,
> but also afraid that it might have unintended consequences, so I will
> leave things as they are for now.
> On Aug 4, 7:26 pm, Key_Master wrote:
> > Hi Drewz,
> > I blocked Googlebot from indexing WML pages and blocked Googlebot-
> > Mobile from indexing HTML pages a few weeks ago. The bogus 404 errors
> > for existing HTML pages went away (hundreds of them) and have not
> > returned.
> > On the other hand, Googlebot-Mobile is still reporting bogus 404
> > errors for existing WML pages but I can live with that for now.
> > Oh, it also appears that Google fixed that WML POST bug I reported
> > weeks ago.
> > On Aug 4, 3:24 pm, DrewZ wrote:
> > > John wrote "even if we were not able to crawl some URLs
> > > correctly once or twice, this should not affect a site's crawling,
> > > indexing or ranking in our search engine"
> > > Google WMT has shown 404 response to some 300+ urls on our sitewww.proz.com > > > and has our home page 404 on multiple occasions (more than once or
> > > twice) and our pages are dropping out of the index and pages are being
> > > delivered in the serps lower and lower (which makes perfect sense - if
> > > Googlebot thinks we are not accessible and 404 why would Google
> > > deliver those urls to searchers - that would entail a bad user
> > > experience for the web searcher). BUT if this 404 issue is related to
> > > Googles mobile crawler i wish that someone would post something
> > > concrete. Googlebot activity to our home page has also slowed down
> > > (by more than 50%) - apparently Googlebot thinks we are 404 so often
> > > that he/she no longer wants to even come back as frequently as they
> > > used to.
> > > Has anyone else with these problems tried blocking Googlebot mobile?
> > > Has anyone done anything that has helpped? Our site's natural traffic
> > > is plummeting with no end in site, no solution, no answers and nothing
> > > we can find on our end that would indicate a 404 delivery of our home
> > > page - much less repeatedly over the last few months.
> > > DrewZ
> > > On Aug 4, 9:02 am, Phil Payne wrote:
> > > > > We're currently analyzing the situation and will give you more
> > > > > information as soon as we have it.
> > > > I haven't seen one of these for a couple of days now, John.
> > > > Did someone fix something?
> > > > (Or did someone you were talking to turn bright red and rush out of
> > > > the room?)- Hide quoted text -
Despite of the fact that google claims that it should not affect a
site's crawling,
indexing or ranking in our search engine I can see that my site is
getting lower and lower in ranking for more than 200 keywords.
That's a huge traffic. I haven't tried blocking google mobile yet but
I think I will.
> John wrote "even if we were not able to crawl some URLs
> correctly once or twice, this should not affect a site's crawling,
> indexing or ranking in our search engine"
> Google WMT has shown 404 response to some 300+ urls on our sitewww.proz.com > and has our home page 404 on multiple occasions (more than once or
> twice) and our pages are dropping out of the index and pages are being
> delivered in the serps lower and lower (which makes perfect sense - if
> Googlebot thinks we are not accessible and 404 why would Google
> deliver those urls to searchers - that would entail a bad user
> experience for the web searcher). BUT if this 404 issue is related to
> Googles mobile crawler i wish that someone would post something
> concrete. Googlebot activity to our home page has also slowed down
> (by more than 50%) - apparently Googlebot thinks we are 404 so often
> that he/she no longer wants to even come back as frequently as they
> used to.
> Has anyone else with these problems tried blocking Googlebot mobile?
> Has anyone done anything that has helpped? Our site's natural traffic
> is plummeting with no end in site, no solution, no answers and nothing
> we can find on our end that would indicate a 404 delivery of our home
> page - much less repeatedly over the last few months.
> DrewZ
> On Aug 4, 9:02 am, Phil Payne wrote:
> > > We're currently analyzing the situation and will give you more
> > > information as soon as we have it.
> > I haven't seen one of these for a couple of days now, John.
> > Did someone fix something?
> > (Or did someone you were talking to turn bright red and rush out of
> > the room?)
Can anyone suggest what I should do? Please remember that I am a
simple man who doesn't talk the way you do...
For example, in this "thread" I find advice like:
"I blocked Googlebot from indexing WML pages and blocked Googlebot-
Mobile from indexing HTML pages..."
Which means nothing to me. Does this have anything to do with my
questions?
I don't seem to get the 404 errors anymore (last 3 days), but I still
have no clue why Google reported that my site was, "...your query
looks similar to automated requests from a computer virus or spyware
application..."
I still have no idea how to get my site(s) to "verify", what ever that
is? Is verification by Google necessary? Why? What IS a "sitemap"? Why
do I need one? I assume it's so big that if I had one I would have to
put it on the floor and "crawl" over it? Or does "crawl" mean
something else? Is "crawling" painful? Do I really NEED to use Google
Webmaster Tools? Who ARE you people? ;-) Do I need to get my lawyer on
this (seems like today nothing can be "fixed" w/o a lawyer being
involved). :-( Will Barack or John be most likely to help me solve
this? If NM and AZ refuse to build new electrical transmission lines
for T. Boone's wind generators, will you run out of juice to power
your server farms? How soon will this happen?
BTW, I have been reading your help files and FAQs until my eyes and
brain hurt.
Sorry for being so simple...I hope you sorta enjoyed my humor too. The
bottom line is that until recently, I was able to manage my simple
websites w/o even knowing about all this backroom voodoo. I used to be
a Google fan. After this week, I have to rank Google right down there
with Unka Bills evil minions...I'm depressed that it turned out that
way.
> Please excuse my stupidity! I'm new here and don't understand about
> 90% of what is going on. Recently (since approx. 1 month ago?), I have
> been getting LOTS of strange Google-based "signals" that something is
> wrong. Like when I launch IE7, my sites fail with 404 errors about 30%
> of the time. I see Google messages when this happens: "...your query
> looks similar to automated requests from a computer virus or spyware
> application..." Previously, I have never seen these kinda things.
> Today I did a Google search and it returned one of my sites with a
> disturbing annotation: "This site may harm your computer." as a link
> tohttp://www.google.com/support/bin/answer.py? > answer=45449&topic=360&hl=en&sa=X&oi=malwarewarninglink&resnum=6&ct=help.
> This page says: "We're sorry, but the information you've requested
> cannot be found." Duh!
> Do your lawyers think such unwarranted messages are liability
> exposures for your company? ;-)
> OK, I figured out that I needed to use your "webmaster tools" and
> tried both ways to get my site to "Verify". Nada. Apparently w/o being
> verified, I can't submit a sitemap?
> What the @#$%? I used to like Google, but this leaves a bitter tase in
> my mouth...
> Can anyone please tell a simple man what he's supposed to do??? In a
> simple and comprehendable way???
> TIA
> On Jul 30, 10:13 am, JohnMu wrote:
> > Hi everyone!
> > It's come to our attention that some URLs are listed as 404s for some
> > sites in Webmaster Tools even though they were apparently crawled
> > correctly. In general, even if we were not able to crawl some URLs
> > correctly once or twice, this should not affect a site's crawling,
> > indexing or ranking in our search engine.
> > We're currently analyzing the situation and will give you more
> > information as soon as we have it.
> > Thanks for using Webmaster Tools and thank you for your ongoing
> > feedback!
I am having the same kind of headache myself. I committed my .htaccess
file to a 301-redirect. That is, I removed the www prefix from my
domain name. All of a sudden, my Webmaster Tools account shows just 1
indexed page. Before the canonization of my domain, Webmaster Tools
reported over 700 indexed pages in the sitemap I had submitted. Before
the .htaccess edit, my website had 0 (zero) 404 errors. Now, the
report shows 99 pages with the 404 HTML error! How come? My sitemap
has only 1 page indexed! There are also reports of other strange
errors, such as URLs with duplicate titles. Look at this pair of
duplicate titles:
That type of error should never come out (except, perhaps, for kids
creating their own search engines!) Every folder must have an index
file to prevent unauthorized access (e.g. listing) to all resources in
a directory.
I wrote in several newsgroups and forums about this serious problem,
but nobody had a clue. There was no explanation. Now, I found out that
not even Google (or other search engines) have a clue!
I think I found the clue (if not the answer). I got the same type of
errors with my web hosting company. They have many features, including
site analysis. The reports offered by my webhost were shocking: Every
page at my site had broken links! But absolutely all errors were
bogus! I clicked on any of the “faulty” URLs and the browser open the
corresponding web pages without any problem!
I reported the problem several times to my web host. I always got the
stupidest of the responses. The respondents acted simply like idiots!
I realized the customer support was outsourced! The guys at the tech
support were NOT in the U.S. I have no problem with that. Except that
all tech support personnel must be well trained in the language they
are working with. Better still, a U.S. company, for example, should
train the outsourced tech personnel in the U.S. for a period of time.
Worse, the tech support hadn’t even localized their keyboards for
U.S. English! They were dealing with a Webmaster in the U.S., but
their keyboard was on their locale! I checked the responses from the
outside locale, and had the answer to my puzzle. The URLs in the
outsourced responses (with 404 HTML errors) were listed with weird
characters! I can’t reproduce them here because my locale is U.S. My
original messages to the outsourced tech support listed the URLs
correctly, with no strange characters.
That’s why I summarized my answer in one word: GLOBALIZATION!
Google and all major search engines spread their businesses all over
the world. Call it globalization or call it exaggerated outsourcing.
Problem is, the languages are very different and the keyboards act
very differently. WC3 must intervene decisively and set a unified
standard. We all must use the same set of characters that behaves
exactly the same in any locale, regardless of keyboard settings. That
old:
<!doctype html public "-//w3c//dtd html 3.2//en">
or the likes no longer work properly. Those weirdo characters like @@%
%20^& or so in URLs cause tons of headaches around the world.
Possibly, parts of the outsourced Google algorithms are not set for
the U.S. keyboard. Hence, big headaches for Webmasters in the United
States!
Standards are good — put nationalistic pride aside. The Internet is
not soccer for the hooligans to thrive! The Internet worked better at
the beginning of the Internet. There were infinitely fewer web sites.
Just about everybody followed the U.S. models, especially the
language. I don’t care if they set the Internet models at the United
Nations (in Geneva, or in Beijing, or in Moscow). But have one clear
standard. But, without a doubt, the American English has the advantage
of simplicity. Let’s be honest. Your alphabets are cumbersome computer-
wise, Webmasters in China, or Japan, or Russia, or Arabs, or Jews.
American English uses a reduced set of the Latin alphabet. Latin is
the most precise language in human history. Contemporary law and
science can’t do without terminology in Latin.
Ion Saliu,
International Webmaster At-Large
(Webmasterus Internationalus Sine Restrictionae)
http://saliu.com (formerly a well indexed www.saliu.com)
> It's come to our attention that some URLs are listed as 404s for some
> sites in Webmaster Tools even though they were apparently crawled
> correctly. In general, even if we were not able to crawl some URLs
> correctly once or twice, this should not affect a site's crawling,
> indexing or ranking in our search engine.
> We're currently analyzing the situation and will give you more
> information as soon as we have it.
> Thanks for using Webmaster Tools and thank you for your ongoing
> feedback!
Chuxter, it sounds like your computer and the server you host your
sites on are infected with malware of some type. Since there really
isn't a simple way to explain these issues to you and how to resolve
them, I suggest you hire a computer tech to help you to resolve the
security related problems and to help prevent them from happening in
the future. They are serious issues.
In the meantime, I suggest you study up on HTML, CSS, Apache HTTP
Server Documentation, .htaccess configuration, W3C document standards
and validation, and web accessibility. You have a lot of work ahead of
you.
Don't take this advice the wrong way, I'm not talking down to you- I'm
just trying to steer you in the right direction. Sitemaps are the
least of your concerns right now.
> Please excuse my stupidity! I'm new here and don't understand about
> 90% of what is going on. Recently (since approx. 1 month ago?), I have
> been getting LOTS of strange Google-based "signals" that something is
> wrong. Like when I launch IE7, my sites fail with 404 errors about 30%
> of the time. I see Google messages when this happens: "...your query
> looks similar to automated requests from a computer virus or spyware
> application..." Previously, I have never seen these kinda things.
> Today I did a Google search and it returned one of my sites with a
> disturbing annotation: "This site may harm your computer." as a link
> tohttp://www.google.com/support/bin/answer.py? > answer=45449&topic=360&hl=en&sa=X&oi=malwarewarninglink&resnum=6&ct=help.
> This page says: "We're sorry, but the information you've requested
> cannot be found." Duh!
> Do your lawyers think such unwarranted messages are liability
> exposures for your company? ;-)
> OK, I figured out that I needed to use your "webmaster tools" and
> tried both ways to get my site to "Verify". Nada. Apparently w/o being
> verified, I can't submit a sitemap?
> What the @#$%? I used to like Google, but this leaves a bitter tase in
> my mouth...
> Can anyone please tell a simple man what he's supposed to do??? In a
> simple and comprehendable way???
> TIA
> On Jul 30, 10:13 am, JohnMu wrote:
> > Hi everyone!
> > It's come to our attention that some URLs are listed as 404s for some
> > sites in Webmaster Tools even though they were apparently crawled
> > correctly. In general, even if we were not able to crawl some URLs
> > correctly once or twice, this should not affect a site's crawling,
> > indexing or ranking in our search engine.
> > We're currently analyzing the situation and will give you more
> > information as soon as we have it.
> > Thanks for using Webmaster Tools and thank you for your ongoing
> > feedback!
Has there been any update on this yet? My site www.bestinthecountry.co.uk has around 150 of my old html pages returning 404 errors in webmaster
tools when the pages are correctly 301 redirected to my new aspx
pages. The pages in question are all performing very badly for
keywords that previously gave top 10 results, in some cases dropping
from #1 to several hundred positions below. I appreciate rankings
naturally fluctuate but this seems to be way beyond natural
fluctuation. The pages are still indexed but are performing very
poorly
Is there any indication when this will be resolved? Please tell me
this is going to be reversible! We have had many years effort in
improving our ranking undone overnight and this is having a severe
effect on our business
> Chuxter, it sounds like your computer and the server you host your
> sites on are infected with malware of some type. Since there really
> isn't a simple way to explain these issues to you and how to resolve
> them, I suggest you hire a computer tech to help you to resolve the
> security related problems and to help prevent them from happening in
> the future. They are serious issues.
> In the meantime, I suggest you study up on HTML, CSS, Apache HTTP
> Server Documentation, .htaccess configuration, W3C document standards
> and validation, and web accessibility. You have a lot of work ahead of
> you.
> Don't take this advice the wrong way, I'm not talking down to you- I'm
> just trying to steer you in the right direction. Sitemaps are the
> least of your concerns right now.
> On Aug 3, 9:52 pm, chuxter wrote:
> > Please excuse my stupidity! I'm new here and don't understand about
> > 90% of what is going on. Recently (since approx. 1 month ago?), I have
> > been getting LOTS of strange Google-based "signals" that something is
> > wrong. Like when I launch IE7, my sites fail with 404 errors about 30%
> > of the time. I see Google messages when this happens: "...your query
> > looks similar to automated requests from a computer virus or spyware
> > application..." Previously, I have never seen these kinda things.
> > Today I did a Google search and it returned one of my sites with a
> > disturbing annotation: "This site may harm your computer." as a link
> > tohttp://www.google.com/support/bin/answer.py? > > answer=45449&topic=360&hl=en&sa=X&oi=malwarewarninglink&resnum=6&ct=help.
> > This page says: "We're sorry, but the information you've requested
> > cannot be found." Duh!
> > Do your lawyers think such unwarranted messages are liability
> > exposures for your company? ;-)
> > OK, I figured out that I needed to use your "webmaster tools" and
> > tried both ways to get my site to "Verify". Nada. Apparently w/o being
> > verified, I can't submit a sitemap?
> > What the @#$%? I used to like Google, but this leaves a bitter tase in
> > my mouth...
> > Can anyone please tell a simple man what he's supposed to do??? In a
> > simple and comprehendable way???
> > TIA
> > On Jul 30, 10:13 am, JohnMu wrote:
> > > Hi everyone!
> > > It's come to our attention that some URLs are listed as 404s for some
> > > sites in Webmaster Tools even though they were apparently crawled
> > > correctly. In general, even if we were not able to crawl some URLs
> > > correctly once or twice, this should not affect a site's crawling,
> > > indexing or ranking in our search engine.
> > > We're currently analyzing the situation and will give you more
> > > information as soon as we have it.
> > > Thanks for using Webmaster Tools and thank you for your ongoing
> > > feedback!
> Has there been any update on this yet? My site www.bestinthecountry.co.uk > has around 150 of my old html pages returning 404 errors in webmaster
> tools when the pages are correctly 301 redirected to my new aspx
> pages.
In this case the site URL is insufficient. Can we have some sample
full URLs and the date that Webmaster Tools last reported a 404 on
each?
That's the most fascinating site I've seen this year.
And I've seen a few.
It needs some tidying up - it doesn't scale perfectly to my 1280x1024
display, and the HTML is not properly terminated. But the approach is
real fun. I don't find the graphic visually that intuitive but it
seemed to get most things right.
I commend it to my colleagues for a good sniff.
I'm not sure what the problem is, but there's an unquantifiable
variable in http://www.intodns.com/1derful.info - see the "i" notes #2
and #4. It's possible something in that loop is causing long enough
delays to trigger Google's 'synthetic' 404s.
> > Has there been any update on this yet? My sitewww.bestinthecountry.co.uk > > has around 150 of my old html pages returning 404 errors in webmaster
> > tools when the pages are correctly 301 redirected to my new aspx
> > pages.
> In this case the site URL is insufficient. Can we have some sample
> full URLs and the date that Webmaster Tools last reported a 404 on
> each?
|
