"We had problems crawling the pages listed here, and as a result they
won't be added to our index and will not appear in search results."
Can someone please explain the above error and why google wont index
my site, url is www.weddingwizard.com.au
The table Errors for urls in sitemaps(0), http errors(0), is showing
no errors.
I have submitted a sitemap and its OK(25 urls), my robots text file is
OK. If I site:www.weddingwizard.com.au my site there are no matching
documents and yet google analytics is reporting visitors only of
course direct through the typing of the url. Are there more diagnostic
tools available to see what the error is, Im pulling my hair out.
I have the exact same issue for my site (www.csharveyimages.com). I've
searched for that phrase and come up with very little information.
There are a couple of threads in this forum talking about it but I
didn't see any really clear answers.
similar problem i am having now, but my site was included in the index
and in my webmaster's tools dashboard, it says that my site is
included in the index but when doing a site:www.shopbotica.com it
can't find my website. i know my site's products are provided by my
affiliate, but i have re-writed the contents, and it doesn't link to
any other affiliate page/sites.
I apologize if I hijacked... Figured since I have the exact same
problem as the OP that I should post here instead of making a new one
to show that Weddingwizard is not alone in this query.
Thanks for your help guys, I think there is a bigger problem here
though.
I should still see something with site:www.weddingwizard.com.au or
site:weddingwizard.com.au but I get no matching documents.
Also the message from webmaster tools is,
"We had problems crawling the pages listed here, and as a result they
won't be added to our index and will not appear in search results."
> Also the message from webmaster tools is,
> "We had problems crawling the pages listed here, and as a result they
> won't be added to our index and will not appear in search results."
Where in Webmaster Tools are you seeing this message?
Web crawl
www.weddingwizard.com.auGooglebot crawls sites by following links from
page to page. We had problems crawling the pages listed here, and as a
result they won't be added to our index and will not appear in search
results.
Review the errors below and check any affected page for problems. For
example, URLs not followed errors can be a clue that some of your
pages contain content (such as rich media files or images) that
Googlebot can't easily crawl, or that their URL structure is not
Google-friendly.
> > Also the message from webmaster tools is,
> > "We had problems crawling the pages listed here, and as a result they
> > won't be added to our index and will not appear in search results."
> Where in Webmaster Tools are you seeing this message?
Alas indeed... I think you just answered you own question. :-)
That text ("Googlebot crawls sites...") appears on this page for
everyone, regardless of whether their site has errors or not, to
explain what kind of data that feature shows. If your site doesn't
have any errors to report, the table will be empty and will say "No
errors found." So it sounds like your site is fine and has no errors.
I thought that if you searched for "site:weddingwizard.com.au" in
google at least it would say that the website existed even if it was
NOT indexed
And also what about this line...is this normal?
"We had problems crawling the pages listed here, and as a result they
won't be added to our index and will not appear in search results."
Its quite explicit, "they wont be added to our index"
If I google weddingwizard.com.au I get this thread and nothing else,
after over 200 direct hits according to google analytics.
Is it possible I have been suspended or penalised for something?
> Alas indeed... I think you just answered you own question. :-)
> That text ("Googlebot crawls sites...") appears on this page for
> everyone, regardless of whether their site has errors or not, to
> explain what kind of data that feature shows. If your site doesn't
> have any errors to report, the table will be empty and will say "No
> errors found." So it sounds like your site is fine and has no errors.
> I thought that if you searched for "site:weddingwizard.com.au" in
> google at least it would say that the website existed even if it was
> NOT indexed
Not true. site: just shows you pages from that site that are indexed,
so if nothing is indexed it will show you 0 results.
> And also what about this line...is this normal?
> "We had problems crawling the pages listed here, and as a result they
> won't be added to our index and will not appear in search results."
> Its quite explicit, "they wont be added to our index"
Yes, but it says "the pages listed here", so if there are no pages
listed there, it means there's nothing that we had problems crawling.
> Is it possible I have been suspended or penalised for something?
How old is your site? Have you previously been indexed? Can you tell
us more about the history of the site?
I have had the site up for about 6 months with a robots text file to
deny all. 2 weeks ago ,I allowed all robots access to my site. So
relatively speaking its only been there for about 2 weeks.
Theres no previous history to this site. I updated a sitemap file
about 3 days ago.
I read another thread saying that the host denied access for googlebot
to crawl the site, could this be a possibility?
> > I thought that if you searched for "site:weddingwizard.com.au" in
> > google at least it would say that the website existed even if it was
> > NOT indexed
> Not true. site: just shows you pages from that site that are indexed,
> so if nothing is indexed it will show you 0 results.
> > And also what about this line...is this normal?
> > "We had problems crawling the pages listed here, and as a result they
> > won't be added to our index and will not appear in search results."
> > Its quite explicit, "they wont be added to our index"
> Yes, but it says "the pages listed here", so if there are no pages
> listed there, it means there's nothing that we had problems crawling.
> > Is it possible I have been suspended or penalised for something?
> How old is your site? Have you previously been indexed? Can you tell
> us more about the history of the site?
If it's been disallowed from crawling for 6 months and only opened up
recently, the current behaviour sounds pretty normal. It can take
awhile for a site to start getting indexed, especially if your site
was disallowed for a long time. Googlebot crawls sites periodically,
and the more frequently a site tells us to go away, the less
frequently we crawl it. I sometimes use a telephone metaphor to
explain this--imagine you called someone and their answering machine
said "I can't come to the phone right now, please don't call me." If
you called back an hour later and got the same message, and then
called back the next day and got the same message, and then called
back the next week and got the same message, you'd probably call less
and less frequently, assuming the response would be the same, right?
The same is true of Googlebot; if your site consistently sends the
message "Don't crawl me," we may wait longer and longer between the
times when we attempt to crawl it again.
Along with submitting a Sitemap, a good way to entice Googlebot to
come back more quickly would be to do the same sorts of things you'd
do when launching a new site--do marketing, online and offline
advertising, get the word out, get people talking about your site and
linking to it and visiting it. Keep building up good content and
developing the site. Once it starts being "active" in the online
ecosystem, it'll naturally end up in search results and start ranking
accordingly.
> If it's been disallowed from crawling for 6 months and only opened up
> recently, the current behaviour sounds pretty normal. It can take
> awhile for a site to start getting indexed, especially if your site
> was disallowed for a long time. Googlebot crawls sites periodically,
> and the more frequently a site tells us to go away, the less
> frequently we crawl it. I sometimes use a telephone metaphor to
> explain this--imagine you called someone and their answering machine
> said "I can't come to the phone right now, please don't call me." If
> you called back an hour later and got the same message, and then
> called back the next day and got the same message, and then called
> back the next week and got the same message, you'd probably call less
> and less frequently, assuming the response would be the same, right?
> The same is true of Googlebot; if your site consistently sends the
> message "Don't crawl me," we may wait longer and longer between the
> times when we attempt to crawl it again.
> Along with submitting a Sitemap, a good way to entice Googlebot to
> come back more quickly would be to do the same sorts of things you'd
> do when launching a new site--do marketing, online and offline
> advertising, get the word out, get people talking about your site and
> linking to it and visiting it. Keep building up good content and
> developing the site. Once it starts being "active" in the online
> ecosystem, it'll naturally end up in search results and start ranking
> accordingly.
> I read another thread saying that the host denied access for googlebot
> to crawl the site, could this be a possibility?
Usually there would be an error message (in your web crawl errors
report) letting you know if this were the case. My guess is that the
new-site-robots.txt-stuff is why your site isn't indexed yet; but if
you're worried about your hoster, you could send them this article and
ask them to double-check whether they're blocking by IP/whitelist:
Thanks for taking the time to answer this question Susan! I guess my
site got indexed between the time I posted in this thread and when you
searched for it! It wasn't there this morning... It is nice to know
that the message in the crawl diagnostic area is basically a "wait a
while" if you don't have errors.
> > I read another thread saying that the host denied access for googlebot
> > to crawl the site, could this be a possibility?
> Usually there would be an error message (in your web crawl errors
> report) letting you know if this were the case. My guess is that the
> new-site-robots.txt-stuff is why your site isn't indexed yet; but if
> you're worried about your hoster, you could send them this article and
> ask them to double-check whether they're blocking by IP/whitelist:
