We run a site called wikihow.com, a wiki-based site for original how-to
information. We've been up and running now since January, 2005 and we
seem to have some problems being indexed in Google.

We recently ran a sample test of 1,000 URLS and found that 629 of the
URLs were not indexed in Google. The average age of these URLs is 184
days - about 6 months. We currently have over 14,000 how-to articles on
the site.

These results are vey surprising. wikiHow is a popular website (Alexa
2000) with hundreds of thousands of real inbound links. We have been
running a Google sitemap of our URLs for well over a year, which has
been verified to be working by the Google webmaster tools. In the
sitemap, we have been using a priority of 1.0 sparingly for pages that
have been created in the previous 7 days.

Each of our articles is reachable by 4 clicks or less from the main
page through our category structure http://www.wikihow.com/Categories
(although we are aware there are a few catgories with 200 or more
articles in them).

While we haven't been using a robots.txt out of fear that we would
unnecessarily exclude good pages, we have been using meta tags to
specific index,follow on all of our article pages, while using
noindex,follow appropriate on dynamic pages and older revisions of
articles.

Our software is based on Mediawiki, the same platform that Wikipedia
runs on and we are not showing different versions to bots than we are
to anonymous/regular users. Nor are we trying any other search engine
trickery.

Does anyone have any suggestions of what the problem(s) could be?

How many indexable pages do you have on the site and how many of those
are indexed? It sounds like you expect about 14,000 pages to be indexed
and around 11,000 are. Is that correct?

Looking at your site in the search results, it appears that your pages
would be well served by meta description tags. For most queries, the
generated snippet is based on where the query terms are found on the
page, and in those cases, your results are fine. But for some more
generic queries, where a logical snippet isn't found in the text, the
generated snippet seems to be coming from the first bits of text from
the page -- in this case, boilerplate navigation that is the same for
every page.

Is there a static link to every page on the site?

Hi,

Thanks for your response.

We have about 14,000 how-to articles on the site and it looks like only
about 4,000-5,000 are indexed at the moment. We have several thousand
more URLs that have less priority such as 'talk' pages for articles,
category pages, image pages, etc, but we're focusing on the article
pages since they are most relevant.

Thanks for the suggestion about the description. The absence of this
tag shouldn't be interfering with pages being indexed though, should
it?

Yes, there is a static link to every page on the site. Each page is
categorized and can be reached from the main page through our category
structure in 3 clicks or less, with a few small exceptions.

I've also done some analysis of our web access logs for the month of
October and it does seem like the Googlebot is reaching all of our
pages on the site, as it has made several thousand requests for main
how-to article pages in October.

One possibility is that we transferred domains from wiki.ehow.com to
wikihow.com in May, 2006. We've properly implemented 301 directs for
the old wiki.ehow.com URLs, but some pre-May 2006 articles appear to be
neither indexed in wiki.ehow.com nor wikihow.com - could these pages be
in a limbo state somewhere? If so, this would still only account for
part of our problem, there are still several URLs that were created
after May, 2006 that should be indexed that haven't been.

Here's an example:

http://www.wikihow.com/Sit-in-a-Kilt

It's a pretty informative article that was first written in July, 2006.
If one searches on Google for

site:wikihow.com sit kilt

or just

how to sit in a kilt

This page does not appear in the search results.

If you have any ideas or suggestions, we'd appreciate it.

Thanks.

You're right. The absense of a meta description tag won't impact
indexing. It sounds like you're doing the right things -- I'll take a
look at the things you've mentioned and let you know if I see anything
that might be helpful.

Hi Vanessa,

Were you able to look into this?

Thanks.