Configuring Google Mini to properly crawl MediaWiki?

[sprak]

We currently have a Google Mini setup to crawl our MediaWiki sites;
however, the Mini
is not keeping all the pages in its index. Here is the behavior we
are seeing; a page gets
crawled by the Mini and is placed in the index. The crawler moves
along and eventually
comes to a MediaWiki page that links back to the first page.

This page is flagged as 'nofollow' in its meta data; the Mini does not
crawl this page and
add it to the index, and it removes the first page from the index
because of the link to it.
This is not what we want to happen and have tried to setup patterns to
exclude the
nofollow pages.

We are clearly not catching them all though, and I have not seen a
guide floating around
that outlines the best patterns/practices for getting a Mini to play
nice and index the
content of a MediaWiki site. We could edit the MW source and just
have it not
output the nofollow meta data, but this is not an optimal solution
overall.

Does anyone know of a good guide or can post their best patterns/
practices. For reference,
here are the patterns we are currently using:

#wiki exclusion list
contains:/Special:
contains:/User:
contains:/Talk:
contains:_Talk
contains:_talk
contains:=history
contains:&diff=
contains:action=edit
contains:Recentchanges&
contains:=Talk
contains:skins
#end wiki exclusions



Thanks.

- Luis Cruz

[bria...@gmail.com]

Hi Luis,

If I understand this issue correctly (please correct me if I am
wrong), you have a page that you want in the index. This page is
crawled but later on, when the Mini crawls pages that have a link and
a NOFOLLOW tag on them, when looking in Crawl Diagnostics, you see an
error saying that this page is rejected by robots? Is this correct? If
this is the case, there is a known issue where this is incorrectly
reported in the Crawl Diagnostics but the page should still be
crawled. The problem was that this is a reporting issue and the page
should still correctly show up in search. Can you verify if these
pages are are actually in the index (i.e. they return in searches?) If
so, then you can ignore those messages.

Brian

[piggybankpie]

Here's my exception rules for Media wiki... what a mess!

contains:&action=history
contains:&action=edit
contains:&printable=yes
contains:/Special:Upload
contains:/Special:Recentchanges
contains:/Special:Whatlinkshere
contains:_talk:
contains:&oldid=
contains:&limit=
contains:&offset=
contains:&feed=
http://wiki.%our_wiki%/wiki/images/deleted/
http://wiki.%our_wiki%/wiki/images/
http://wiki.%our_wiki%/wiki/skins/
http://wiki.%our_wiki%/wiki/math/
http://wiki.%our_wiki%/wiki/extensions/
http://wiki.%our_wiki%/wiki/bin/
http://wiki.%our_wiki%/wiki/docs/
http://wiki.%our_wiki%/wiki/languages/
http://wiki.%our_wiki%/wiki/locale/
http://wiki.%our_wiki%/wiki/logo/
http://wiki.%our_wiki%/wiki/maintenance/
http://wiki.%our_wiki%/wiki/serialized/
http://wiki.%our_wiki%/wiki/index.php/Special:Contributions/
http://wiki.%our_wiki%/wiki/index.php/Special:Export
http://wiki.%our_wiki%/wiki/index.php/Special:ListUsers
http://wiki.%our_wiki%/wiki/index.php/Special:Listuser
http://wiki.%our_wiki%/wiki/index.php/Special:Log
http://wiki.%our_wiki%/wiki/index.php?Title=Special:Log
http://wiki.%our_wiki%/wiki/index.php/Special:Mypage
http://wiki.%our_wiki%/wiki/index.php/Special:Search
http://wiki.%our_wiki%/wiki/index.php?title=Special:Recentchanges
http://wiki.%our_wiki%/wiki/t/
http://wiki.%our_wiki%/wiki/tests/
http://wiki.%our_wiki%/wiki/index.php/Help
http://wiki.%our_wiki%/wiki/index.php/KBI_talk
http://wiki.%our_wiki%/wiki/index.php/KBH_talk
http://wiki.%our_wiki%/wiki/index.php/User
http://wiki.%our_wiki%/wiki/index.php/User_talk
http://wiki.%our_wiki%/wiki/index.php/Wiki:About
http://wiki.%our_wiki%/fckeditor/
http://wiki.%our_wiki%/wiki/includes/
http://wiki.%our_wiki%/wiki/index.php/Talk:/
http://wiki.%our_wiki%/wiki/index.php/Backup_single/
http://wiki.%our_wiki%/wiki/index.php/Category:Help/
http://wiki.%our_wiki%/wiki/index.php/Example_page/
http://wiki.%our_wiki%/wiki/index.php/Manual:Image_Administration/
http://wiki.%our_wiki%/wiki/index.php/Manual:Interface/
http://wiki.%our_wiki%/wiki/index.php/Manual:LocalSettings.php/
http://wiki.%our_wiki%/wiki/index.php/MediaWiki/
http://wiki.%our_wiki%/wiki/index.php/MediaWiki:Registry/
http://wiki.%our_wiki%/wiki/index.php/Portal:Contents/
http://wiki.%our_wiki%/wiki/index.php/Talk:/
http://wiki.%our_wiki%/wiki/index.php/Talk:Backup_single/
http://wiki.%our_wiki%/wiki/index.php/Talk:Example_page/
http://wiki.%our_wiki%/wiki/index.php/Talk:Manual:Image_Administration/
http://wiki.%our_wiki%/wiki/index.php/Talk:Manual:Interface/
http://wiki.%our_wiki%/wiki/index.php/Talk:Manual:LocalSettings.php/
http://wiki.%our_wiki%/wiki/index.php/Talk:MediaWiki/
http://wiki.%our_wiki%/wiki/index.php/Talk:Portal:Contents/
http://wiki.%our_wiki%/wiki/index.php/Template:Admin_tip/
http://wiki.%our_wiki%/wiki/index.php/Template:Languages/
http://wiki.%our_wiki%/wiki/index.php/Template:Meta/
http://wiki.%our_wiki%/wiki/index.php/Template:Note/
http://wiki.%our_wiki%/wiki/index.php/Template:PD_Help_Page/
http://wiki.%our_wiki%/wiki/index.php/Template:POTD_protected/
http://wiki.%our_wiki%/wiki/index.php/Template:Thankyou/