From Google Public Policy Blog that links to the recs:
http://googlepublicpolicy.blogspot.com/2009/06/our-recommendations-for-increasing.html
Given the tremendous volume of information online -- more than 1
trillion unique URLs and counting -- the ability for users to search
for and find relevant content is critical. This couldn't be more true
for the tens of millions of pages of content stored on government
websites. Unfortunately, many agencies make it difficult or impossible
for search engines to index their sites and make information available
to citizens who are searching for it.
In response to President Obama's call for ideas on how to open up the
government to its citizens, Google put forward recommendations last
Friday in which we point to two simple steps government webmasters can
take to make sure that search engine queries lead users to the right
websites and hopefully, the right answers.
First, agencies can adopt the Sitemaps protocol, which allow search
engines to crawl websites more intelligently. Most search engines
offer free Sitemap generator tools -- check out Google Sitemap
Generator.
Second, agencies can review their robots.txt files. Many agencies
currently block large portions of their websites from search engines
with robots.txt files, sometimes unknowingly. By reviewing and
selectively using these files, webmasters can easily open up large
amounts of content to citizens. Free analysis tools like Google's
robots.txt test can help webmasters identify which pages are
accidentally being blocked.
The next stage in the campaign for open government will come when the
Administration encourages agencies to publish their most popular,
timely, and relevant data on their websites and
data.gov. Static,
obscure, and dated information is not useful to citizens who want data
relevant to their everyday lives, nor is it helpful to third parties
who want to build tools that citizens can use to understand that data.
As it works towards its goal to bring greater transparency to
government, we hope that the Administration continues to take the
steps necessary to make government information more easily accessible
to citizens on the web.