The WebCrawler Index is now available for searching! The index is broad:
it contains information from as many different servers as possible. It's
a great tool for locating several different starting points for exploring
by hand. The current index is based on the contents of documents located
on nearly 4000 servers, world-wide.
Check it out at: http://www.biotech.washington.edu/WebCrawler/WebQuery.html
Other information is available from there, including a description of the
WebCrawler (the robot itself), and a list of the 25 most frequently
referenced sites on the Web.
Dept of Computer Science and Engineering
University of Washington