Perhaps Wikipedia would suffice:

http://download.wikimedia.org/#wikipedia

The current english language edition, gzipped, is about 1GB, but as you
see, there are many Wikipedias in different languages.  I doubt they
total over 4GB gzipped.

Compressing multiple languages would have a very good side effect:
Machine translation technology could be dramatically enhanced and
properly constructed compressors would discover things about the
various cultures that might not otherwise be obvious.

Part of my preference for picking a big corpus to begin with is so that
if the contest goes big time -- which it would if proper funding levels
were applied to it -- some sort of rules might be constructed to allow
some divisions of the contest for large computer systems to crunch away
doing more brute-force statistics such as Turney used, while other
divisions could do a fairly sampled subset of the total corpus.  But
this is something we can deal with later when the competition (and
prize) gets much bigger.

PS:  Another reason I'd like to see huge corpora ultimately is that I'd
really like to see this system beat humans by a substantial amount:
We're so stupid we desperately need all the help we can get.