It would be hard to conduct a test with a 1 TB corpus.  Few people have
enough computing power to do this, and I am sure you don't want to host
the corpus on your website.

I would suggest something on the order of 1 GB.  Turing estimated in
1950 that the AI problem would take 10^9 bits of memory.  He did not
say how he came up with this number.  But consider a human exposed to
150 words per minute, 12 hours a day for 20 years.  This is about 4 GB
of text, or a 1 GB zip file.  So a machine learning algorithm should
have a similar constraint.

Even this may be too high.  The average human vocabulary is 30K common
words plus an equal number of proper nouns.  To learn a word you have
to be exposed to it about 20 times.  By Zipf's law, the n'th most
frequent word in English is 0.1/n.  This means that you need about 12
million words or 50 MB.

Semantic models using LSA have used about 250 MB, like the WSJ corpus.
However, with a larger corpus there is a tendency to get lazy.  For
example, with LSA you use the fuzzy equivalence A ~ B, meaning "A has
similar meaning to B" or "A and B appear in the same proximity".

A ~ A (reflexive, used by most data compressors)
A ~ B implies B ~ A (symmetric, used in distant bigram models)
A ~ B and B ~ C implies A ~ C (transitive, used by LSA)

LSA uses the transitive property to predict that C follows A even if
those two words were never seen together before.  However with a larger
corpus you can collect the statistics directly rather than use LSA to
infer them, so you tend take the easy route.  In the above paper on
word analogy solving, I see a lot of steps were data was discarded, but
for a 1 TB corpus it didn't matter.

You might be able to construct a high quality (diverse) corpus by
finding the minimum set of documents that contain all the words in a
dictionary.  You might also consider a corpus of several languages to
test whether a model can learn a generic language without hardcoded
rules for English.

-- Matt Mahoney