English Word Frequency

[Carmen Kalua]

Youcould use Counter and defaultdict in the Python 2.7 collections module in a two-step process. First use Counter to create a dictionary where each word is a key with the associated frequency count. This is fairly trivial.

Secondly defaultdict could be used to create an inverted or reversed dictionary where the keys are the frequency of occurrence and the associated values are lists of the word or words that were encountered that many times. Here's what I mean:

NLTK will allow you to do all manner of other interesting analysis with text, too, should the need arise. Here's a quick example of how you would find the top 10 bigrams that occur more than 3 times in the text:

I have a csv file with a number of records with paragraphs of text stored as free text strings and I would like to search across all the entries in the column and produce a frequency table of the most used words within all the records

Thanks, Phil. Yes, the concordance should work; I had thought there might be a way to list the most frequent words in order of occurrence. My guess is to set the parameters and then, run the data. That should work, shouldn't it? Someone has done a lot of work in creating Logos.

Usually you need to calculate your own word frequencies from your own corpus. Very large corpora are better but of course this takes a lot of work, space, time, etc, and distracts from the task at hand.

This is problematic because it may suggest spurious regularity.Footnote 6 The problem can be best understood by a simple example. Imagine that all words in language were actually equally probable. In any sample (corpus) of words, we will find that some words occur more than others just by chance. When plotted in the standard manner, we will find a strikingly decreasing plot, erroneously suggesting that the true frequency-rank relationship has some interesting structure to be explained. This spurious structure is especially problematic for low-frequency words, whose frequencies are measured least precisely. Additionally, in the standard plot, deviations from the Zipfian curve are difficult to interpret, due to the correlation of measurement errors; it is hard to tell systematic deviations from noise.

Figure 1a shows such a plot, giving the frequency/frequency-rank relationship from the American National Corpus (ANC; Reppen & Ide, 2004), a freely available collection of written American English. All figures in this paper follow this plotting procedure unless otherwise noted. The plot shows a two-dimensional histogram of where words fall in frequency/frequency-rank space.Footnote 7 The shading of the histogram is done logarithmically with the number of words falling into each hexagonal bin and is white for zero-count bins. Because the plot has a logarithmic y-axis, words with zero frequency after the split are not shown. The fit of Eq. 2 using a maximum-likelihood method on the separate frequency and frequency rank portions of the corpus is shown in the red solid line. Additionally, a locally smoothed regression line (LOESS) (Cleveland, Grosse, & Shyu, 1992) is shown in gray. This line corresponds to a local estimate of the mean value of the data and is presented as a comparison point to see how well the fit of Eq. 2 matches the expected value of the points for each frequency rank (x-value). In the corner, several key values are reported: the fit α and β, an R 2 measure giving the amount of variance explained by the red line fit, and an adjusted R 2 adj capturing the proportion of explainable variance captured by the fit, taking the smoothed regression as an estimate of the maximum amount of variance explainable. For simplicity, statistics are computed only on the original R 2, and its significance is shown with standard star notation (three stars means p This plot makes explicit several important properties of the distribution. First, it is approximately linear on a log-log plot, meaning that the word frequency distribution is approximately a power law, and moreover, is fit very well by Eq. 2 according to the correlation measures. This plot shows higher variability toward the low-frequency end, (accurately) indicating that we cannot estimate the curve reliably for low-frequency words. While the scatter of points is no longer monotonic, note that the true plot relating frequency to frequency rank must be monotonic by definition. Thus, one might imagine estimating the true curve by drawing any monotonic curve through these data. At the low-frequency end, we have more noise and, so, greater uncertainty about the shape of that curve. This plot also shows that Eq. 2 provides a fairly accurate fit (red) to the overall structure of the frequency-rank relationship across both corpora.

In general, it is not so important which simple distributional form is a better approximation to human language. What matters more are the general properties of word frequencies that are informative about the underlying mechanisms behind the observed distribution. This section tries to bring out those general properties. Do the distributions appear near-Zipfian for systematic subsets of words? Are distributions that look similar to power laws common across word types, or are they restricted to words with certain syntactic or semantic features? Any psychologically justified theory of the word frequency distribution will depend on appreciating, connecting to, and explaining these types of high-level features of the lexical frequency distribution.

In this plot, because the rank ordering is fixed across all languages, not only do frequencies fall off like Eq. 2, but they do so with roughly the same coefficients cross-linguistically. If frequency was not systematically related to meaning, these plots would reveal no such trends.

Frequency distribution of words within several syntactic categories from the Penn Treebank: determiners (DTs), prepositions or subordinating conjunctions (INs), modals (MDs), nouns (NNs), past participle verbs (VBNs), third-person singular present verbs (VBZs). These plots represent a post-hoc selected subset of all syntactic categories

The resulting subject-average frequency distribution is shown in Fig. 8. This clearly demonstrates near-Zipfian scaling in frequency, despite the fact that all words are, in some sense, equivalent in the prompt; participants are not told, for instance, that one creature is extra salient or that they should primarily describe one character. The context was chosen to bias them as little as possible as to how much to describe each creature and what role it should play in their novel story. Moreover, subjects show this distribution even though they are told almost nothing about the creatures (other than that they crashed from an alien ship) and are told absolutely nothing about what happens next. Even in this context, words still approximately follow the power law distribution, although larger-scale studies should be used to check that this effect is seen within individuals and is not the result of averaging together subjects.

Beyond the theoretical arguments against random-typing accounts, such accounts are not compatible with several empirical facts reviewed earlier. The systematicity of word frequencies across meanings (see Semantics strongly influences word frequency) are particularly problematic for random-typing models, since any process that is remotely like random typing will be unable to explain such patterns. One certainly would not be able to explain why cardinal number words also follow a near-Zipfian distribution, ordered precisely by magnitude. Moreover, random-typing accounts cannot explain the variability across syntactic categories (see The fit of Zipfian distributions vary by category). Why would certain word categories appear not to follow the model? Nor can it explain the tendency of subjects to follow the distribution for novel words (see Power laws arise from [almost] nothing), and the simplest forms of random-typing models are incompatible with the nonstationarity word frequencies exhibit (see The distribution of word frequencies is not stationary).

This is certainly not to say that there is no way a communicative theory could account for the facts or that communicative influences play no role. An adequate theory has just not been formalized or empirically evaluated yet.Footnote 17

Such estimation also violates the assumptions of typical algorithms used to fit Zipfian exponents, since most fitting algorithms assume that x is known perfectly and only y is measured with error. This concern applies in principle to maximum-likelihood estimation, least squares (on log-log values), and any other technique that places all of measurement error on frequencies, rather than both frequencies and frequency ranks.

In evaluating theories, one might wonder whether these semantic comparisons are essentially just random subsets of words and whether a random subset of a Zipfian distribution may tend to look Zipfian. Therefore, it may not be very strong evidence against theories based on meaning that we still see Zipfian distributions when we control or constrain meaning. However, note that theories based on meaning explain the distribution starting from semantics. They explain patterns across the entire lexicon by appealing to semantic properties of single words and, so, cannot explain the subsets of words that look Zipfian but do not have the required semantic properties.

I do that sort of analysis in AntConc too. It does more than just word frequency lists. There are other concordance tools around but AntConc is readily available and has both Mac and Windows variants.

Use a hash table (e.g., HashMap) to collect all words and their frequencies. Then, use counting sort to sort the words in order of decreasing frequency. Since all frequencies are integers in the range $1..n$, counting sort takes $O(n)$ time. The total expected running time is $O(n)$, which is more than likely more than sufficient for all practical purposes (unless the interviewer mentioned something that was left out of your question). Make sure to mention that this is expected running time rather than worst-case running time.

3a8082e126