models.phrases.Phrases min_count effect on unigrams

[Pogens Porgue]

https://radimrehurek.com/gensim/models/phrases.html

models.phrases.Phrases documentation says:

   min_count (float, optional) – Ignore all words and bigrams with total collected count lower than this value

and while the bigrams original_scorer does clearly use min_count, it does not appear to have any effect on unigrams, so I'm not sure what 'words' refers to in the "Ignore all words and bigrams" 

I've looked at the code for models.phrases._sentence2token , as called by the Phrases.__getitem__ , and it only uses Phrases.analyze_sentence() to join unigrams to bigrams, then uses new_s.append(words) two lines from the end to return a list of all unigrams and bigrams.  My interpretation of min_count as described above is that it would not return words below the min_count value, i.e., would have instead

      if phrase_class.vocab[words] >= phrase_class.min_count: new_s.append(words)

I do want that functionality and of course can just subclass and redefine _sentence2token , but I'm wondering what I'm misinterpreting in the above documentation. Tks

[Gordon Mohr]

Yes, that looks like a misleading bit of documentation. The effect of `min_count` is dependent on the `scoring` choice, and doesn't appear to me to ever be as simple as "ignoring" all unigram or bigram tokens below this count. 

(If your actual aim is to simply elide any unigrams/bigrams below a certain count, similar to the operation of `min_count` in the `Word2Vec`/etc classes, that might be cleanest to do as a separate step/pass – even if you leverage the frequency-survey made by the `Phrases` class.)

- Gordon

[Pogens Porgue]

Thanks for confirming that the documentation is misleading (maybe it can be fixed?).

In any event, I had independently come to the conclusion that a separate step/pass was more sensible, so completely agree.

(In this case am using the min_df parameter in sklearn.feature_extraction.text.CountVectorizer.)

But it was nonetheless worthwhile to have examined the code for the phrases class, since I'm also manipulating my Phraser.phrasegrams