What are the benefit of using "lemmatize=True" in gensim.corpora.wikicorpus.WikiCorpus(full_path, lemmatize=lemmatize)?

[Willi S.]

Hi,

I'm fairly new to gensim. And I tried to figure out what the differences of lemmatize=True compared to lemmatize=False are. The documentation of this is very short.

If I compare both outputs, I would say that the output generated by lemmatize=False looks a bit better.

So what can you tell me about the differences?

Best regards

Willi S.

[Radim Řehůřek]

Hi Will,

lemmatize=True will turn on lemmatization, which is the process of replacing each word with its lemma = base form. For example, "model = models = modelling = modelled" etc.

https://en.wikipedia.org/wiki/Lemmatisation

Gensim use a 3rd party library called "pattern" for lemmatisation. There have been reports that "pattern" is broken on Python 3, so we may drop it altogether. If `lemmatize=False` works better for you, great.

HTH,

Radim