Difference between Gensim's LDA Mallet Wrapper and Gensim LDA

[Vikash Bajaj]

I am currently using Gensim's LDA Mallet wrapper. While the topics/keywords generated using Mallet are great, it is very slow when it comes to topic assignment. I tried using the Gensim's Multicore LDA and its is very very fast but the topics generated are very poor relative to the Mallet. My question is, Is there any difference in the LDA implementation in Mallet and Gensim's LDA or am I missing something in my implementation? I can provide sample codes if required.

thanks

[Lev Konstantinovskiy]

Hi Vikash,

The inference algorithms in Mallet and Gensim are indeed different. Mallet uses Gibbs Sampling which is more precise than Gensim's faster and online Variational Bayes. There is a way to get relatively performance by increasing number of passes.

Regards

Lev

[Zafar Shadman]

On Monday, March 20, 2017 at 1:20:35 PM UTC+5:30, Vikash Bajaj wrote:

I am currently using Gensim's LDA Mallet wrapper. While the topics/keywords generated using Mallet are great, it is very slow when it comes to topic assignment. I tried using the Gensim's Multicore LDA and its is very very fast but the topics generated are very poor relative to the Mallet. My question is, Is there any difference in the LDA implementation in Mallet and Gensim's LDA or am I missing something in my implementation? I can provide sample codes if required.

thanks

After training the mallet it is not possible to train the mallet in one system and test it in another system, mallet seems to hold the mallet path and prefix path of exact same system it has been trained on and nearly impossible to test the trained model on another system. If anyone have any solution please provide.

[Aashish Khadka]

I am having similar issues with training and testing mallet model in different system or console and I have looked through the internet for 2 weeks now but can't find solution.

[Radim Řehůřek]

Yes, Mallet produces files which are required if you want to do LDA inference later.

These files are stored under the given filesystem prefix (temp by default, IIRC).

If you move your Mallet model later to a different location, you'll have to update this prefix path, so files are found again: lda_mallet_model.prefix = '/some/other/prefix'

Note that these files are not produced by Gensim, and Gensim doesn't manage them, nor make any compatibility warranties about their content.

HTH,
Radim

[Aashish Khadka]

Radim, below is the link to similar question I posted. Unfortunately none replied. Can you please look into it?

https://groups.google.com/d/msg/gensim/vVO0_t9jRUo/ZYFdq9_TBgAJ