Why does increasing the number of topics increase perplexity?

Skip to first unread message


Mar 3, 2023, 4:54:26 AMMar 3
to Gensim
I'm trying to calculate the perplexity using the LDA model log_perplexity. The official documentation says ”Also output the calculated statistics, including the perplexity=2^(-bound), to log at INFO level.” Therefore, the following calculation is performed in the program to obtain the actual perplexity value. ”np.exp(-lda.log_perplexity(corpus))” In general, perplexity should decrease as the number of topics increases. But in my program it rises exponentially. Is the formula to get the actual perplexity value wrong? Or is the model not learning well? Or is it some other cause? Everyone in the world, please tell me. from Japan
Reply all
Reply to author
0 new messages