Why does increasing the number of topics increase perplexity?

61 views
Skip to first unread message

日出間健本社総合企画部

unread,
Mar 3, 2023, 4:54:26 AM3/3/23
to Gensim
I'm trying to calculate the perplexity using the LDA model log_perplexity. The official documentation says ”Also output the calculated statistics, including the perplexity=2^(-bound), to log at INFO level.” Therefore, the following calculation is performed in the program to obtain the actual perplexity value. ”np.exp(-lda.log_perplexity(corpus))” In general, perplexity should decrease as the number of topics increases. But in my program it rises exponentially. Is the formula to get the actual perplexity value wrong? Or is the model not learning well? Or is it some other cause? Everyone in the world, please tell me. from Japan
Reply all
Reply to author
Forward
0 new messages