I'm using gensim's ldamodel in python to generate topic models for my corpus. To evaluate my model and tune the hyper-parameters, I plan to use log_perplexity as evaluation metric.
However, computing log_perplexity (using predefined LdaModel.log_perplexity function) on the training (as well on test) corpus returns a negative value (~ -6). I'm a little confused here if negative values for log perplexity make sense and if they do, how to decide which log perplexity value is better ? Should I try to minimize magnitude of log perplexity?
Following are the parameters I'm using while training -
num_topics = 50
alpha = 0.02
eta = 0.02
iterations = 100
passes = 10
Other optional parameters are default
Training corpus details -
Number of documents ~ 30,000
Vocabulary size (after removing stop words, verbs, adjectives, etc.) ~ 35000
Median document size (after removing stop words, etc.) ~ 50
Gensim python library version - 3.4.0
Thanks!