Mutual Information in Feature Selection

[Ahmad Ahmadian]

Hi,

in feature selection, one way for finding best single features is computing the mutual information between the labels and the each individual feature. This, however, involves estimation of joint probabilities of labels and features. Given that labels are real-valued and not discrete, how is mutual information computed? Because obviously if one had the real joint distribution at hand then the rest wouldn't matter and even the learning task would be done. (and feature selection)

Thanks in advance

[Maksym Andriushchenko]

Hi,

You are right, and this point was mentioned in the lecture slides as a disadvantage of the filter method with mutual information:

And that's why Fisher Score and Correlation were introduced as alternatives to mutual information for the filter method.

But in case we want to really use mutual information -- estimating joint density p(x, y) is indeed a hard task, usually even harder than just estimating p(y|x) which is usually the goal in many classification approaches. 

And it's true that if one knows p(x, y), then one knows basically everything, and can classify data points, generate new samples, perform data completion (e.g. if some x_i is unknown), etc.

However, one can still get *some* approximations of p(x, y), which might be useful in some cases. For example, if x and y are continuous, one can fit e.g. a mixture of Gaussians to the training data, and then get an estimate of p(x, y). Or one can discretize x and y, and then simply perform counting as can be done for the discrete case. 

And also note that although the discrete case sounds conceptually simpler, this kind of counting gets directly affected by the curse of dimensionality, so it may be again not very practical.

Best,
Maksym

--
You received this message because you are subscribed to the Google Groups "Machine Learning WS18/19" group.
To unsubscribe from this group and stop receiving emails from it, send an email to machine-learning-...@googlegroups.com.
To post to this group, send email to machine-lea...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/machine-learning-ws1819/CANCk4M%2BQ7zsS6gcH0djtGMWV6%3DdKReKpP3nagsVPPaUMtHTNpQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.