using Boosting for Random Forest

[Soomi]

We know that Random Forest scheme take advantages of two Bagging and Randomization scheme. For improving the results of Random Forest even more, is that reasonable to use the AdaBoost for Random Forest too? 

[Soomi]

Somewhere in the text book it is written that "It can be shown theoretically that boosting only fails on fresh data
if the individual classifiers are too complex". Does it mean that we can not use the boosting for Random Forest?

[Ian Witten]

Excellent questions!

As you know, boosting produces an initial model, then a second model that concentrates on instances the initial model misclassifies, then a third that concentrates on instances the first two misclassify, and so on. If you use a learner that produces a simple model that does not overfit, this process is guaranteed to converge to a good classifier.

However, if the learner overfits the training set all bets are off. In the extreme, the very first model might correctly classify all training instances (making further iterations unnecessary), and yet generalise poorly to fresh test instances.

Random Forest produces a complex model and is prone to overfitting, so you are correct in thinking that it is probably unsuitable for boosting.

However, if you limit the depth of the trees that Random Forest uses (there is a maxDepth parameter that does this), you force a simpler model — which might be worth boosting.

cheers

ian

--
You received this message because you are subscribed to the Google Groups "WekaMOOC-general" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wekamooc-gener...@googlegroups.com.
To post to this group, send email to wekamooc...@googlegroups.com.
Visit this group at https://groups.google.com/group/wekamooc-general.
To view this discussion on the web, visit https://groups.google.com/d/msgid/wekamooc-general/95e45fb1-9b28-4a21-8a53-134ebca9f84a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.