Confidence intervals

[J F]

Hi all,

Another question: we are asked for confidence intervals around our predictions. However, the method i'm currently using does not easily provide any confidence intervals. I'm wondering what I should do.

One option would be to bootstrap the training data repeatedly and fit, say, a hundred models to the different resamplings and obtain a standard error around the mean for the predictions of each individual, but that would be highly costly computionally.

Is it strictly necessary to calculate confidence intervals?

In the details it is mentioned that " Methods or algorithms that do not produce probabilistic estimates can still participate, by setting binary probabilities (zero or one) and default confidence intervals" but i'm unclear about 'default' confidence intervals are. Could I simply use something like mean+-0.1*mean and mostly compare my model to others in the leaderboard using the MAE?

Kind regards,
Jeroen

[Neil Oxtoby]

Yep — it's necessary to provide confidence intervals. Part of the challenge!

You should be able to find the default confidence intervals in the evaluation script evalOneSubmissionD4.py.

Neil