HW3 Q3

[alex smirnov]

Hi,

Does anybody know how to answer the  "Comment on the potential validity of other measures of model precision"  (HW III, problem 3)?

[Nicholas Woo]

I'm not sure if this is the correct answer, but here's what I put:

"When using a linear probability model to forecast dichotomous events, the value of R-Square is not necessarily a good predictor of how well the model can classify events. R-Square in the context of this problem is approximately 72%, yet was able to correctly classify events 100% of the time using a 50% cut-off. However, t-statistics and P-values remain to be valid measures of statistical significance when measuring the strength of relationships between individual variables and expected values of y."

How about the question on whether to use the 50% cutoff in practice? I was less sure about my response to that question, but currently have the following:

"In practice, I would begin by evaluating the results of my model using various cut-offs (i.e. 25%, 50%, 75%, etc.) in order to determine which cut-off % predicts with the highest percentage of accuracy given the historical data. In the context of this problem, it turns out that the 50% cut-off predicted with the highest percentage accuracy in comparison to other cut-off alternatives, but this may not always be the case since it depends on the nature of the data."

Let me know on the answers you decide on...

--Nick

[alex smirnov]

Hi Nick,

Thank you,

This is what I put for this part of the hw

If 50% probability of default as the cutoff level is used, the model is 100% accurate (age is not a significant variable) While R square is only 72%, the model is much more predictive than R square indicates Even with a 100% accuracy given the 20 data points, I would not want to use 50% as a cutoff. The reason is that I would rather deny a loan to someone who would pay it off and forgo the interest rather than loan money to someone who will default and loose the principal.   Given that Default = 1, and Successful Payment = 0, I would set the cutoff closer to 30% This only produces an 80% acuracy, but is safer from a credit union perspective  

Regards,

Alex