MSE vs. RMSE

[Raphael Volz]

Hi-

apologies for the "newby" question. In training / validation metrics is MSE rooted or not ? (aka. RMSE/RMSD). I am slightly confused as residual deviance is often shown in the scale of the predictor (rooted) but when compared to results of another system the MSE computed by H20 in the example below seems not to be square rooted:

MSE	0.000966
r2	0.867530
mean_residual_deviance	0.000966

Thanks

Raphael

[SriSatish Ambati]

Raphael,

I'd like to get the science team to comment.

It looks like we are doing just squared error - not root mean squared error -

Sri

https://github.com/h2oai/h2o-3/blob/dbc1245c1a6f9baef4479f32ad6361639159d0cd/h2o-core/src/main/java/hex/ModelMetricsRegression.java#L63

public ModelMetrics makeModelMetrics( Model m, Frame f) {
	double mse = _sumsqe / _wcount;
	double meanResDeviance = _sumdeviance / _wcount; //mean residual deviance
	return m._output.addModelMetrics(new ModelMetricsRegression( m, f, mse, weightedSigma(), meanResDeviance));
	}

--
You received this message because you are subscribed to the Google Groups "H2O Open Source Scalable Machine Learning - h2ostream" group.
To unsubscribe from this group and stop receiving emails from it, send an email to h2ostream+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

culture . code . customer

c o m m u n i t y, h2o.ai, 

+1-408.316.8192

[Arno Candel]

Raphael,

GLM uses the following deviances: https://github.com/h2oai/h2o-3/blob/arno_jenkins/h2o-algos/src/main/java/hex/glm/GLMModel.java#L240-240

GBM/DeepLearning uses these: https://github.com/h2oai/h2o-3/blob/arno_jenkins/h2o-core/src/main/java/hex/Distribution.java#L72-72

In general, we use the same conventions as R for (mean) residual deviances, and rigorously test that the result match. For example, for GBM:

Gaussian:

mean residual deviance = mean squared error (not rooted)

https://github.com/h2oai/h2o-3/blob/master/h2o-r/tests/testdir_algos/gbm/runit_GBM_offset_gaussian.R

https://github.com/harrysouthworth/gbm/blob/master/src/gaussian.cpp#L96

Poisson:

mean residual deviance != mean squared error

https://github.com/h2oai/h2o-3/blob/master/h2o-r/tests/testdir_algos/gbm/runit_GBM_offset_poisson.R

https://github.com/harrysouthworth/gbm/blob/master/src/poisson.cpp#L81

Gamma:

mean residual deviance != mean squared error

https://github.com/h2oai/h2o-3/blob/master/h2o-r/tests/testdir_algos/gbm/runit_GBM_offset_gamma.R

https://github.com/harrysouthworth/gbm/blob/master/src/gamma.cpp#L97

Tweedie:

mean residual deviance != mean squared error

https://github.com/h2oai/h2o-3/blob/master/h2o-r/tests/testdir_algos/gbm/runit_GBM_offset_tweedie.R

https://github.com/harrysouthworth/gbm/blob/master/src/tweedie.cpp#L100

Example for GLM, there’s more:

https://github.com/h2oai/h2o-3/blob/master/h2o-r/tests/testdir_algos/glm/runit_GLM_link_functions_poisson.R

Please let me know if you have any further questions.

Thanks,

Arno