Cross Validation Can Estimate How Well Prediction Variance Correlates with Error

[Felipe A. C. Viana]

Dear all,

Here it is a reference on analysis of uncertainty structures of
surrogate models:

F. A. C. Viana and R.T. Haftka, "Cross Validation Can Estimate How
Well Prediction Variance Correlates with Error," AIAA Journal, Volume
47, Number 9, pp. 2266-2270, 2009 (DOI: 10.2514/1.42162).

The use of surrogates for facilitating optimization and statistical
analysis of computationally expensive simulations has become
commonplace. They offer easy-to-compute prediction and in some cases
(such as kriging and polynomial response surfaces), they also furnish
the prediction variance as a measure of uncertainty. Adaptive sampling
and optimization methods use the prediction variance to select the
next sampling point. For example, the Efficient Global Optimization
(EGO) algorithm uses the kriging prediction variance to seek the point
of maximum expected improvement as the next simulation for the
optimization process. For such methods, it is important to assess the
accuracy of the prediction variance; but presently, this is not
available. We propose using cross validation for estimating the
correlation between the prediction variance and the errors.
Specifically we propose to use the correlation between the absolute
values of the cross-validation errors and the square root of the
prediction variance at the points that were left out. The approach was
tested on two algebraic examples for kriging and polynomial response
surface surrogates. For these examples we found that while we may
obtain only a rough estimate of the correlation between their
prediction variance and actual absolute errors, we succeeded in
selecting surrogates with good correlation. Surprisingly, 1) the
statistically based prediction variance may not always correlate well
to the errors; 2) the surrogate with the most accurate predictions did
not necessarily have the best correlation; 3) with sparse data sets,
the trend function influenced the quality of the correlation of
kriging surrogates; and 4) the uncertainty structure of polynomial
response surfaces was almost as good as (and sometimes better than)
the best kriging surrogate.

You can find more about it online:
http://fchegury.googlepages.com/publications


All the best,
Felipe A. C. Viana