Surrogate Variables
Phil Sherrod
(http://www.dtreg.com) that I think others in this group may be interested
in. The new feature is "Surrogate Variables". This is a method for
imputing the value of a missing predictor variable based on another
(surrogate) variable. The idea of using surrogate variables has been around
for a long time, but I'm not personally familiar with any other predictive
modeling program that has the feature built in as a standard option. I
don't think may people are familar with the idea of surrogate variables.
A Surrogate Variable is another predictor variable that is "associated"
(correlated) with a primary predictor variable and can be used to fill in
missing values for its associated primary variable.
Before the model building process starts, DTREG examines each potential
surrogate variable for each primary predictor variable and computes the
association between the variables. Continuous and categorical predictor
variables with two categories may have surrogates and be used as surrogates.
Categorical variables with more than two categories cannot have surrogates
nor can they be used as surrogates. The mode (most common category) is used
as the replacement value for categorical variables with more than two
categories.
If there are n eligible variables, then n*(n-1) potential matches must be
evaluated. Fortunately, fiting a function using linear regression is very
fast. For each potential variable pair, the association is calculated; the
association measures how closely the variables are related. Association
values range from 0 (no association) to 100 (perfect association). The
surrogates with the highest association are connected to the primary
predictor. So each predictor has a different set of surrogate variable
functions.
The method used to compute the association depends on the type of the
predictor:
1. Continuous predictors � Linear regression is used to fit a function:
predictor=f(surrogate).
The association is then computed as 100 times the proportion of the variance
of the predictor explained by the function. So if the function output
exactly matches the predictor, the association is 100. Linear, quadratic,
and cubic functions may be used.
2. Categorical predictors � A slightly different method is used to compute
the association for categorical predictors with two categories.
If the potential surrogate is also categorical, the values of the predictor
and the surrogate are compared and the proportion of the values that match
(have the same category) is computed; call this MatchProportion. Then
association is computed using the formula:
Association=200*abs(MatchProportion-0.5).
If the proportion of matching rows is 0.5, then the association is 0.0,
because there is a 50/50 chance of a match. If the proportion matching is
either 1.0 or -1.0 then the association is 100. A negative match proportion
means that the variables are associated in the opposite direction. A match
proportion of (-1.0) means that the category values are exactly opposite. A
match proportion of -1.0 is as good as +1.0, because the predictor value can
be imputed by reversing the category value of the surrogate. If the primary
predictor is categorical and the surrogate is continuous, a function is
fitted to the 0/1 predictor values and a threshold of 0.5 is used to convert
the value computed by the function to the predictor category value.
When a predictor variable is encountered with a missing value, DTREG
examines each of the associated surrogate variables looking for one that has
a non-missing value on that data row. The surrogates are examined in the
order of decreasing association values. When a surrogate variable is found
with a non-missing value, the surrogate function is used to compute the
replacement value for the variable. If all surrogates have missing values,
the median/mode is used replace the missing value.
Surrogate variables are used (1) during the model building process, (2) when
using the Score function to predict values for a data file, and (3) when the
DTREG COM library is used to predict values. If the Translate procedure is
used to generate C source code for a model, surrogate variable calculations
are included in the generated source code.
Surrogate variables may be used for all of the model types DTREG can build:
SVM, MLP, Probabilistic neural networks, RBF networks, GMDH polynomial
networks, Cascade correlation, Discriminant analysis, Gene expression
programming, Logistic regression, K-Means clustering, and Linear regression.
--- Surrogate Splitters ---
A "Surrogate Splitter" is similar to a surrogate variable, but it is
specialized for decision tree based models � Single Trees, TreeBoost, and
Decision Tree Forests.
When a decision tree is created, each predictor variable is evaluated at
each split point to determine how well it can partition the values. After
the best predictor has been determined, other candidate predictors are
examined and the splits generated by them are compared with the primary
split. The association is computed by comparing the split generated by the
predictor with the primary predictor. The best surrogate splitters are
stored along with the primary splitter. If the primary splitter value is
missing, surrogate splits are examined looking for a non-missing value on a
surrogate predictor.
One of the key differences between surrogate variables and surrogate
splitters is that a different set of surrogate splitters is stored for each
split. So the same predictor may have different surrogate splitter
variables at different spit points in the decision tree. In contrast,
surrogate variables are computed once before the model building process
begins, and the same set of surrogate variables is always used for a
particular predictor variable.
For additional information see http://www.dtreg.com/MissingValues.htm
--
Phil Sherrod
http://www.dtreg.com -- Neural networks, SVM, Decision trees
Ray Koopman
wrote:
Sounds a lot like a scheme I used back in the mainframe days. It was
designed for replacing missing interval-level data, and used linear
regression; for each missing observation, it chose from among the
mean, the best single surrogate for which data was available, and the
best pair of surrogates for which data were available, where 'best'
meant minimum expected squared error for that particular prediction.
Thus it was sensitive to the number of cases on which each regression
equations was based, the general goodness of each regression, and
the 1- or 2-dimensional Mahalanobis distance from their centroids of
the surrogate values on which the predictions were based. It worked
reasonably well, but I was never totally totally at peace with the
discontinuities that sometimes occurred when small changes in the
data led to different surrogates being used, with corresponding
large changes in the estimates of the missing values.
Phil Sherrod
On 16-Sep-2009, Ray Koopman <koo...@sfu.ca> wrote:
> Sounds a lot like a scheme I used back in the mainframe days. It was
> designed for replacing missing interval-level data, and used linear
> regression; for each missing observation, it chose from among the
> mean, the best single surrogate for which data was available, and the
> best pair of surrogates for which data were available, where 'best'
> meant minimum expected squared error for that particular prediction.
> Thus it was sensitive to the number of cases on which each regression
> equations was based, the general goodness of each regression, and
> the 1- or 2-dimensional Mahalanobis distance from their centroids of
> the surrogate values on which the predictions were based. It worked
> reasonably well, but I was never totally totally at peace with the
> discontinuities that sometimes occurred when small changes in the
> data led to different surrogates being used, with corresponding
> large changes in the estimates of the missing values.
Yes, that's the idea of surrogate variables.
A lot of my clients have "uncontrolled" data collected from surveys where
missing values are scattered throughout it. Others have medical data where
some tests or observations are performed on some of the patients but not
all. So I was trying to come up with something more accurate than just
using the median for the missing value. After I fit a surrogate function, I
compare its predicted values with just using the median/mode value and only
use the surrogate if it is an improvement over the median/mode.
In principle, you could use more shophisticated surrogate functions than
linear, quadratic, and cubic: You could have neural nets, SVM, etc. But
since it has to generate and test n*(n-1) functions for n variables, I
decided to stick with simple surrogate functions feeding into a more complex
main function.
--
Phil Sherrod
http://www.dtreg.com -- Neural networks, SVM, Decision trees
ddd
Maybe something other is to look at the work of Marco Ramoni and Paola
Sebastiani: "The use of exogeneous Knowledge to Learn Bayesian Networks
from Incomplete Databases."
It introduces a method called Bound and Collapse to estimate missing values.