Standardizing MSE - is it reasonable?

[Pedro Emmanuel Alvarenga Americano do Brasil]

Dear stat masters,

I come seeking for light.

My researhc mates and I are conducting a simulation study where we intend to estimate bias of  some estimators and compare this bias among all the estimators. For this purpose we simulated several populations with  a variety of formats, including normal, binormal, gama, and some mixed distribuitions.

The following point came up. To reach the disired populations formats it was necessary to adopt different parameter scales ranges. So in some population it may range from 20 to 400 and in others from 0 to 2000. We believe, although we did not test it, that tthe MSE will have larger variation in the larger scale ranges and thus we tought a procedure to standardize the scales. However, we believe that in standardizing the parameter scale, the population format may substantially change. Therefore the second idea was to standardize the MSE by dividing it by the parameter measure of that population, following the idea of the variance coefficient. However, im  not sure if this pocedure for standardization is the most appropriate and it will allow us to reach a desired conclusion.

What the masters would do in my place?

Pedro Brasil
via Android (:)=

[Steve Simon, P.Mean Consulting]

If you had collected this data on a group of animals or humans, you
would obviously build a regression model. Because it is a simulation,
you do not consider such a possibility. Why not? Surely you could see if
the scale range predicts variation, holding the other variables (such as
distribution and estimator) constant. Then you could make an adjustment,
such as standardizing, and see if the scale range predicts variation
after the adjustment.

It's such an easy thing to do, but no one ever does it. Instead, you see
these dense tables and impenetrable graphs. And people make choices like
standardizing the MSE without looking at the data first to see if the
data supports such a choice. I don't want to sound like I'm picking on
you. It is a problem with almost all published simulation studies.

I heard someone comment many years ago about this phenomenon of not
subjecting simulation studies to a rigorous statistical analysis as
being a tendency for statisticians to use complex models for everybody
else's data but their own.

Steve Simon, n...@pmean.com, Standard Disclaimer.
Sign up for the Monthly Mean, the newsletter that
dares to call itself average at www.pmean.com/news

[Pedro Emmanuel Alvarenga Americano do Brasil]

Steve,

Is interesting you put the problem this way. I thruth, at the very first draft of this project, we intended to run an analysis with regression taking the bias (here represented by the MSE) as the outcome and check which of the studied conditions (estimators, outcome prevalence, sample size, population format, the parameter accuracy etc) may adjust the bias of the estimators and then have an idea of what estimator is less bias or in what condition there are differences in bias of the estimators. However Im not sure which model will be conducted, ols, hierarchical etc.

The question about the scale range, is that, different from other elements, we did not anticipate that the scale range could be a problem in confusing estimator bias. If it is to be adjusted, how to better represent the scale problem into a regression or into a standardization of MSE?

Regards,

Dr. Pedro Emmanuel A. A. do Brasil

Curriculum Lattes:  http://lattes.cnpq.br/6597654894290806 
Instituto de Pesquisa Clínica Evandro Chagas
Fundação Oswaldo Cruz
Rio de Janeiro - Brasil
Av. Brasil 4365,
CEP 21040-360,
Tel 55 21 3865-9648
email: pedro....@ipec.fiocruz.br
email: emmanue...@gmail.com

---Apoio aos softwares livres
www.zotero.org - gerenciamento de referências bibliográficas.
www.broffice.org ou www.libreoffice.org - textos, planilhas ou apresentações.
www.epidata.dk - entrada de dados.
www.r-project.org - análise de dados.
www.ubuntu.com - sistema operacional

2013/2/19 Steve Simon, P.Mean Consulting <n...@pmean.com>

[Steve Simon, P.Mean Consulting]

Well, you should always use regression to analyze simulation data. I
can't say if standardardizing MSE makes sense or not. What I can say is
that if you show that if you standardize and the regression term
associated with scale range goes to zero or very close to zero, then
that is an indication that standardizing MSE has worked. If the
regression term stays about the same or only decreases by a little bit,
then standardizing MSE did not solve the problem of different scale ranges.

In some settings, you might be able to mathematically establish that
standardizing MSE makes sense. But often the math is too complicated for
most of us.

Steve Simon, n...@pmean.com, Standard Disclaimer.
Sign up for the Monthly Mean, the newsletter that
dares to call itself average at www.pmean.com/news

On 2/19/2013 9:22 PM, Pedro Emmanuel Alvarenga Americano do Brasil wrote:
> Steve,
>
> Is interesting you put the problem this way. I thruth, at the very first
> draft of this project, we intended to run an analysis with regression
> taking the bias (here represented by the MSE) as the outcome and check
> which of the studied conditions (estimators, outcome prevalence, sample
> size, population format, the parameter accuracy etc) may adjust the bias
> of the estimators and then have an idea of what estimator is less bias
> or in what condition there are differences in bias of the estimators.
> However Im not sure which model will be conducted, ols, hierarchical etc.
>
> The question about the scale range, is that, different from other
> elements, we did not anticipate that the scale range could be a problem
> in confusing estimator bias. If it is to be adjusted, how to better
> represent the scale problem into a regression or into a standardization
> of MSE?
>
> Regards,
>
>
> Dr. Pedro Emmanuel A. A. do Brasil
> Curriculum Lattes: http://lattes.cnpq.br/6597654894290806

> Instituto de Pesquisa Cl�nica Evandro Chagas
> Funda��o Oswaldo Cruz

> Rio de Janeiro - Brasil
> Av. Brasil 4365,
> CEP 21040-360,
> Tel 55 21 3865-9648

> email: pedro....@ipec.fiocruz.br <mailto:pedro....@ipec.fiocruz.br>
> email: emmanue...@gmail.com <mailto:emmanue...@gmail.com>
>
> ---Apoio aos softwares livres
> www.zotero.org <http://www.zotero.org> - gerenciamento de refer�ncias
> bibliogr�ficas.
> www.broffice.org <http://www.broffice.org> ou www.libreoffice.org
> <http://www.libreoffice.org/> - textos, planilhas ou apresenta��es.
> www.epidata.dk <http://www.epidata.dk> - entrada de dados.
> www.r-project.org <http://www.r-project.org> - an�lise de dados.
> www.ubuntu.com <http://www.ubuntu.com> - sistema operacional

>
>
> 2013/2/19 Steve Simon, P.Mean Consulting <n...@pmean.com

> <mailto:n...@pmean.com>>

> Steve Simon, n...@pmean.com <mailto:n...@pmean.com>, Standard Disclaimer.

> Sign up for the Monthly Mean, the newsletter that
> dares to call itself average at www.pmean.com/news

> <http://www.pmean.com/news>
>
>