Statistical Simulation Experiments

[Randall D. Tobias]

I've been looking at a lot of statistical papers reporting simulation
experiments lately and I must say that I'm pretty appalled by
statisticians' disregard of basic design and analysis principles when
it comes to their own experiments. For example, one reference has a
table of 120 empirical p-values, each based on 1000 replications; the
nominal level is 0.05 and the authors flag as unacceptable any p-value
that is bigger than 0.06. But the standard error for the numbers is
about 0.007: you're going to see 8 or 9 p-values bigger than 0.06 on
the basis of chance alone! Questions of multiple inference and power
were not considered at all.

Is there a good source that discusses the design and analysis of
statistical simulations, and especially how to compute and interpret a
relatively large number of empirical p-values? Something that
includes recommendations on sample size and randomization method---ie.
whether or not to use the same random seeds for all experiments--would
be nice.

--
Randy Tobias SAS Institute Inc. sas...@unx.sas.com
(919) 677-8000 x7933 SAS Campus Dr. 72450...@compuserve.com
(919) 677-8123 (Fax) Cary, NC 27513-2414

Faith, faith is an island in the setting sun.
But proof, yes: proof is the bottom line for everyone.
-- Paul Simon