[ez4r] Help in interpreting ezMixed bits column, request for literature
Henrik Singmann
I am playing around with the new ezMixed function (from the development version ez 4) but have problems interpreting the output. Specifically I would like to know on how to interpret the bits column. Is there any reference or any paper you know that either recommends or used this approach?
I am asking because I am unsure on how to interpret the following output. I would like that there is no effect of phase at all, but the main effect shows an effect of 1.8 bits in direction of the unrestricted model (all other effects of phase are luckily negative).
Does this already suffice to assume that there is an effect? Or better, should everything above 0 be treated as evidence that this effect is significant?
The output:
> pr.know1.mixed[["summary"]]
effect errors warnings bits
1 condition FALSE FALSE 9.335
2 phase FALSE FALSE 1.815
3 problem FALSE FALSE 88.471
4 topic FALSE FALSE 938.539
5 condition:phase FALSE FALSE -4.441
6 condition:problem FALSE FALSE 38.230
7 condition:topic FALSE FALSE -0.613
8 phase:problem FALSE FALSE -4.062
9 phase:topic FALSE FALSE -2.310
10 problem:topic FALSE FALSE 2268.776
11 condition:phase:problem FALSE FALSE -16.989
12 condition:phase:topic FALSE FALSE -15.534
13 condition:problem:topic FALSE FALSE 9.119
14 phase:problem:topic FALSE FALSE -10.136
15 condition:phase:problem:topic FALSE FALSE -43.761
Is there a way to obtain other information in the ezMixed output, such as the likelihood values (as in the old ez output), For example by setting detailed = TRUE or something like that?
Thanks a lot for ez,
Henrik
Mike Lawrence
First, apologies for the delayed response, I've been out ill this week.
The bits column contains the AIC-corrected likelihood ratio converted
to the log-base-2 scale. On this scale, positive evidence signifies
data that favour the effect while negative evidence signifies data
that favor the null. The absolute value of the bits reflects the
strength of evidence (zero bits reflects perfect ambiguity). There is
no "critical bits value" at which one can objectively claim an effect
is real (or non-real) because this is not within the scope of the
purpose of likelihood ratios. To put it succinctly (and to paraphrase
http://philosophy.wisc.edu/forster/Likelihood/Why%20Likelihood.pdf):
1. Frequentist Significance Testing will tell you how to make a
decision (while keeping one type of decision error at a fixed rate
across multiple decisions) given a set of data (ignoring data/beliefs
obtained elsewhere).
2. Bayesian methods will tell you how to update your beliefs in the
face of new data (presuming you can operationalize your prior beliefs
quantitatively)
3. Likelihood methods will tell you to what degree your data favor a
given model (ignoring data/beliefs obtained elsewhere).
I argue that for the purposes of scientific reporting, it's
inefficient to represent the data as a decision (as in a significance
test) and infeasible to account for the priors of all possible readers
(though, maybe someday with interactive e-publications...). Thus the
likelihood-based approach is the only way to go.
Granted, it's rather inevitable that, given a likelihood-based
representation of evidence from a given data set, most
researchers/readers will want to make decisions or update their
beliefs. Mapping the likelihood ratio to a significance test is
relatively easy; my simulations (of an admittedly very simple
scenario) show that the analog of an alpha of .05 type 1 error rate is
achieved by using a criterion of 3 bits (which happens to match quite
nicely with the recommendations that I've seen in the past but whose
citations escape my googling at the moment). Using a likelihood ratio
for Bayesian belief updating is a little trickier because even though
the Bayes Factor may look like a likelihood ratio, apparently it
represents the relative likelihood integrated over all possible
parameter values of each model, not simply the MLE values as achieved
by the standard likelihood ratio. However, I wonder if a BIC-corrected
likelihood ratio achieves the same end? I'm not sufficiently
mathematically informed to say.
Mike
Thom
I'd prefer to see just the change in AIC (dAIC). You can convert this
to
an AIC-corrected LR via exp(dAIC/2). Also, what is the null model
here?
One advantage likelihood approaches is to move away from sharp cut-
offs
towards strength of evidence.
Your argument about Bayesian stats assumes a subjective prior
approach - whereas an objective prior approach seems more common in
psychology. BIC approximates a Bayes factor with a unit information
prior
under certain circumstances (e.g., see Wasserman, 2004). Under the
objective Bayes approach you are adding in fairly weak assumptions
about
the effect (e.g., that most effects are small-ish). It is worth
looking at
Jeff Rouder and colleagues work on Bayesian t tests.
Thom
On Mar 2, 5:58 am, Mike Lawrence <Mike.Lawre...@dal.ca> wrote:
> Hi Henrik,
>
> First, apologies for the delayed response, I've been out ill this week.
>
> The bits column contains the AIC-corrected likelihood ratio converted
> to the log-base-2 scale. On this scale, positive evidence signifies
> data that favour the effect while negative evidence signifies data
> that favor the null. The absolute value of the bits reflects the
> strength of evidence (zero bits reflects perfect ambiguity). There is
> no "critical bits value" at which one can objectively claim an effect
> is real (or non-real) because this is not within the scope of the
>
> - Show quoted text -
Mike Lawrence
> I'd prefer to see just the change in AIC (dAIC). You can convert this
> to
> an AIC-corrected LR via exp(dAIC/2).
I believe you have something awry in your derivation of that
conversion; dAIC is simply the log-base-e likelihood ratio, so the
conversion between dAIC and bits should simply involve log2(exp(1))
> Also, what is the null model
> here?
In the context of ezMixed, the null ("restricted") model always
includes all effects included in the unrestricted model except the
effect of interest (equivalently, the effect of interest is restricted
to zero in the "restricted model"). So, for a main effect:
restricted = grandMean
unrestricted = grandMean + Effect
And for a 2-way interaction:
restricted = grandMean + Effect1 + Effect2
unrestricted = grandMean + Effect1 + Effect2 + Effect1:Effect2
> One advantage likelihood approaches is to move away from sharp cut-
> offs
> towards strength of evidence.
Hear, hear!
> Your argument about Bayesian stats assumes a subjective prior
> approach - whereas an objective prior approach seems more common in
> psychology. BIC approximates a Bayes factor with a unit information
> prior
> under certain circumstances (e.g., see Wasserman, 2004). Under the
> objective Bayes approach you are adding in fairly weak assumptions
> about
> the effect (e.g., that most effects are small-ish). It is worth
> looking at
> Jeff Rouder and colleagues work on Bayesian t tests.
I have seen some of this work, and will take another look, but I doubt
they've solved the inherent subjectivity of the process of quantifying
the prior; even with meta-analytic data available for specifying
priors, my "true" prior may differ from yours if I have different
degrees of trust in each of the reports supplying meta-analytic data,
etc. If it's simply a matter of making the prior "weaker", then it
strikes me as simply moving toward representing the data by its bayes
factor, which as I noted seems awfully similar in function to a
complexity corrected likelihood ratio and, personally, I prefer the
"prediction-optimized' correction provided by AIC over BIC's
correction.
Henrik Singmann
thanks to both of you for the detailed answer. I still have two questions/opinions that I would like to add to the debate (although they seems kind of disconnected to the previous mail).
First, I don't quite understand why you actually decided to report AIC based comparisons when it would be able to use likelihood ratio (i.e., Chi-square) based null-hypothesis tests to indicate if the restricted or unrestricted model is more adequate. As I understand the summary table the results do not permit comparisons between rows but only within. Or am I misunderstanding the table insofar as one can indeed assume that the row with the highest positive bit value reflects the model that describes the data most adequately?
I know that the restricted model has the issue of restricting one parameter to the boundary of the parameter space and the simple chi-square distribution is not the "correct" distribution to perform a likelihood ratio test. But if this is the issue I would gladly help in developing an algorithm to obtain the correct mixture chi-square distribution (we have done so in a recent paper).
Second, I miss the possibility to allow for conditional or grouped random effects. As I understand ezMixed, it simply takes a random term for the random variable, (1|random), and adds the concatenation of all fixed effects by + or : (e.g. the main effect for cue in example 1 in ?ezMixed is "rt ~ (1|subnum) + cue").
This way to specify a mixed model somehow contrasts with the way a model is specified in the "reference" paper from Baayen, Davidson, & Bates (2008, JML) in which the random effects for subject is conditionalized on the within-subject effect (e.g., "rt ~ (1 + cue|subnum) + cue" in the above example). Perhaps one could have two options to specify a fixed effect. One that is added to the random effect as done by Baayen et al. (e.g., the within-subjects variables), and one that is not (as is done so far).
Best,
Henrik
Henrik Singmann
after rereading your first answer to my initial mail you, Mike, already answered my first question of the previous mail: Because likelihood ratios seem to you to be more valuable as a scientific tool than binary significant/not significant null hypothesis significance test (NHST) decisions. However, I think I am not the only one to be still interested in using NHSTs, because sometimes you want binary decisions (did my experiment effect this parameter or not). Perhaps it would be cool to have both options.
Furthermore, when everything is about evidence strength the question of between row comparisons (i.e. global model selection) always comes to my mind. Can you say anything regarding this (i.e., How does ezMixed help you in decide which mode best describes your data overall?).
The second issue of my previous mail was, so far, not effected by rereading.
Sorry for "flooding" the mailing list,
Henrik
Mike Lawrence
> First, I don't quite understand why you actually decided to report AIC based
> comparisons when it would be able to use likelihood ratio (i.e., Chi-square)
> based null-hypothesis tests to indicate if the restricted or unrestricted
> model is more adequate.
Applying NHST to mixed effects modelling is apparently a rather tricky
endeavour. I haven't bothered to catch up on why this is, precisely,
because as a scientist (and not a decision maker) I prefer reporting
and evaluating evidence via likelihood ratios. I do have this quote
from Pinero & Bates (2000 page 87-88) which speaks to the
inadvisability of attempting LRTs on models with different fixed
effects (as are computed by ezMixed):
"Even though a likelihood ratio test for the ML fits of models with
different fixed effects can be calculated, we do not recommend using
such tests. Such likelihood ratio tests using the standard χ2
reference distribution tend to be "anticonservative"--sometimes quite
badly so."
Now, maybe they're not talking about the special case of comparing
nested models, where one model simply restricts one parameter to zero
that the other model leaves unrestricted. Certainly the prevalence of
LRTs in such cases I've seen lately might suggest that a few people
out there think it's ok (or don't realize their folly?).
> As I understand the summary table the results do not
> permit comparisons between rows but only within. Or am I misunderstanding
> the table insofar as one can indeed assume that the row with the highest
> positive bit value reflects the model that describes the data most
> adequately?
Each row represents the strength of evidence for that row's named
effect. Comparison of the values between rows speaks to differences
between effects in their strength of evidence. I'd say that you'd have
to have a very nuanced theory in order for those sorts of comparisons
to be meaningful.
> I know that the restricted model has the issue of restricting one parameter
> to the boundary of the parameter space and the simple chi-square
> distribution is not the "correct" distribution to perform a likelihood ratio
> test. But if this is the issue I would gladly help in developing an
> algorithm to obtain the correct mixture chi-square distribution (we have
> done so in a recent paper).
Actually, in the case of fixed effects, the restricted model is not
restricting the parameter to a boundary; the space is -Inf to Inf, so
0 is precisely in the center. I think you're thinking about random
effects, where a model that excludes a random effect indeed implicitly
restricts its variance estimate to zero, which is the lower bound for
variances. This is why, to my knowledge, LRTs are particularly
questionable in the context of comparing models with different random
effects structures.
> Second, I miss the possibility to allow for conditional or grouped random
> effects. As I understand ezMixed, it simply takes a random term for the
> random variable, (1|random), and adds the concatenation of all fixed
> effects by + or : (e.g. the main effect for cue in example 1 in ?ezMixed is
> "rt ~ (1|subnum) + cue").
> This way to specify a mixed model somehow contrasts with the way a model is
> specified in the "reference" paper from Baayen, Davidson, & Bates (2008,
> JML) in which the random effects for subject is conditionalized on the
> within-subject effect (e.g., "rt ~ (1 + cue|subnum) + cue" in the above
> example). Perhaps one could have two options to specify a fixed effect. One
> that is added to the random effect as done by Baayen et al. (e.g., the
> within-subjects variables), and one that is not (as is done so far).
The two models you describe:
rt ~ cue + (1|subnum)
vs
rt ~ cue + (1+cue|subnum)
differ in that the former has 2 extra parameters: one representing
variance in the effect of cue across levels of subnum, and one
representing a correlation that I won't bother discussing here for
brevity's purposes (for fuller explanation:
http://stats.stackexchange.com/questions/13166/rs-lmer-cheat-sheet/13173#13173).
While it seems reasonable to permit the effect of a fixed effect to
vary across levels of a random effect (particularly in my own domain
of cognitive science, where human participants are highly variable
from one to another) I'm frankly not sure it's worth it. Specifically,
models that permit a fixed effect to vary across levels of a random
effect can take much longer to compute than models that don't, and
while I once began coding ezMixed to indeed explore these more nuanced
models I found that they quickly expanded the number of models that
needed to be fit and compared. For example, with just one fixed effect
and one random effect, you'd have:
m1 = rt ~ 0 + (1|subnum)
m2 = rt ~ cue + (1|subnum)
m3 = rt ~ cue + (1|subnum) + (0+cue|subnum)
m4 = rt ~ cue + (1+cue|subnum)
where the m1-m2 comparison would provide an estimate of evidence for a
fixed effect of cue, the m2-m3 comparison would provide an estimate of
the evidence for variance in the effect of cue across levels of
subnum, the m3-m4 comparison would provide an estimate of the evidence
for a correlation between the subnum intercepts and subnum cue
effects. But what if the m2-m3 comparison is weak while the m3-m4
comparison is strong? And when you add a second fixed effect and start
exploring interactions, you need to decide on what main effects
structures to carry-over. I realized that it was getting unwieldy to
attempt to make automated so I gave up. I also ran some simulations
and found that if you're only interested in evaluating evidence for a
fixed effect, failing to account for its possible variance across
levels of the random effect doesn't actually hurt too much (the
obtained LRs aren't that much smaller than those obtained when there
isn't variance there to be accounted for). So I decided to leave
ezMixed as a tool for evaluating fixed effects and leave the
exploration of individual differences to manual computation for those
that know what they're doing.
Thom
> > I'd prefer to see just the change in AIC (dAIC). You can convert this
> > to
> > an AIC-corrected LR via exp(dAIC/2).
>
> I believe you have something awry in your derivation of that
> conversion; dAIC is simply the log-base-e likelihood ratio, so the
> conversion between dAIC and bits should simply involve log2(exp(1))
other
transformations of AIC (LR, bits).
> > Also, what is the null model
> > here?
>
> In the context of ezMixed, the null ("restricted") model always
> includes all effects included in the unrestricted model except the
> effect of interest (equivalently, the effect of interest is restricted
> to zero in the "restricted model"). So, for a main effect:
>
> restricted = grandMean
> unrestricted = grandMean + Effect
>
> And for a 2-way interaction:
>
> restricted = grandMean + Effect1 + Effect2
> unrestricted = grandMean + Effect1 + Effect2 + Effect1:Effect2
> I have seen some of this work, and will take another look, but I doubt
> they've solved the inherent subjectivity of the process of quantifying
> the prior; even with meta-analytic data available for specifying
> priors, my "true" prior may differ from yours if I have different
> degrees of trust in each of the reports supplying meta-analytic data,
> etc. If it's simply a matter of making the prior "weaker", then it
> strikes me as simply moving toward representing the data by its bayes
> factor, which as I noted seems awfully similar in function to a
> complexity corrected likelihood ratio and, personally, I prefer the
> "prediction-optimized' correction provided by AIC over BIC's
> correction.
sometimes an
approximation to one). I think the objective Bayes approach has real
promise and isn't about solving the subjectivity problem (which arises
in
all statistical analyses but in different guises) - it is about
building in realistic constraints to a model. Sometimes you don't need
to do that, but other times you do to get sensible answers out.
Thom
getting accurate p values extend also to AIC and to obtaining
interval estimates.
I cover this (briefly) in my forthcoming book (in the last chapter)
which
also uses the ez package in a couple of places.
http://seriousstats.wordpress.com/2012/02/01/intro/
Two problems are bundled up here - one is the boundary effect (the
estimating difficulty of estimating parameters such as variances
that are bounded when they are close a boundary). The other is
establishing the effective number of parameters and hence df for
tests.
My rough summary:
i) Sometimes effects are clear cut and there is sufficient data that
the standard approaches are good enough (e.g., large t statistics or
very large differences in AIC)
ii) In certain cases the model reduces to the equivalent of a repeated
measures ANOVA and the standard F ratios etc. work fine (if
assumptions such as sphericity hold). Even models that relax these
assumptions (e.g., different covariance structures in nlme) probably
behave
quite well for testing fixed effects.
iii) For other situations it may be necessary to use alternative
methods such
as parametric bootstrapping or HPD intervals. This is not too tricky
for
simple cases (e.g., using MCMCglmm).
Thom