Interpretation of standard errors of Log transformed data
michael roughton
the standard errors of log transformed variables? A colleage of mine
has values taken at 3 time points (baseline, 6 and 12 months) of some
blood measurements, some of which are highly skewed. He wants to plot
the mean value for the 32 patients at the time points, along with std
error bars. The raw and transformed data are given below (n=32)
Raw measure
mean: 2123
SD: 1737
SE: 307
Log-transformed
mean: 7.36
SD: 0.14
SE: 0.82
II know (I think!) that after anti-logging the SD the interpretation is
the average multiplicative distance by which measurements differ from
the geometric mean.
But I am unsure as to the interpretation of the SE. And thus, unsure as
to which values the errod bars should be calculated from.
If anyone could help it would be much appreciated.
Regards,
Michael
Bland, M.
confidence interval. You can calculate this on the log scale then back
transform.
Martin
michael roughton wrote:
--
***************************************************
J. Martin Bland
Prof. of Health Statistics
Dept. of Health Sciences
Seebohm Rowntree Building Area 2
University of York
Heslington
York YO10 5DD
Email: mb...@york.ac.uk
Phone: 01904 321334
Fax: 01904 321382
Web site: http://www-users.york.ac.uk/~mb55/
***************************************************
Frederic Villamayor Forcada
Michael
Standard errors are the same thing in original variables and transformed variables: The standard deviations of the mean, taken as a random variable itself. Based on them, you can compute the confidence interval for the mean. Since your original distribution is biased, and if log-transformed data does better adjust to it, then compute the confidence interval for the transformed data, and back transform the boundaries to obtain an asimetric confidence interval for the original mean. Never plot the Standard Error of the mean, if this what you intended to do. It is a meassure of the accuracy of the estimation of the true population mean by the sample mean. Better plot the confidence interval boundaries. Since standard error depend on the sample size, plotting them does not offer any useful information but how precise is your estimation. Back-transforming of the SE values alters this information. When bars around the mean are plotted, one expects to find how likely is the true mean to be found inside the interval. This is the concept behind confidence intervals, no matter if they are simmetric or no, since back-transforming the boundaries does not alter it.
HTH
Frederic
%%%%%%%%%%%%%%%%%%%%%%%
Frederic Villamayor
Unitat de Bioestadística
Àrea de Desenvolupament Preclínic
CIDF Ferrer Grupo
Juan de Sada, 32
08028 - Barcelona
Espanya
E-mail: fvillamayo...@ferrergrupo.com
Tel: +34 935093236
Fax: +34 934112764
WWW: www.ferrergrupo.com
%%%%%%%%%%%%%%%%%%%%%%%
"Sanity is not statistical"
1984 (George Orwell)
| "michael roughton"
<mike_j_...@hotmail.com>
Enviado por: MedS...@googlegroups.com 12/06/2006 14:36
|
|
Ted Harding
> Michael
>
> Standard errors are the same thing in original variables and
> transformed variables: The standard deviations of the mean,
> taken as a random variable itself. Based on them, you can
> compute the confidence interval for the mean. Since your original
> distribution is biased, and if log-transformed data does better
> adjust to it, then compute the confidence interval for the
> transformed data, and back transform the boundaries to obtain an
> asimetric confidence interval for the original mean. Never plot
> the Standard Error of the mean, if this what you intended to do.
> It is a meassure of the accuracy of the estimation of the true
> population mean by the sample mean. Better plot the confidence
> interval boundaries. Since standard error depend on the sample size,
> plotting them does not offer any useful information but how precise
> is your estimation.
> Back-transforming of the SE values alters this information. When
> bars around the mean are plotted, one expects to find how likely
> is the true mean to be found inside the interval. This is the
> concept behind confidence intervals, no matter if they are simmetric
> or no, since back-transforming the boundaries does not alter it.
Sorry, but the above is seriously misleading; possibly for language
reasons, but misleading none the less.
Consider the situation where a logarithmic transformation
Y = log(X)
turns a skew distribution of X into a distribution for Y which
is (near enough) Normal. We could describe X as being log-normally
distributed.
The sample mean of Y, +/- a multiple of the SD of Y, is then a
confidence interval for the mean of Y.
You can then transform all three numbers back to the X scale:
exp(mean(X)), exp(mean(X) +/- k*SD)
Then the two numbers exp(mean(X) +/- k*SD) are the limits of
a confidence interval for exp(the population mean of Y). This
is because the relationship is unaltered by any monotonic
transformation.
However, exp(the population mean of Y) is not the same as
the population mean of X. Conversely, log(the population
mean of X) is not the same as the population mean of log(X).
Frederic's posting above suggests that they are the same.
In fact, denoting by M the population mean of Y = log(X),
and by V the population variance of Y, we have
population mean of X = exp(M + V/2)
when Y has a true log-normal distribution (i.e. X has a
Normal distribution).
So, when you back-transform the Ymean +/- k*SD, which is
a confidence interval for the mean of Y, you do not get a
confidence interval for the mean of X.
Best wishes
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 12-Jun-06 Time: 19:30:50
------------------------------ XFMail ------------------------------
Ted Harding
XX> Normal distribution).
> when X has a true log-normal distribution (i.e. Y has a
> Normal distribution).
> So, when you back-transform the Ymean +/- k*SD, which is
> a confidence interval for the mean of Y, you do not get a
> confidence interval for the mean of X.
>
> Best wishes
> Ted.
>
> --------------------------------------------------------------------
> E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
> Fax-to-email: +44 (0)870 094 0861
> Date: 12-Jun-06 Time: 19:30:50
> ------------------------------ XFMail ------------------------------
>
>
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 12-Jun-06 Time: 19:44:52
------------------------------ XFMail ------------------------------
Frederic Villamayor Forcada
Thanks, Ted
You are right. Sorry for any inconvenience.
I confused arithmetic mean with geometric mean conceptuses. There's a Wikipedia entry about this theme.
http://en.wikipedia.org/wiki/Log-normal_distribution
I should have read it before.
I hope it will be more usefull than my own nonsense comments.
Frederic
%%%%%%%%%%%%%%%%%%%%%%%
Frederic Villamayor
Unitat de Bioestadística
Àrea de Desenvolupament Preclínic
CIDF Ferrer Grupo
Juan de Sada, 32
08028 - Barcelona
Espanya
E-mail: fvillamayo...@ferrergrupo.com
Tel: +34 935093236
Fax: +34 934112764
WWW: www.ferrergrupo.com
%%%%%%%%%%%%%%%%%%%%%%%
"Sanity is not statistical"
1984 (George Orwell)
| (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Enviado por: MedS...@googlegroups.com 12/06/2006 20:30 |
|
|
Miller IOM
Ted, wouldn't it be more helpful to say
The sample mean of Y, +/- a multiple of the SE of Y, is then a
confidence interval for the mean of Y. (i.e. SE, not SD - yes, I know
they're multiples of each other, but...)
Second, the original post quotes SD(Y) = 0.14, SE(Y) = 0.82.
Something wrong here, surely?
Ted Harding
>
> Two points here:
>
> Ted, wouldn't it be more helpful to say
>
> The sample mean of Y, +/- a multiple of the SE of Y, is then a
> confidence interval for the mean of Y. (i.e. SE, not SD - yes, I know
> they're multiples of each other, but...)
Yes, if you like! Though, as you point out, it's effectively
the same thing.
> Second, the original post quotes SD(Y) = 0.14, SE(Y) = 0.82.
>
> Something wrong here, surely?
I hadn't spotted that -- but I think the explanation is simple.
We're given that n=32, and:
Log-transformed
mean: 7.36
SD: 0.14
SE: 0.82
and now: 0.82/sqrt(32) = 0.14 (more precisely 0.144957 so it's
within a whisker of being rounded up to 0.15, but as it is it's 0.14).
So I think that all that's happened is that "SD" and "SE" are the
wrong way round. (The check is fine for SD and SE in the Raw data).
Best wishes,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 13-Jun-06 Time: 11:18:00
------------------------------ XFMail ------------------------------
Frederic Villamayor Forcada
Another useful link can be this
http://www.amstat.org/publications/jse/v13n1/olsson.html
Despite my errors, I would like to remark the fact that it is bad practice to plot "deviation lines" around the mean on a bar chart on the base of standard errors of the mean or standard deviations. Confidence intervals are better, if one computes them properly. Nevertheless, I wonder if arithmetic mean is an adequate descriptive statistic for log-normally distributed data...
Greetings
Frederic
%%%%%%%%%%%%%%%%%%%%%%%
Frederic Villamayor
Unitat de Bioestadística
Àrea de Desenvolupament Preclínic
CIDF Ferrer Grupo
Juan de Sada, 32
08028 - Barcelona
Espanya
E-mail: fvillamayo...@ferrergrupo.com
Tel: +34 935093236
Fax: +34 934112764
WWW: www.ferrergrupo.com
%%%%%%%%%%%%%%%%%%%%%%%
"Sanity is not statistical"
1984 (George Orwell)
| "Miller IOM"
<brian....@iom-world.org>
Enviado por: MedS...@googlegroups.com 13/06/2006 11:57 |
|
|
Ted Harding
> Another useful link can be this
>
> http://www.amstat.org/publications/jse/v13n1/olsson.html
>
> Despite my errors, I would like to remark the fact that it is bad
> practice to plot "deviation lines" around the mean on a bar chart
> on the base of standard errors of the mean or standard deviations.
> Confidence intervals are better, if one computes them properly.
> Nevertheless, I wonder if arithmetic mean is an adequate descriptive
> statistic for log-normally distributed data...
Hi Frederic,
Never mind the earlier confusions -- it's cleared up now!
The only comments I'd make about SD (or SE) versus confidence
interval are
a) If you use a confidence interval there's an extra parameter
to be specified, namely the confidence level (e.g. 95%, 99%).
At least the SE is what it is, and if you want a CU then
you find the factor appropriate to the confidence level and
consider the SE as "expanded" by that factor. If a 95%
confidence interval is given, and the reader wants the 99%
interval, then extra work is involved since you need to find
both factors, and then take their ratio.
b) Some people will prefer confidence intervals, especially when
comparing several groups (on lines like "groups differ if the
confidence intervals do no overlap"). Other people will prefer
to see SDs or SEs, for other purposes. In the end, what should
be used is very much a matter fo what you want to express, and
of what your readers wll want to know.
Regarding arithmetic mean for log-normal data:
1. It has the merit that it is an unbiased estimator of the
population mean, and it is simple to compute, but it is
not the best estimator to use, since it loses information.
2. Given data X1,...,Xn log-normally distributed, this means
that Y1,...,Yn (Y = log(X)) are normally distributed with
say population mean M and population variance V. The population
mean of X is exp(M + V/2).
Then the sum of Yi and the sum of Yi^2 are sufficient statistics
for M and V (equivalently, the sample mean and sample variance
of the Yi).
Since the sample (arithmetic) mean of the Yi is the log of the
geometric mean of the Xi, the geometric mean of the Xi is
an equivalent sufficient statistic to the arithmetic mean of
the Yi. But it is not so simple for the sum of Yi^2, since
Yi^2 = log(Xi)*log(Xi) = log(Xi^(log(Xi))
3. The maximum-likelihood estimates of M and V are (say)
m = arithmetic mean of log(Xi)
v = sample variance of log(Xi)
which (see (2)) are sufficient statistics for M and V.
The maximum-likelihood estimate of the population mean exp(M+V/2)
is exp(m + v/2). But this is not an unbiased estimate. But for
large samples it becomes increasingly nearly unbiased.
Hoping this helps,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 13-Jun-06 Time: 14:19:51
------------------------------ XFMail ------------------------------
John Whittington
Despite my errors, I would like to remark the fact that it is bad practice to plot "deviation lines" around the mean on a bar chart on the base of standard errors of the mean or standard deviations. Confidence intervals are better, if one computes them properly....
Whilst that is undoubtedly the correct 'party line', I'm not sure that it is necessarily quite as clear-cut as it would seem, and I find that statements/advice such as you give are often a cause of confusion to students and non-statisticians....
...I think the problem is that such statements give the impression that there is a fundamental difference between SE bars/lines and Confidence Intervals. The reality, of course, is that in a given situation, +/- 1 SE _is_ a 'confidence interval', just as is +/- 1.96 (or whatever) SE - obviously not a 95% confidence interval, but nevertheless a confidence interval (about 71% when a Normal distribution is appropriate).
I have spent more time than I care to recall trying to persuade people that they should not be obsessed with the 'magic' p=0.05 threshold of 'significance' for hypothesis tests, and I think/hope that most statisticians would agree with me. However, if when we move from inference to estimation we present just a 95% CI, that is close to being a perpetuation of the same obsession - and the only real way around it, if one is using CIs, is to present a whole family of CIs, not just the 'arbitrary' 95% one.
The crucial thing, of course, whether one presents SEs or CIs, is that those looking at them should be aware of the nature of what they are looking at, and of how to interpret it. Perhaps one merit of SEs is that the reader can then easily convert that to any CI (not necessarily 95%) that (s)he may wish, whereas the converse conversion is not quite so simple.
That's how I see it, anyway.
Kindest Regards,
Frederic Villamayor Forcada
John
Yes, the boundaries of confidence intervals are computed multiplying SE by a 'constant'. When sample sizes are big, z values from normal distribution for alpha= 0.05 or 0.01 are used as the constant. But for smaller samples (the ones I usually work with), t values from Student's t distribution must be used. In fact since t distribution converges with normal diatribution as sample size grows, its a good idea to use always Student's t distribution. If we are trying to compare two samples, with different sample sizes, in the eventuality that SE is the same for both of them, the confidence interval will be broader for the smallest sample, since t values depend on the confidence leven and on the degrees of freedom. This is the reason for not using SE as a meassure of "something" liying around the mean value, which undoubtiously refers to a meassure of how likely the true population mean will fall near the computed estimation. In my opinion, confidence intervals do have such interpretation, but SE does not. It is not a constant what multiplies to SE, since its value depends on the degrees of freedon (and, consequently, on sample size).
Please correct me again if I'm wrong. I usually have to report and to justify the methods used to non-statisticians who simply refuse to try undenstand formulae. I must then try to translate the underliying conceps to a more accessible language for them. On the counterpart, its very difficult for me to undenstand what are they talking about on their speciality subjects. Once I had to reply someone who argued the use of SE around the mean value 'because they are narrower intervals and plots look better". My answer was very similar to the one I've exposed above. I hope some of you will support it.
Frederic
%%%%%%%%%%%%%%%%%%%%%%%
Frederic Villamayor
Unitat de Bioestadística
Àrea de Desenvolupament Preclínic
CIDF Ferrer Grupo
Juan de Sada, 32
08028 - Barcelona
Espanya
E-mail: fvillamayo...@ferrergrupo.com
Tel: +34 935093236
Fax: +34 934112764
WWW: www.ferrergrupo.com
%%%%%%%%%%%%%%%%%%%%%%%
"Sanity is not statistical"
1984 (George Orwell)
| John Whittington <Joh...@mediscience.co.uk>
Enviado por: MedS...@googlegroups.com 13/06/2006 16:03 |
|
|
John Whittington
Yes, the boundaries of confidence intervals are computed multiplying SE by a 'constant'. When sample sizes are big, z values from normal distribution for alpha= 0.05 or 0.01 are used as the constant. But for smaller samples (the ones I usually work with), t values from Student's t distribution must be used. In fact since t distribution converges with normal diatribution as sample size grows, its a good idea to use always Student's t distribution. If we are trying to compare two samples, with different sample sizes, in the eventuality that SE is the same for both of them, the confidence interval will be broader for the smallest sample, since t values depend on the confidence leven and on the degrees of freedom. This is the reason for not using SE as a meassure of "something" liying around the mean value, which undoubtiously refers to a meassure of how likely the true population mean will fall near the computed estimation. In my opinion, confidence intervals do have such interpretation, but SE does not. It is not a constant what multiplies to SE, since its value depends on the degrees of freedon (and, consequently, on sample size).
In terms of the the actual computational issues, yes, the one situation in which looking at SEs can be difficult to interpret, and potentially misleading, is when one is comparing two groups with very different samples sizes, at least one of which samples is small (say <20). In all other situations, the 'multiplier' will be similar for both groups.
As I am sure you will realise, I am really taking the role of a "devil's advocate" here. Like you and many/most others, I am a strong believer in presenting CIs - which is what I generally do, and I usually am also guilty of presenting just a single, 95%, one!
However, I am aware of the fact that
Kind Regards,
John
Ted Harding
On 13-Jun-06 Frederic Villamayor Forcada wrote:
> [explanation omitted]
> Please correct me again if I'm wrong. I usually have to report
> and to justify the methods used to non-statisticians who simply
> refuse to try undenstand formulae. I must then try to translate
> the underliying conceps to a more accessible language for them.
Here, you are doing exactly what I described in an earlier reply:
"In the end, what should be used is very much a matter
of what you want to express, and of what your readers
will want to know."
Simce you have adapted your presentation, and explanations, to
your audience, without losing the information you want to express,
that's fine!
> On the counterpart, its very difficult for me to undenstand
> what are they talking about on their speciality subjects.
> Once I had to reply someone who argued the use of SE around
> the mean value 'because they are narrower intervals and plots
> look better". My answer was very similar to the one I've
> exposed above. I hope some of you will support it.
Very much so! And I admire your restraint and patience (though
it is undoubtedly illegal, in Spain, to pull out a gun).
Best wishes,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 13-Jun-06 Time: 17:32:33
------------------------------ XFMail ------------------------------
Thom
I'm surprised no one mentioned plotting 1.4 SE (which should probably
be more commonly done when comapring indepependent means).
Thom
John Sorkin
Whence comes 1.4SE? I have never come across this before. I would be
happy to learn from you.
John
John Sorkin M.D., Ph.D.
Chief, Biostatistics and Informatics
Baltimore VA Medical Center GRECC and
University of Maryland School of Medicine Claude Pepper OAIC
University of Maryland School of Medicine
Division of Gerontology
Baltimore VA Medical Center
10 North Greene Street
GRECC (BT/18/GR)
Baltimore, MD 21201-1524
410-605-7119
jso...@grecc.umaryland.edu
>>> tsba...@yahoo.com 6/16/2006 9:52 AM >>>
John Whittington
>I'm surprised no one mentioned plotting 1.4 SE (which should probably
>be more commonly done when comapring indepependent means).
Thom, do I take it that you're talking about 1.414... [sqrt(20]? If so, is
that not only going to be relevant (as a 'combined SE') when the SEs and Ns
of the two samples are the same? - and, even then, I'm not see that it is
an appropriate thing to plot around the indidividual sample means.
... not that I claim to be much of a theoretician, particularly on a hot
Friday afternoon!
Kind Regards,
John Whittington
>Thom, do I take it that you're talking about 1.414... [sqrt(20]? ...
Whoops. As most of you will probably have realised, "0" is what one gets
if one tries to type a ")" but doesn't push the shift key hard enough. I
obviously meant [sqrt(2)]
Apologies,
John Sorkin
thread with a twist.
As has been noted if x is a log normal distribution, i.e.
y<-rnorm(100,5,1) #Define a normal distribution, return 100 values,
with mean 5 SD 1
x<-exp(y) #Create a log normal distribition from the
original normal distribition
(1) The mean and standard deviation of lognormal distribition can be
used to recover the mean of the normal distribution used to generate the
lognormal distribution:
mean of y <- exp(x + (var(x)/2) )
A good deal of discussion has appeared on the last as to what should be
done with SE(x), i.e. the SE of the log normal distribtion. I think the
discussion misses an important point.
Given that the original distribution was log trasformed because the
analysist did not feel it justified to work with the non-log transformed
data why not simply report exp(mean(x)), i.e. the anti-log of the mean
of the log transformed values, and exp(mean(x)+ or -1.96*SE(x)) i.e. the
anti-log of the 95% CI of the log transformed values?
It is true that the mean(x) is not equal to the mean(y). Similarly
SE(x) is not equal to SE(y) and by extension the 95% CI for x will not
be the same as the 95% CI for y, but who cares? If there was a need to
log transform the original data it implies that statistics based on the
original non-transformed data were not good representations of central
tendency and spread. Taking the antilog of the statistics computed from
the log transformed data will provide statistics that are good
representation of central tendency and spread and will allow the reader
to easily interpret the values because the are measured on the same
scale as the original nonlog-tranformed data!
Thom
Yes - I meant root 2 SE. I agree entirely that it isn't appropriate for
individual means - but it is (arguably) the correct way to plot CIs
from independent means if you are intending to argue two group means
are different. Equal n is a fair point - but in practice similar n is
often sufficient for many purposes. There are some good references on
this (though John W obviously knows some of them), for example:
Goldstein H & Healy M J R (1995) The Graphical Presentation of a
Collection of Means. Journal of the Royal Statistical Society 581, Part
1, pp 175-177.
[Actually ... I first learned this at an MlWin workshop which Harvey
Goldstein presented at - most of the 'students' were experienced
statisticians and most were surprised at the idea of plotting 1.4 SE.
Root 2 makes more sense conceptually, but most plots don't have
sufficient precision for it to matter in practice].
It does, however, illustrate that there may be no correct way to plot a
CI that satisfies all the reasonable uses one might put the CI to. For
example if one wants a plot that simultaneously demonstrates that two
indepedent group means are different and also that one or both means
differs from a population value - it isn't straight-forward].
My feeling is that it is problematic to rely on graphical CIs as
substitutes for significance tests for this reason. That said, I think
plotting CIs is good practice for other reasons (to give an indication
of plausible effect sizes and to indicate precision of measurement and
statistical power).
Thom
John Whittington
>Yes - I meant root 2 SE. I agree entirely that it isn't appropriate for
>individual means - but it is (arguably) the correct way to plot CIs
>from independent means if you are intending to argue two group means
>are different.
If I understand correctly, then I'm not sure I can agree with that, since
it could be very misleading. I think/presume you are suggesting that one
should plot the (single) pooled SE [which you are approximating as
SE*sqrt(2)] around BOTH of the independent means, even if the truth were
that the SEs of the two means were substantially different. Indeed, if the
two means _are_ appreciably different, I'm not at all sure what you mean by
"SE*sqrt(2)". Whilst I agree that approach mirrors the hypothesis test,
in that non-overlapping CIs would then correspond roughly to a 'significant
difference' (p<0.05), I wouldn't mind betting that the great majority of
people looking at the plot would, at least subconsciously, take it to
indicate that the two means had identical SEs, and therefore possibly be
seriously misled.
If one really wants to 'mirror the hypothesis test' graphically, then
surely what one should do is plot just the DIFFERENCE (between means)
surrounded by a CI of roughly +/- 2*SEc (SEc being the combined SE) - so
that non-overlap of this CI with zero would correspond to the hypothesis
test. In that way, you would remove the serious risk that people would
assume that 'bars' around individual means actually related to the error
associated with the individual means.
>Equal n is a fair point - but in practice similar n is
>often sufficient for many purposes.
Perhaps, if the SEs are also pretty similar. However, since SDs tend to be
similar in different groups, unequal N would tend to result in unequal SEs,
thereby doubling the problem. These 'rough rules of thumb' are fine for
discussions over beers, but since it's so easy to calculate the _true_
combined SE, for give Ns and SEs, I don't see why anyone should even think
of going so far as _plotting_ these approximations.
>It does, however, illustrate that there may be no correct way to plot a
>CI that satisfies all the reasonable uses one might put the CI to. For
>example if one wants a plot that simultaneously demonstrates that two
>indepedent group means are different and also that one or both means
>differs from a population value - it isn't straight-forward].
Agreed but, as above, if one's interest is in the difference between two
group means, why not PLOT the group mean with the appropriate (calculated,
not rule-of-thumbed!) CI around it. That is then totally unambiguous, as
far as I can see.
>My feeling is that it is problematic to rely on graphical CIs as
>substitutes for significance tests for this reason.
As above, it can be done (for a given test), provided one plots the right
thing(s) and the right CI(s).
>That said, I think plotting CIs is good practice for other reasons (to
>give an indication of plausible effect sizes and to indicate precision of
>measurement and statistical power).
Agreed. CIs of means are far more straightforward than the CIs of
differences that you have been talking about, and I agree that they are
invaluable.
Kind Regards,
John Whittington
>Agreed but, as above, if one's interest is in the difference between two
>group means, why not PLOT the group mean with the appropriate (calculated,
>not rule-of-thumbed!) CI around it. That is then totally unambiguous, as
>far as I can see.
... what my brain meant my fingers to type was, of course: "...why not PLOT
the difference between means with the appropriate .....".
Apologies for any confusion I may have caused!
Thom
I think Goldstein's point was that with a collection of group means
plotting all the possible differences is not really satisfactory. I
suspect that it has its uses - but they may be specialized (e.g., in
comparing residuals for a multilevel model where each residual
represents an estimate of an individual school effect - a consumer
might want to compare their school against one or two other schools -
but the precise comparisons would vary from user to user). It is also
arguably more useful for experimental research (where N is equal and
unit-treatment additivity might be reasnable).
I suppose one could argue that any errors from plotting 1.4 SE as an
approximation when n is not exactly equal and SDs are not exactly equal
is less problematic than concluding that because two +/- 1.96 SE CIs
overlap the difference would not be statistically significant - there
is certainly some evidence that this is a surprisingly common error in
published research reporting CIs.
My own view is that precision isn't what plots are for. Plots are for
patterns and exact tests of hypotheses should be reported in other ways
(e.g., as an exact CI, test or whatever in text or table).
Thom
John Whittington
>I think Goldstein's point was that with a collection of group means
>plotting all the possible differences is not really satisfactory.
>... [snip] ....
>I suppose one could argue that any errors from plotting 1.4 SE as an
>approximation when n is not exactly equal and SDs are not exactly equal
>is less problematic than concluding that because two +/- 1.96 SE CIs
>overlap the difference would not be statistically significant
>... [snip] ....
>My own view is that precision isn't what plots are for. Plots are for
>patterns and exact tests of hypotheses should be reported in other ways
>(e.g., as an exact CI, test or whatever in text or table).
Thom, I agree totally with your last point and this is really the basis of
my 'disagreement' with you on the other issues.
Error bars and CIs plotted around means should surely be reflections of the
confidence associated with THAT mean - i.e. matters of estimation, not
inference.
This whole discussion, and the concept of "1.4*SE bars" only arises if one
is attempting to produce a visual equivalent of a hypothesis test. Like
you, I see no reason why that should be done and I remain of the view that
to do it in the manner which has been discussed (e.g. the 1.4*SE bars) is
potentially very confusing and misleading ....
.... stated very simply, my view is that when an 'error bar' or CI is
plotted around a mean, that bar/CI should relate specifically to the mean
around which it is plotted - and it seems to me to be very
meddlesome/confusing to include a bar/CI about a mean when that bar relates
to something other than just that mean (i.e. if it is based on the combined
SE of that mean and some other one).
If one approximates (assuming equal Ns and equal SEs) by plotting a +/-
1.4*SE bar around a mean, then, in terms of the mean about which one has
plotted it, one is effectively plotting an "84% CI" (or thereabouts,
83.84...%) of that mean - and I find that potentially very
confusing/misleading.
If one simply keeps estimation and inference separate, and does not try to
present hypothesis tests graphically, then I think that all these problems
go away, and clarity reigns (or should do!).
Thom
That's my preference (though it is a fairly recent conversion on my
part).
The only thing I'd add, is that I'd _settle_ for i) clear labelling of
whatever is plotted or reported, and, ii) a correlation between what is
plotted and reported and what the authors of a study are trying to
show. It is sad that so amny studies fail on those (fairly basic)
points.
Thom
John Whittington
>The only thing I'd add, is that I'd _settle_ for i) clear labelling of
>whatever is plotted or reported, and, ii) a correlation between what is
>plotted and reported and what the authors of a study are trying to
>show.
That (ii) sounds horribly to me as if it is a perpetuation of the
inappropriate muddling of estimation and inference that I've been talking
about.
'What the authors are trying to show' may well be a significant difference
between two means, but I remain strongly of the view that (despite that)
they should NOT attempt to present that hypothesis test graphically, and if
they choose to present the individual means graphically, then they should
have 'proper' (relative to the individual means) SE bars or CIs, NOT 'bars'
which relate to the hypothesis test (rather than to the individual means).
If I understand you correctly, it sounds as if what I feel is logical and
appropriate (and, indeed, what most people have always done) is a practice
that you would frown upon as being 'a lack of correlation between what is
plotted and what the authors are trying to show'.
michael roughton
someone like myself, who is fairly 'career young' its always really
useful to learn new things from people with more experience than
myself.
Thanks again,
Michael
Thom
> That (ii) sounds horribly to me as if it is a perpetuation of the
> inappropriate muddling of estimation and inference that I've been talking
> about.
>
> 'What the authors are trying to show' may well be a significant difference
> between two means, but I remain strongly of the view that (despite that)
> they should NOT attempt to present that hypothesis test graphically, and if
> they choose to present the individual means graphically, then they should
> have 'proper' (relative to the individual means) SE bars or CIs, NOT 'bars'
> which relate to the hypothesis test (rather than to the individual means).
I meant it in a broader sense. If I'm trying to argue that, say,
performance is consistently above chance on some measure it might be
useful to have a plot that shows a set of means relative to the chance
value. If I'm trying to argue for a particular pattern in a response a
plot that illustrates that pattern is useful.
> If I understand you correctly, it sounds as if what I feel is logical and
> appropriate (and, indeed, what most people have always done) is a practice
> that you would frown upon as being 'a lack of correlation between what is
> plotted and what the authors are trying to show'.
Mea culpa. That wasn't my intention. I don't think there needs to be a
1-to-1 correspondence (and indeed such a correspondence could be
dangerous). The reported summary statistics and plots need to support
inferences beyond the narrow set of hypotheses envisaged by the
authors.
However, reports of statistics and plots have a rhetorical role to play
in a paper that requires a degree of mapping between them and the broad
aims of the authors.
(As an aside my main point is far more pragmatic - too many people plot
or report what they think they are expected to report rather than what
they ought to report to support their aims or to help the reader gain
something from the paper).
Thom
Ted Harding
> [...]
> (As an aside my main point is far more pragmatic - too many people plot
> or report what they think they are expected to report rather than what
> they ought to report to support their aims or to help the reader gain
> something from the paper).
>
> Thom
Absolutely! Absobloodylutely!!!
People who find themselves coerced into reporting findings in
a form which does not express the information they want to
communicate, or passively go along with customary formats are
-- perhaps through forces beyond their control -- colluding
in misrepresentation, which is a euphemism for lying and deception.
If you have something to say, say it as it is. Or stand for
Parliament.
I could express this in different language, but I fear your
SPAM filters would then prevent the message from reaching you.
Best wishes to all,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 23-Jun-06 Time: 10:58:55
------------------------------ XFMail ------------------------------
John Whittington
>I meant it in a broader sense. If I'm trying to argue that, say,
>performance is consistently above chance on some measure it might be
>useful to have a plot that shows a set of means relative to the chance
>value. If I'm trying to argue for a particular pattern in a response a
>plot that illustrates that pattern is useful.
Fair enough. If the means are independent, then that's straightforward,
but one would presumably want to use 'proper' (i.e. 95% or whatever) CIs
(ideally a 'family' of CIs), not the "84% CI" that we've been
discussing. If the means are not independent upon one another, and the
patterns one is interested in relate to situations in which two or more are
simultaneously 'above chance', then it's obviously much more complicated,
and I seriously doubt that there is anything one could plot which would
enable all such possibilties to be 'examinable visually'.
>Mea culpa. That wasn't my intention. I don't think there needs to be a
>1-to-1 correspondence (and indeed such a correspondence could be
>dangerous). The reported summary statistics and plots need to support
>inferences beyond the narrow set of hypotheses envisaged by the
>authors.
My apologies for being repetitive, but you again seem to be talking about
plotting things 'which support inferences' - which, as you will realise by
now, is not something I think we should generally do (directly). In my
view, graphical displays should be of summary statistics/estimations, and
inference should be dealt with by undertaking the appropriate hypothesis
tests - and, in an ideal world I might envisage 'never would the twain meet'!
>However, reports of statistics and plots have a rhetorical role to play
>in a paper that requires a degree of mapping between them and the broad
>aims of the authors.
Indeed so, but (in my opinion) the plots of summary statistics/estimates
should be there to SUGGEST what other hypotheses might be worthy of
exploration (and maybe testing), NOT to present an attempt to allow people
do undertake tests of those hypotheses 'visually'.
>(As an aside my main point is far more pragmatic - too many people plot
>or report what they think they are expected to report rather than what
>they ought to report to support their aims or to help the reader gain
>something from the paper).
Here I join you and Ted in total agreement, but with recognition of one
great catch. We are only able to report (in publications) that which the
authors/referees are prepared to publish - which, unfortunately (at least
in the past) may be 'what they expect us to report' rather than what we
(and others) believe should be reported.
Thom
> That (ii) sounds horribly to me as if it is a perpetuation of the
> inappropriate muddling of estimation and inference that I've been talking
> about.
>
> 'What the authors are trying to show' may well be a significant difference
> between two means, but I remain strongly of the view that (despite that)
> they should NOT attempt to present that hypothesis test graphically, and if
> they choose to present the individual means graphically, then they should
> have 'proper' (relative to the individual means) SE bars or CIs, NOT 'bars'
> which relate to the hypothesis test (rather than to the individual means).
I meant it in a broader sense. If I'm trying to argue that, say,
performance is consistently above chance on some measure it might be
useful to have a plot that shows a set of means relative to the chance
value. If I'm trying to argue for a particular pattern in a response a
plot that illustrates that pattern is useful.
> If I understand you correctly, it sounds as if what I feel is logical and
> appropriate (and, indeed, what most people have always done) is a practice
> that you would frown upon as being 'a lack of correlation between what is
> plotted and what the authors are trying to show'.
Mea culpa. That wasn't my intention. I don't think there needs to be a
1-to-1 correspondence (and indeed such a correspondence could be
dangerous). The reported summary statistics and plots need to support
inferences beyond the narrow set of hypotheses envisaged by the
authors.
However, reports of statistics and plots have a rhetorical role to play
in a paper that requires a degree of mapping between them and the broad
aims of the authors.
(As an aside my main point is far more pragmatic - too many people plot
or report what they think they are expected to report rather than what
they ought to report to support their aims or to help the reader gain
something from the paper).
Thom
Thom
> My apologies for being repetitive, but you again seem to be talking about
> plotting things 'which support inferences' - which, as you will realise by
> now, is not something I think we should generally do (directly). In my
> view, graphical displays should be of summary statistics/estimations, and
> inference should be dealt with by undertaking the appropriate hypothesis
> tests - and, in an ideal world I might envisage 'never would the twain meet'!
I'm not necessarily in agreement. I think that plots shouldn't be used
for formal tests of hypotheses. (Although, I don't necessarily think
that formal hypothesis tests should be used for this - at least not in
all cases). I think that plots should be used to support informal
reasoning, though this role is partly rhetorical.
> >However, reports of statistics and plots have a rhetorical role to play
> >in a paper that requires a degree of mapping between them and the broad
> >aims of the authors.
>
> Indeed so, but (in my opinion) the plots of summary statistics/estimates
> should be there to SUGGEST what other hypotheses might be worthy of
> exploration (and maybe testing), NOT to present an attempt to allow people
> do undertake tests of those hypotheses 'visually'.
Agreed, though a good plot (or indeed table etc.) might eliminate the
need to make formal hypothesis tests if it is sufficiently persuasive.
I'm thinking, for example, of cases where the patterns are so clear
that multiple formal tests would obscure what's going on.
> Here I join you and Ted in total agreement, but with recognition of one
> great catch. We are only able to report (in publications) that which the
> authors/referees are prepared to publish - which, unfortunately (at least
> in the past) may be 'what they expect us to report' rather than what we
> (and others) believe should be reported.
I think that's true. I can certainly look at my early publications and
cringe at some of what I wrote and reported (partly my expecations and
partly what reviewers and editors accept or require).
With experience my attitudes have changed (and I hope my knowledge) and
I try, as an editor or reviewer I try and improve the quality of
reporting. That said, I'm reluctant be dogmatic about it. I think that
insisting on changes in reporting has negative repurcussions - not
least because people do what they are told because they have to rather
than because they want to. That has more to do with the social
psychology of persuasion than anything else. As an example, I'd cite
reporting of effect size - which many journals (e.g., in education or
psychology) now insist on. This has resulted in people's practice
changing but no increase in quality of reporting and no real
improvement in understanding. [In many fields people now report partial
eta-squared (often incorrectly labelled eta-squared )as a measure of
effect size even though it is rarely fit-for-purpose - and they usually
show no understanding of its limitations. This seems to me to be an
example of a good idea implemented in such a dogmatic way].
Thom