multiple testing problem

[Troels Ring]

Dear friends - an important paper was just published on a serious side
effect to a drug occurring in renal patients and in addition compared
cases to controls with regard to 7 biochemical measurements. Besides
just one which was different and higher in those with the side effect (P
= 0.01) all the other measurements came out with P values between 0.49
and 0.94. However, the 0.01 was corrected with Bonferroni and
significance vanished. The found (and then gone..) effect was clinically
very important. Andrew Gelman has a reference (Why we (usually) don't
have to worry about multiple comparisons) easily googled and downloaded
in which he argues that the problem tend to vanish from a Bayesian point
of perspective. I do not know about that but in the current context it
seems a pitty that an important effect might be lost just because of
many effects were examined. I do not have the data and hence cannot
apply a less conservative multiple comparison correction. The authors do
report the uncorrected P value and then discuss the finding as a trend
towards, hence showing they are uneasy about the issue. What is your advice?
Best wishes
Troels Ring, Aalborg, Denmark

Jeg bruger BullGuard til at holde min computer ren.
Pr�v BullGuard gratis: www.bullguard.com

[Thompson,Paul]

Can you provide the reference?
________________________________________
From: meds...@googlegroups.com [meds...@googlegroups.com] on behalf of Troels Ring [tr...@gvdnet.dk]
Sent: Saturday, October 15, 2011 5:53 PM
To: meds...@googlegroups.com
Subject: {MEDSTATS} multiple testing problem

Prøv BullGuard gratis: www.bullguard.com

--
To post a new thread to MedStats, send email to MedS...@googlegroups.com .
MedStats' home page is http://groups.google.com/group/MedStats .
Rules: http://groups.google.com/group/MedStats/web/medstats-rules

-----------------------------------------------------------------------
Confidentiality Notice: This e-mail message, including any attachments,
is for the sole use of the intended recipient(s) and may contain
privileged and confidential information. Any unauthorized review, use,
disclosure or distribution is prohibited. If you are not the intended
recipient, please contact the sender by reply e-mail and destroy
all copies of the original message.

[John Whittington]

At 00:53 16/10/2011 +0200, Troels Ring wrote:
>Dear friends - an important paper was just published on a serious side
>effect to a drug occurring in renal patients and in addition compared
>cases to controls with regard to 7 biochemical measurements. Besides just
>one which was different and higher in those with the side effect (P =
>0.01) all the other measurements came out with P values between 0.49 and
>0.94. However, the 0.01 was corrected with Bonferroni and significance
>vanished. The found (and then gone..) effect was clinically very
>important. Andrew Gelman has a reference (Why we (usually) don't have to
>worry about multiple comparisons) easily googled and downloaded in which
>he argues that the problem tend to vanish from a Bayesian point of
>perspective. I do not know about that but in the current context it seems
>a pitty that an important effect might be lost just because of many
>effects were examined. I do not have the data and hence cannot apply a
>less conservative multiple comparison correction. The authors do report
>the uncorrected P value and then discuss the finding as a trend towards,
>hence showing they are uneasy about the issue. What is your advice?

It is only a couple of days since this forum saw a number of messages
graphically describing the folly of p-values, and I think that is very
relevant here. I think the problem is not so much the matter of p-value
corrections for multiple testing but, rather, the fact that p-values are
being used at all!

In terms of drug safety/side effects, when laboratory tests are looked at,
one does not commonly look at summary statistics (mean, median etc.), let
alone p-values - it is very common for very large changes in biochemical
measurements (of considerable clinical importance) to be so rare that their
effect on mean results is hardly noticeable, and 'statistical significance'
unattainable without impracticably large sample sizes. Far more important
are the magnitude of the abnormalities and the proportion of patients in
whom they occur.

In the situation you describe, I certainly wouldn't undertake any
corrections, and most certainly would not use corrected p-values to say
that the abnormality was 'not significant'. Rather, as you say the authors
have also done, if I were presenting p-values at all (which I probably
wouldn't) I would present the uncorrected p-values and discuss them, with
some mention of the 'multiple testing' issue. In fact, the results you
describe are much easier to deal with than is often the case - whether one
corrects the p-values or not, we have a situation in which the p-value for
one biochemical test is 50-100 times smaller than that for the other 6
measurements - which is a very good reason to single that one measurement
out for discussion and consideration.

That's how I see it,anyway!

Kindest Regards,

John

----------------------------------------------------------------
Dr John Whittington, Voice: +44 (0) 1296 730225
Mediscience Services Fax: +44 (0) 1296 738893
Twyford Manor, Twyford, E-mail: Joh...@mediscience.co.uk
Buckingham MK18 4EL, UK
----------------------------------------------------------------

[Troels Ring]

Dear friends - thanks a lot for rapid answers - and answers I like (:-)
:-) ). Here was the Gelman paper mentioned
http://www.stat.columbia.edu/~gelman/research/unpublished/multiple2.pdf
All the best
Troels

Jeg bruger BullGuard til at holde min computer ren.

[Bjoern]

Without knowing more details the reaction of the authors seems quite
reasonable and laudably honest.

They looked at several things and thus, quite rightly express that the
finding could have occurred just due to chance. On the other hand,
they also do not totally ignore the finding, for which they may also
simply not have evidence due to the size of the dataset.

There are a number of publications recently that suggest that Bayesian
Hierarchical Models may be a good approach for systematic safety
signal detection (see also e.g. Xia HA, Ma H, Carlin BP: Bayesian
hierarchical modeling for detecting safety signals in clinical trials.
J Biopharm Stat 2011 Sep; 21(5):1006-29.), but that would not be a
rationale for stopping to worry about multiplicity when one has done a
frequentist analysis in the first place.

Best Regards,
Björn

[Thompson,Paul]

Can you also provide the reference to the medical paper with the serious side effect?

________________________________________
From: meds...@googlegroups.com [meds...@googlegroups.com] on behalf of Troels Ring [tr...@gvdnet.dk]

Sent: Sunday, October 16, 2011 2:41 AM
To: meds...@googlegroups.com
Subject: Re: {MEDSTATS} multiple testing problem

[Bruce Weaver]

For a slightly different view, look at Craig Bennett's interesting
fMRI study carried out on a dead Atlantic salmon.

http://www.jsur.org/ar/jsur_ben102010.pdf
http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf
http://prefrontal.org/blog/2009/09/the-story-behind-the-atlantic-salmon/

Cheers,
Bruce

On Oct 16, 3:41 am, Troels Ring <tr...@gvdnet.dk> wrote:
> Dear friends - thanks a lot for rapid answers - and answers I like (:-)

> :-) ). Here was the Gelman paper mentionedhttp://www.stat.columbia.edu/~gelman/research/unpublished/multiple2.pdf
> All the best
> Troels
>

[Troels Ring]

Talked to the senior author, so here is the ref: Br J Dermatol 2011;
165: 828 - 836
Best wishes
Troels

> Pr�v BullGuard gratis: www.bullguard.com

>
>
> --
> To post a new thread to MedStats, send email to MedS...@googlegroups.com .
> MedStats' home page is http://groups.google.com/group/MedStats .
> Rules: http://groups.google.com/group/MedStats/web/medstats-rules
>
> -----------------------------------------------------------------------
> Confidentiality Notice: This e-mail message, including any attachments,
> is for the sole use of the intended recipient(s) and may contain
> privileged and confidential information. Any unauthorized review, use,
> disclosure or distribution is prohibited. If you are not the intended
> recipient, please contact the sender by reply e-mail and destroy
> all copies of the original message.
>

Jeg bruger BullGuard til at holde min computer ren.

[Thompson,Paul]

Thank you. I see no need to talk to an author about this. This is a published paper in a journal. It is in the public record.

________________________________________
From: meds...@googlegroups.com [meds...@googlegroups.com] on behalf of Troels Ring [tr...@gvdnet.dk]

Sent: Sunday, October 16, 2011 3:23 PM

> Prøv BullGuard gratis: www.bullguard.com

>
>
> --
> To post a new thread to MedStats, send email to MedS...@googlegroups.com .
> MedStats' home page is http://groups.google.com/group/MedStats .
> Rules: http://groups.google.com/group/MedStats/web/medstats-rules
>
> -----------------------------------------------------------------------
> Confidentiality Notice: This e-mail message, including any attachments,
> is for the sole use of the intended recipient(s) and may contain
> privileged and confidential information. Any unauthorized review, use,
> disclosure or distribution is prohibited. If you are not the intended
> recipient, please contact the sender by reply e-mail and destroy
> all copies of the original message.
>

Jeg bruger BullGuard til at holde min computer ren.

Prøv BullGuard gratis: www.bullguard.com

[Wei]

i think they should scan more salmons. this will increase the sample
size and provide the replication. I agree certain significance will
appear without multiple testing correction. But what's the chance that
a false positive right happens in brain? it seems a big fish

wei

[Wei]

I want to recommend this paper to the group.
http://onlinelibrary.wiley.com/doi/10.1111/j.1740-9713.2011.00511.x/abstract