How to get p-values from multcompare?

[Markus Due Jakobsen]

Hi
When using Anova1 to test for differences between group, Anova 1 returns a p-value that shows whether there is a difference or not. To make a further pairwise comparison between groups you can use the multcompare-function. Is this some sort of post hoc test? It uses ‘tukey-kramer’ as default ..
How is it possible to get the p-values of each pairwise comparison?
------
A multcompare example:
strength = [82 86 79 83 84 85 86 87 74 82 78 75 76 77 79 79 77... 78 82 79];
alloy = {'st','st','st','st','st','st','st','st', ... 'al1','al1','al1','al1','al1','al1','al2','al2','al2','al2','al2','al2'};
[p,a,s] = anova1(strength,alloy);
[c,m,h,nms] = multcompare(s);
-------
Holm.m from file exchange uses Holm-Sidak procedure for multiple Student's t tests to compare differences between groups. This m-file show all comparisons with p-values:

% Test p-value alpha Comment
% 1-4 0.0013 0.0085 Reject Ho
% 1-3 0.0078 0.0102 Reject Ho
% 1-2 0.0553 0.0127 Fail to reject Ho
% 2-4 0.1254 No comparison made Ho is accepted
% 2-3 0.3563 No comparison made Ho is accepted
% 3-4 0.5606 No comparison made Ho is accepted
-----------
Is it possible to do the same with multcompare? The information lies within the graph-output, however it is not included in the command-output.

Any suggestions
Markus

[Tom Lane]

> When using Anova1 to test for differences between group, Anova 1 returns a
> p-value that shows whether there is a difference or not. To make a further
> pairwise comparison between groups you can use the multcompare-function.
> Is this some sort of post hoc test? It uses ‘tukey-kramer’ as
> default ..
> How is it possible to get the p-values of each pairwise comparison?

Markus, the p-values of individual comparisons aren't computed in
multicompare and so there's no way to get them from that function. The
function basically computes intervals around each comparison and looks for
ones that overlap zero, instead of computing p-values and looking for ones
less than 0.05.

I haven't thought much about this before. The idea behind multcompare is to
strive for an overall significance level of, say, 0.05, and test individual
differences preserving this overall level if the null hypothesis is true. I
suppose the p-values for the individual tests would be defined as the
overall p-value such that that individual difference is just significant.
Is that right?

This might be something we could consider for the future, but we don't have
it now.

-- Tom

[Markus Due Jakobsen]

Hi Tom,
Thanks for your reply. It looks like I have to try and make my own m.file to do the ‘post hoc’-test to compare pairwise differences between groups. However, I haven’t done that much statistics and I have 2 (perhaps simple) questions concerning calculations of confidence interval and graph comparison intervals in multcompare:

Question 1: How is the confidence interval calculated in multcompare? Here's the Matlab example:

strength = [82 86 79 83 84 85 86 87 74 82 ...
78 75 76 77 79 79 77 78 82 79];

alloy = {'st','st','st','st','st','st','st','st',...

'al1','al1','al1','al1','al1','al1',...

'al2','al2','al2','al2','al2','al2'};
[p,a,s] = anova1(strength,alloy);

[c,m,h,nms] = multcompare(s);
[nms num2cell(c)]
ans =
'st' [1] [2] [ 3.6064] [ 7] [10.3936]
'al1' [1] [3] [ 1.6064] [ 5] [ 8.3936]
'al2' [2] [3] [-5.6280] [-2] [ 1.6280]

In this example the confidence interval 95% for ‘st’ vs. ‘al1’ is 3.6064 to 10.3936 and 7 (mean differences) is Mean differences= mean(‘st’)=84 minus mean(‘al1’)=77.
This might be a stupid question but how is the confidence interval calculated?

As: CI=Mean differences +/- 1.96*std(M)/sqrt(n)?
Where M=[‘st’ ‘al1’] and n=length(M).

However, this calculation of CI is not the same as in the matlab example.

Question 2: How are the lengths of the comparison intervals in the multcompare-qraph calculated?

Markus

"Tom Lane" <tl...@mathworks.com> wrote in message <gf9jlj$5fh$1...@fred.mathworks.com>...

[Armin]

Hi Tom,

Getting back to this question, I was wondering if you could think of any modification in the multcompare code so that it can measure p-values for all the pairwise comparisons that it is performing. if not, do you know any other code which it can do that?

Thanks
Armin

"Tom Lane" <tl...@mathworks.com> wrote in message <gf9jlj$5fh$1...@fred.mathworks.com>...

[Beate]

Hi Tom,

I agree that it would be highly beneficial if such a modification could be done and I am rather sad that it did not happen so far because I am still an absolute MatLab-beginner and incapable of doing it myself.

Would highly appreciate it.
Beate

"Armin" wrote in message <ih57h3$jlu$1...@fred.mathworks.com>...

> Hi Tom,
>
> Getting back to this question, I was wondering if you could think of any modification in the multcompare code so that it can measure p-values for all the pairwise comparisons that it is performing. if not, do you know any other code which it can do that?
>
> Thanks
> Armin
>
>
>
> "Tom Lane" <tl...@mathworks.com> wrote in message <gf9jlj$5fh$1...@fred.mathworks.com>...
> > > When using Anova1 to test for differences between group, Anova 1 returns a
> > > p-value that shows whether there is a difference or not. To make a further
> > > pairwise comparison between groups you can use the multcompare-function.

> > > Is this some sort of post hoc test? It uses ‘tukey-kramer’ as

[Diemo Schwarz]

Has this request now be granted? In version R2015a, help multcompare says it outputs the p-values as the last column:

"The return value COMPARISON is a matrix
with one row per comparison and six columns. Columns 1-2 are the
indices of the two samples being compared. Columns 3-5 are a lower
bound, estimate, and upper bound for their difference. Column 6 is the
p-value for each individual comparison. "