How to phrase/interpret the result in a paper?

[MartinH]

Hello group,

I wonder how the results of an estimation statistics would be phrased in the text of an article.

For example, the default data for two independent groups on estimationstats.com gives this result:

“The unpaired mean difference between Control and Test is -3.35 [95.0%CI -5.59, -1.34]. “

I understand that the zero on the “Mean difference” axis is outside the 95% confidence interval and that we therefore can be rather sure that the means of the two groups really differ.

However, I struggle to put this into a sentence that could be written in a paper.

Does anyone have a suggestion or could point me to an example?

Thanks,

Martin

[Joses Ho]

What you wrote is pretty close! I would suggest simply reporting it as the website suggests; if reviewers or readers inquire, you can refer them to this classic paper by Gardner and Altman 1986. <https://www.bmj.com/content/bmj/292/6522/746.full.pdf>

From page 747 of the PDF above:
"By contrast, had zero been within the 95% confidence interval this would have indicated a non-significant result at the 5% level."

I would highly recommend, however, not dichotomizing results as "significant" or "not significant", and instead discussing what the magnitude and precision (or confidence interval size) of the effect size might imply

[MartinH]

Thank you for this explanantion, Joses.

I hope you don´t mind if I continue asking ...

So, if you were a reviewer of a paper describing e.g. the above example + a second example for comparison (-3.47 [95.0%CI -5.21, -1.85]), you would be satisfied with something along the following lines?

" The unpaired mean difference between Control and Test upon treatment with substance X was -3.35 [95.0%CI -5.59, -1.34]. Treatment with substance Y resulted in a mean difference of Control and Test of -3.47 [95.0%CI -5.21, -1.85].

Although the effect size after treatment with substance X and Y was very similar (substance X: -3.35; substance Y: -3.47), the higher precission of the effect after treatment with substance Y (CI width substance X: 4.25; CI width substance Y: 3.36) suggests that substance Y would be the better choice for future experiments. "

And to add a second (related) question:

If the result would e.g. be:

The unpaired mean difference between Control and Test is -1.7 [95.0%CI -3.79, 0.341].

Here we have an example where zero of the right hand axis is within the 95% CI. If I want to avoid "not significant", what could I write? Something like: " Since the observerd difference of Means is within the 95% CI we can not claim to observe a difference between the groups ."

But isn´t this also dichotomizing the results?

You know, the problem I see coming when using this kind of presentation in let´s say a lab progress report are probably as follows:

Me: here I have plotted the results of treating with substance Z. Presented as a Gardner-Altman-Plot.

Principle Investigator: Never heard of. Looks to me like there is a difference. Is it significant?

Me: Well, it´s not about significance - but the mean difference is within the 95% confidence intervall, so we cannot claim that there is a difference.

PI: So you´re saying it´s not significant?

Me: ??

Thank you for taking your time to reply.

Martin

[Joses Ho]

So, if you were a reviewer of a paper describing e.g. the above example + a second example for comparison (-3.47 [95.0%CI -5.21, -1.85]), you would be satisfied with something along the following lines?

" The unpaired mean difference between Control and Test upon treatment with substance X was -3.35 [95.0%CI -5.59, -1.34]. Treatment with substance Y resulted in a mean difference of Control and Test of -3.47 [95.0%CI -5.21, -1.85].

Although the effect size after treatment with substance X and Y was very similar (substance X: -3.35; substance Y: -3.47), the higher precission of the effect after treatment with substance Y (CI width substance X: 4.25; CI width substance Y: 3.36) suggests that substance Y would be the better choice for future experiments. "

This sounds like a perfectly decent reason to choose Substance Y.

And to add a second (related) question:

If the result would e.g. be:

The unpaired mean difference between Control and Test is -1.7 [95.0%CI -3.79, 0.341].

Here we have an example where zero of the right hand axis is within the 95% CI. If I want to avoid "not significant", what could I write? Something like: " Since the observerd difference of Means is within the 95% CI we can not claim to observe a difference between the groups ."

But isn´t this also dichotomizing the results?

You know, the problem I see coming when using this kind of presentation in let´s say a lab progress report are probably as follows:

Me: here I have plotted the results of treating with substance Z. Presented as a Gardner-Altman-Plot.

Principle Investigator: Never heard of. Looks to me like there is a difference. Is it significant?

Me: Well, it´s not about significance - but the mean difference is within the 95% confidence intervall, so we cannot claim that there is a difference.

PI: So you´re saying it´s not significant?

Me: ??

OK, to clarify, the rule-of-thumb is whether zero lies inside the 95CI, not the actual effect size. The effect size will always lies inside the 95Ci.

For PIs or reviewers who insist on "significance" language, I would refer them to these very recent and very widely-read articles:

"Abandon Statistical Significance" by McShane et al. 2019 <https://www.tandfonline.com/doi/full/10.1080/00031305.2018.1527253>

"Scientists Rise Up Against Statistical Significance" by Amrhein et al. 2019 <https://www.nature.com/articles/d41586-019-00857-9>

as well as our own Nature Methods article

"Moving beyond P values: data analysis with estimation graphics" <https://www.nature.com/articles/s41592-019-0470-3>

Using your example of

The unpaired mean difference between Control and Test is -1.7 [95.0%CI -3.79, 0.341].

What exactly does the effect size of -1.7 mean? Is it larger or smaller than what we expected? These are more scientifically important, and interesting, questions, compared to "is the p-value less than some threshold?"

Hope this helps,

Joses

[MartinH]

Thank you Joses

PS: Sorry for the confusion with the zero/effect size within the CI. I meant the correct thing, but didn´t write it down correctly.

[Joses Ho]

No Worries!