sample size for hypothesis testing

[Anne Schumann]

Hi,

I have a rather basic question about the size of samples in corpus linguistics. In our discipline, we usually use very large samples. On the other hand, in statistic, very large samples, if analysed with normal statistical tests, tend to reveal even minor differences as significant ones. I am a bit confused by that, What kind of conclusions can I draw from that and how do I arrive at reasonable analyses (besides by choosing a very high significance level).

Regards,

anne

[Stefan Th. Gries]

My take would be that you don't even need to choose a very small
significance level but that you pay close attention to effect sizes
(and distributional assumptions of methods). I had a case once as an
associate editor where someone had many significant results in
chi-squared tests on corpus data, but Cramer's V values in the range
of 0.01 and less. Much to the frustration of the author, two associate
editors and the editor-in-chief all decided to reject the paper
because the effects it wanted to make a case on were significant but
practically irrelevant.

STG
--
Stefan Th. Gries
-----------------------------------------------
University of California, Santa Barbara
http://www.linguistics.ucsb.edu/faculty/stgries
-----------------------------------------------

[Anne Schumann]

ok, in this case I would like to ask what kind of result would not be irrelevant. I just computed the V values on my data according to the description in your book. They are bigger than 0.01, but not much (the biggest being 0.08). I really have no intuition what kind of effect size would be meaningful for a reasonable comparison of text types or text register, for example.

2013/4/2 Stefan Th. Gries <stg...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "StatForLing with R" group.
To unsubscribe from this group and stop receiving emails from it, send an email to statforling-wit...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Stefan Th. Gries]

For many normalized effect sizes (i.e., effect sizes ranging from -1
to 1 or 0 to 1) there are 'standard ' recommendations out there (many
originally by J. by Cohen) but, as usual, they may differ quite a bit
so you need to look around. Another thing I often find useful are
Proportional Reduction of Error measures because they are more
intuitively interpretable than Cohen's d etc. Off the top of my head,
a V of 0.08 sounds quite small to me ...

[Anne Schumann]

Hi,

thank you. I just realised that I must be more precise with my question. I am comparing two different kinds of texts using the chi-square test. Actually, I do not want to correlate anything, I want to use the test for testing for homogenity of my two samples, as, for example, described in "Angewandte Statistik, Methodensammlung mit R", p. 590 ff. Hoping for a high Cramer index (which is a correlation measure) is probably not what I want to do in this case. Any other suggestions?

Regards,
anne

2013/4/2 Stefan Th. Gries <stg...@gmail.com>

For many normalized effect sizes (i.e., effect sizes ranging from -1

[Stefan Th. Gries]

If you have data like this:

x <- matrix(c(250,14050,460,20075), ncol=2,
dimnames=list(FEATURE=c("m","other"), TEXTTYPE=c("a","b")))

then Cramer's V is possible. "Correlation" does not just pertain to
two numeric variables but can also be used to talk about categorical
variables. In the above case, there would be a significant correlation
(p=0.0013948) such that text type b has more instances of m than
expected, whereas text type a has fewer than expected, and Cramer's V
is a lousy 0.01712228.

[Anne Schumann]

I am trying to understand the reasoning behind using these values. I think the problem is that so many linguistic phenomena are extremely rare. An example is given by Baroni and Evert (in the volume "Corpus Linguistics") who point to a difference of at least 3 percentage points taken from a confidence interval ranging between 0.03 and 0.25 (the Cramer index on their data is what you would probably call a still "lousy" 0.177302). So, what would be a "good" effect size for corpus linguistics or are there other tests that may be more insightful? Is it possible to measure differences in proportions while accounting for the order of magnitude in which we are moving?

Best,
a

2013/4/2 Stefan Th. Gries <stg...@gmail.com>

If you have data like this:

[Anne Schumann]

Hi,

I still would like to come back to this discussion to ask for suggestions or ideas. I read your 2005 paper on null-hypothesis testing and also the related Kilgarriff paper. I tested the Cramer index and the d-values on a set of pos tags taken from two corpus samples (with the tags from both corpora summing up to around 20,000). Statistically speaking, this is a very large sample, but for corpus linguistics it isn't. I also tested p-value adjustment using the R function p.adjust() on my vector of p-values from the tests.
Devising a heuristic based on your paper (Cramer and d indexes must be larger than x, with x being the score that 99% of your data in the paper did not reach) did not remove any data from my table, p-value adjustment worked a little, but only of I apply a maximum p-value of 0.001. I am wondering if these statistical measures are useful at all for characterising my data or whether there simply are no noteworthy effects? Any suggestion?

Regards,
anne

2013/4/2 Anne Schumann <ak47sc...@gmail.com>

[Stefan Th. Gries]

> Any suggestion?
Not without seeing the data and knowing the exact hypothesis you're
trying to test, no. This seems (to me at least) to call for more
differentiated input than a remote diagnosis can provide but maybe
someone else can help (more than I can).

[Anne Schumann]

sure. thanks anyway.

2013/4/3 Stefan Th. Gries <stg...@gmail.com>

[Matías Guzmán]

Sorry for jumping in into the conversation. In a test for two means I'm getting a Glass's Delta (I'm using this because I don't have the sd for one of the means) of 21.4 because the standard deviation of the control group is incredibly small (0.003). Can I interpret this as being a significant effect size?

Thank's a lot.

2013/4/3 Anne Schumann <ak47sc...@gmail.com>