Inconsistency in slides

[Mohit Mendiratta]

Hi, 

While going through the slides, I realized that the definition of p values are different for permutation test and z test. For permutation test p value: P (T(x) > tobs),while for Z test it is P(T(x) >= tobs). Going by the definition, p value is the smallest significance level for which the null hypothesis is rejected. Hence P(T(X) >= tobs) makes sense.Am i missing something here? 

[Maksym Andriushchenko]

Hi,

For a continuous random variable, >= vs > doesn't make any difference.

For a discrete random variable, it does, and it's discussed on page 106 of the lecture notes:

So one way to resolve this problem is just to take the mean of the two values.

Or alternatively, taking >= leads to a more conservative test which would be a safer bet than taking > which would underestimate the p-value.

Best,

Maksym

On Sun, Feb 17, 2019 at 2:17 PM Mohit Mendiratta <mohitmend...@gmail.com> wrote:

Hi, 

While going through the slides, I realized that the definition of p values are different for permutation test and z test. For permutation test p value: P (T(x) > tobs),while for Z test it is P(T(x) >= tobs). Going by the definition, p value is the smallest significance level for which the null hypothesis is rejected. Hence P(T(X) >= tobs) makes sense.Am i missing something here? 

--
You received this message because you are subscribed to the Google Groups "Machine Learning WS18/19" group.
To unsubscribe from this group and stop receiving emails from it, send an email to machine-learning-...@googlegroups.com.
To post to this group, send email to machine-lea...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/machine-learning-ws1819/CAN%2BgTNjutOdiNQntNafF4gz%2BfQegZM3%3DR0HVtc-dNPnjm%3DB5nw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Mohit Mendiratta]

What if the p - value turns out to be zero? For a certain a certain number of samples? Do we consider sampling a higher number of samples or go with a conservative test?

[Maksym Andriushchenko]

ideally, you should do all possible permutations

if you don't do all of them (i.e. if the number of samples is too large), since you are usually bounded by your computational resources, you just have an approximation of the true p-value

then obtaining the p-value of 0 is still fine, but you should keep in mind that it's an approximation and that the true p-value is not necessarily 0.

and of course, if you can sample more permutations, you should alawys do it.