Benefit of XFAIL over SKIP?

[Peter Brady]

This is mostly a software design/engineering question:

There are a large number of XFAILed tests in the test suite and some of them are rather slow.  Here's a sample output:

sympy/core/tests/test_wester.py[396] .........ffffffffffffffffffffffffffffffffff
fXffwffffffffffffffffwffffffffffffffffffffffffffffffwfffffffffffffffffffffffffff
fffffffffffffffffffffff.fffffffwffffffffffffffffwfffwfffffffffff....swssws..wwww
wwwww...........................................................................
................................................................................

sympy/galgebra/tests/test_ga.py[23] .....ffffff............                 [OK]
sympy/geometry/tests/test_geometry.py[24] ........................          [OK]
sympy/integrals/tests/test_deltafunctions.py[2] ..                          [OK]
sympy/integrals/tests/test_failing_integrals.py[20] ffffwfwffffwwffwfwws    [OK]

What is the is benefit of all these XFAIL'ed tests?  I can understand that they serve a purpose in documenting behavior but why not just SKIP them to cut down on testing time?

[Harsh Gupta]

> What is the is benefit of all these XFAIL'ed tests?

A passed XFAIL'ed test is reported. Just as a seemly unrelated change
can make a test fail, a seemly unrelated change can also make the test
pass and it important to be aware both the cases.

> --
> You received this message because you are subscribed to the Google Groups
> "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sympy+un...@googlegroups.com.
> To post to this group, send email to sy...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sympy.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sympy/ef749f67-6db9-40b2-9897-6164fd1d0287%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Harsh

[Ondřej Čertík]

On Thu, Aug 7, 2014 at 9:49 AM, Harsh Gupta <gupta....@gmail.com> wrote:
>> What is the is benefit of all these XFAIL'ed tests?
>
> A passed XFAIL'ed test is reported. Just as a seemly unrelated change
> can make a test fail, a seemly unrelated change can also make the test
> pass and it important to be aware both the cases.

Though, in practice, I don't know if anyone is using this feature.
Usually one has
to make sure the test passes for all tests, so the best in my opinion
is to enable it
in a PR and make sure it passes. Then keep it enabled.

So I personally think we should just skip the slow xfailing tests.

Ondrej

[Sergey Kirpichev]

On Thursday, August 7, 2014 7:52:26 PM UTC+4, Ondřej Čertík wrote:

On Thu, Aug 7, 2014 at 9:49 AM, Harsh Gupta wrote:
>> What is the is benefit of all these XFAIL'ed tests?
>
> A passed XFAIL'ed test is reported. Just as a seemly unrelated change
> can make a test fail, a seemly unrelated change can also make the test
> pass and it important to be aware both the cases.

Though, in practice, I don't know if anyone is using this feature.

I use :)  Well, sometimes.

There may be be a better way.  E.g. we can add these tests as specially formatted
comments for related issues.  And we can automate testing of the master for
such tests with some bot...  AFAIR, something like this was suggested by Harsh.

[Aaron Meurer]

Maybe it would help to finish implementing https://github.com/sympy/sympy/issues/7044. The biggest issue with XFAIL is that there will be some silly error in the way the test is written so that it never XPASSes, even when it should.

We should also do periodic reviews of XFAIL tests to see if they can be made into passing tests. Sometimes the thing that is expected to pass is a little different than what actually ends up passing. 

Aaron Meurer

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/fe56f692-548e-4f0c-990e-5b3a143f3c1c%40googlegroups.com.

[Sergey Kirpichev]

On Sat, Aug 23, 2014 at 11:36 PM, Aaron Meurer <asme...@gmail.com> wrote:
> Maybe it would help to finish implementing
> https://github.com/sympy/sympy/issues/7044. The biggest issue with XFAIL
> is that there will be some silly error in the way the test is written so
> that it never XPASSes, even when it should.

We can add some documentation for developers ("how to write
XFAIL'ed tests"). The simplest case of such errors - when we can't
be sure in the exact form of the answer.

[Aaron Meurer]

That's a good idea. A good XFAIL test is different from a good normal test. In a good XFAIL test, you are really testing if something has started working at all. For example, if you want to test if integrate() can do an integral that it currently can't do, you should just test not integrate(...).has(Integral), rather than integrate(...) == answer, because the answer could take many forms, and the one you pick might not be the one that it ends up giving.

Aaron Meurer

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
To post to this group, send email to sy...@googlegroups.com.
Visit this group at http://groups.google.com/group/sympy.

To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/20140824134858.GA3684%40debian.

[Peter Brady]

I was perusing the pytest docs and found http://pytest.org/latest/skipping.html.  It appears that in the pytest xfail decorator one can specify the error with raises.  Here's a little test

import pytest

@pytest.mark.xfail(raises=IndexError)

def test_false_xfail():

    x= [1, 2, 3]

    assert x[0] == 3

@pytest.mark.xfail(raises=IndexError)

def test_true_xfail():

    x= [1, 2, 3]

    assert x[5] == 3

@pytest.mark.xfail(raises=IndexError)

def test_passing_xfail():

    x= [1, 2, 3]

    assert x[0] == 1

$ py.test test_demo.py  -v

============================= test session starts ==============================

platform linux -- Python 3.4.1 -- py-1.4.25 -- pytest-2.6.3 -- /home/ptb/miniconda3/bin/python3

collected 3 items 

test_demo.py::test_false_xfail FAILED

test_demo.py::test_true_xfail xfail

test_demo.py::test_passing_xfail XPASS

=================================== FAILURES ===================================

_______________________________ test_false_xfail _______________________________

    @pytest.mark.xfail(raises=IndexError)

    def test_false_xfail():

        x= [1, 2, 3]

>       assert x[0] == 3

E       assert 1 == 3

test_demo.py:6: AssertionError

================ 1 failed, 1 xfailed, 1 xpassed in 0.02 seconds ================

This would be a neat feature to add to help ensure good XFAIL tests.

[Aaron Meurer]

Issue for this: https://github.com/sympy/sympy/issues/7044

Aaron Meurer

> https://groups.google.com/d/msgid/sympy/67f959bd-0163-420b-876b-76ae3f812a5f%40googlegroups.com.