Pytest command

[vitalije]

Two days ago I've added to Leo a new command 'execute-pytest' which runs like `run-selected-unit-tests-locally`, but executes test functions in @test nodes as if they were in a file processed by pytest.

To use this command one must have pytest installed. Command will look in the subtree of the selected node for all nodes whose headlines starts with '@test'. If no such node has been found, command will look into the parents of selected node for `@test` headlines. Every found node will be searched for functions whose name starts with 'test_' and each test function will be executed using assert rewriting as pytest does.

Here is an example:

def add_2(x):
    return x + 2

def times_3(x):
    return x * 3

def inc(x):
    return x + 1

def test_a():
    assert inc(3) == 4.1

def test_b():
    assert times_3(add_2(inc(3.1))) == times_3(inc(5))

and here is the output in the Log pane:

----------Failure-------------
assert 4 == 4.1
 +  where 4 = inc(3)
----------Failure-------------
assert 18.299999999999997 == 18
 +  where 18.299999999999997 = times_3(6.1)
 +    where 6.1 = add_2(4.1)
 +      where 4.1 = inc(3.1)
 +  and   18 = times_3(6)
 +    where 6 = inc(5)
failed:2 tests

changing the constants 3.1 to 3 and 4.1 to 4, all tests pass.

Pytest machinery rewrites each assert statement to show not only the final two values that are compared, but all intermediate results in calcualting both sides of the expression. This greatly reduces the amount of work on formatting assertion messages. Quite often pytest generated messages are sufficient.

Of course, test scripts have access to g, c and p as usual.

Vitalije

[vitalije]

Hm, looking in the output it seems that the name of the test node, and failed test function in the output are missed. Revision 064e218 fixes this. Now the output is like:

-------example/test_a failed---------

assert 4 == 4.1
 +  where 4 = inc(3)

-------example/test_b failed---------

assert 18.299999999999997 == 18
 +  where 18.299999999999997 = times_3(6.1)
 +    where 6.1 = add_2(4.1)
 +      where 4.1 = inc(3.1)
 +  and   18 = times_3(6)
 +    where 6 = inc(5)
failed:2 tests

Vitalije

[Edward K. Ream]

On Thu, Sep 26, 2019 at 5:19 AM vitalije <vita...@gmail.com> wrote:

Two days ago I've added to Leo a new command 'execute-pytest' which runs like `run-selected-unit-tests-locally`, but executes test functions in @test nodes as if they were in a file processed by pytest.

Many thanks for this.  It was the principle component of #1222, which I have just closed.  I have just created and closed #1352 to memorialize this excellent work.

Edward

[vitalije]

I was not aware of #1222. :-)

Vitalije

[Brian Theado]

Vitalije,

This looks nice. I'm really impressed with the assertion failure output pytest gives.

I wonder how hard it would be to write a pytest plugin to support leo's unit tests. Their plugin system makes use of hooks like Leo does, only it is like hooks on steroids.

This example code shows how tests can be collected from a yaml file instead of python code: https://docs.pytest.org/en/latest/example/nonpython.html#yaml-plugin. Writing code to get tests from unittest.leo might be similarly straightforward.

https://docs.pytest.org/en/latest/writing_plugins.html

https://docs.pytest.org/en/latest/plugins.html#using-plugins

https://docs.pytest.org/en/latest/reference.html#collection-hooks

https://pluggy.readthedocs.io/en/latest/

That's quite an impressive plugin system pytest has.

Brian

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/b7880eb4-9c2f-4d37-932d-d1f0d72fcfe3%40googlegroups.com.

[Edward K. Ream]

On Thu, Sep 26, 2019 at 12:37 PM Brian Theado <brian....@gmail.com> wrote:

This example code shows how tests can be collected from a yaml file instead of python code: https://docs.pytest.org/en/latest/example/nonpython.html#yaml-plugin. Writing code to get tests from unittest.leo might be similarly straightforward.

https://docs.pytest.org/en/latest/writing_plugins.html

https://docs.pytest.org/en/latest/plugins.html#using-plugins

https://docs.pytest.org/en/latest/reference.html#collection-hooks

https://pluggy.readthedocs.io/en/latest/

That's quite an impressive plugin system pytest has.

Thanks for these links.  I've bookmarked them.

Edward

[Brian Theado]

Vitalije,

I was looking at the execute-pytest code and it looked to me like only the assertion rewrite functionality from pytest is being used. I would guess none of the hooks or fixtures and maybe most plugins will work.

I don't much trust my code reading so I figured I'd better test it, but I only got a stack trace when trying with your example:

Traceback (most recent call last):

  File "/home/btheado/src/leo-editor/leo/core/leoGlobals.py", line 293, in new_cmd_wrapper
    func(self, event=event)

  File "/home/btheado/src/leo-editor/leo/core/leoCommands.py", line 729, in execute_pytest
    self.execute_single_pytest(p)

  File "/home/btheado/src/leo-editor/leo/core/leoCommands.py", line 761, in execute_single_pytest
    rewrite_asserts(tree, script, config=cfg)

  File "/home/btheado/src/pyqt-3.7-venv/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 327, in rewrite_asserts
    AssertionRewriter(module_path, config, source).run(mod)

  File "/home/btheado/src/pyqt-3.7-venv/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 572, in __init__
    "enable_assertion_pass_hook"

  File "/home/btheado/src/pyqt-3.7-venv/lib/python3.7/site-packages/_pytest/config/__init__.py", line 976, in getini
    self._inicache[name] = val = self._getini(name)

  File "/home/btheado/src/pyqt-3.7-venv/lib/python3.7/site-packages/_pytest/config/__init__.py", line 987, in _getini
    value = self.inicfg[name]

AttributeError: 'Config' object has no attribute 'inicfg'

python 3.7.3 and pytest 5.2.0

Do you have any ideas?

On Thu, Sep 26, 2019 at 6:31 AM vitalije <vita...@gmail.com> wrote:

--

[vitalije]

On Thursday, October 3, 2019 at 2:30:23 AM UTC+2, btheado wrote:

Vitalije,

I was looking at the execute-pytest code and it looked to me like only the assertion rewrite functionality from pytest is being used. I would guess none of the hooks or fixtures and maybe most plugins will work.

Yes I guess you are right, I just wished to have assertions from pytest. I haven't used any pytest plugin nor its hooks 

 

I don't much trust my code reading so I figured I'd better test it, but I only got a stack trace when trying with your example:

Traceback (most recent call last):

  File "/home/btheado/src/leo-editor/leo/core/leoGlobals.py", line 293, in new_cmd_wrapper
    func(self, event=event)

  File "/home/btheado/src/leo-editor/leo/core/leoCommands.py", line 729, in execute_pytest
    self.execute_single_pytest(p)

  File "/home/btheado/src/leo-editor/leo/core/leoCommands.py", line 761, in execute_single_pytest
    rewrite_asserts(tree, script, config=cfg)

  File "/home/btheado/src/pyqt-3.7-venv/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 327, in rewrite_asserts
    AssertionRewriter(module_path, config, source).run(mod)

  File "/home/btheado/src/pyqt-3.7-venv/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 572, in __init__
    "enable_assertion_pass_hook"

  File "/home/btheado/src/pyqt-3.7-venv/lib/python3.7/site-packages/_pytest/config/__init__.py", line 976, in getini
    self._inicache[name] = val = self._getini(name)

  File "/home/btheado/src/pyqt-3.7-venv/lib/python3.7/site-packages/_pytest/config/__init__.py", line 987, in _getini
    value = self.inicfg[name]

AttributeError: 'Config' object has no attribute 'inicfg'

python 3.7.3 and pytest 5.2.0

Do you have any ideas?

I have installed pytest 3.8.0 on my machine. Most likely the problem is in pytest incompatible versions. I'll look into it but I can't do it now. 

Vitalije