@test assert True fails !?

[vitalije]

Hello!
I have recently noticed strange behaviour of testing in Leo. Somehow tests run ok, but they are still reported as broken.
Even a node with headline '@test example' and body of just one line 'assert True' is reported as FAILD
Here is traceback:

File ".../leo-editor/trunk/leo/core/leoCommands.py", line 7056, in recolor_now
    c.frame.body.colorizer.colorize(p,
AttributeError: 'NoneType' object has no attribute 'colorize'

----------------------------------------------------------------------
Ran 1 test in 0.004s

FAILED (errors=1)

Revision 4591, Ubuntu 11.04, Python 2.7

Vitalije.

[Edward K. Ream]

On Sat, Oct 15, 2011 at 1:22 AM, vitalije <vita...@gmail.com> wrote:

> I have recently noticed strange behaviour of testing in Leo. Somehow tests
> run ok, but they are still reported as broken.
> Even a node with headline '@test example' and body of just one line 'assert
> True' is reported as FAILD
> Here is traceback:
>
> File ".../leo-editor/trunk/leo/core/leoCommands.py", line 7056, in
> recolor_now
>     c.frame.body.colorizer.colorize(p,
> AttributeError: 'NoneType' object has no attribute 'colorize'

Thanks for this report.

To be clear, this happens only when running unit tests externally (Alt-5).

Rev 4952 contains a *complete* fix for the specific problems you
mention and *partial fixes* for related problems. For example, I have
marked a number of *sets* of unit tests in unitTest.leo as being
unsuitable when run externally: search for "(not valid when run
externally)"

There are two sources of problems:

1. Some unit tests require files or other data that is not
(automatically) present when running unit tests externally.

2. Some unit tests test code that will fail when using a nullGui,
which is the gui in effect when running unit tests externally.

Problem 2 might, perhaps, be fixed by running external tests with the
the QtGui. I don't remember why external tests use the nullGui, but I
suspect there is a real reason. However, Leo may be able to do more
now that Python always supports the subprocess module...

An additional problem is that there is, at present, no way to
distinguish between tests run "internally" from those run
"externally". This might be trickier than one might suppose at
first...

At any rate, the specific problems you mention have been fixed.
However, running unit tests in unitTest.leo externally is not really
recommended, nor is it part of Leo's tests protocol.

Edward

P.S. Here is the checkin log for rev 4952:

A variety of relatively minor changes to reduce (but not eliminate!)
error when running unit tests externally. A rethink is probably
needed.

leoFrame.py: Added nullColorizer to leoFrame.py: it used to be in leoColor.py.

leoGlobals.py & leoKeys.py: A few minor changes to better support
dynamic unit tests.

plugins files (see below): Added guards to protect against missing
ivars, etc, when running unit tests externally.

EKR

[vitalije]

I do test externally all the time and haven't noticed the above problem until recently.
Anyway, thanks for the fix.

PS: I don't think that "assert True" need any gui related function as many other tests. In fact I use a lot of tests just to execute some code without writing script file and then running it from console

[Edward K. Ream]

On Sat, Oct 15, 2011 at 1:06 PM, vitalije <vita...@gmail.com> wrote:
> I do test externally all the time and haven't noticed the above problem
> until recently.

The problems stem from newly enabled plugins.

Edward

[vitalije]

Did I cause appearing of this problem with enabling a new plug-in previously disabled? (I really did change @enabled plug-ins setting recently)

I would prefer if it would be possible to launch tests externally without loading plug-ins.

As I see in Code-->Testing-->@file leoTest.py-->runUnitTestLeoFile
and in Code-->Testing-->@file leoTest.py-->class runTestExternallyHelperClass-->runTests

there is an option to set gui for testing (g.app.unitTestGui) and default is 'nullGui'.

Maybe there could be an option to disable loading of plug-ins also ?

For the tests that I mostly write and run, loading plug-ins is just wasting time.
Vitalije.

[Edward K. Ream]

On Sat, Oct 15, 2011 at 3:54 PM, vitalije <vita...@gmail.com> wrote:
> Did I cause appearing of this problem with enabling a new plug-in previously
> disabled? (I really did change @enabled plug-ins setting recently)

You might have "caused" the problem in some sense, but in a truer
sense you had nothing to do with it :-)

> I would prefer if it would be possible to launch tests externally without loading plug-ins.

Your idea is a good one. I am not sure what happens at present, and
I'm not sure whether any settings apply to these commands. I'll look
into it immediately, while the issues are still fresh in my mind.

Edward

[Edward K. Ream]

On Sat, Oct 15, 2011 at 5:03 PM, Edward K. Ream <edre...@gmail.com> wrote:

> I am not sure what happens at present, and I'm not sure whether any settings apply to these commands.

Good news (mostly): you can set up dynamic unit tests exactly the way you want.

I was going to point you at the documentation but (how can this
possibly be?) the documentation doesn't exist! I really could have
sworn that everyone knew about what I am about to tell you ;-) To
quote Alan Greenspan, "I'm shocked. Shocked."

So here goes...

Let p be the presently selected node when you hit Alt-5
(run-unit-tests). Leo, that is, the code in
runTestExternallyHelperClass, creates dynamicUnitTest.leo by
assembling all of the following nodes from p's descendants.

- All @test nodes,
- All @suite nodes, and, **most importantly**,
- All children of all @mark-for-unit-tests nodes.

You can use @mark-for-unit-test nodes to include any "supporting data"
you want, including, say, "@common test code" to be imported as
follows::

exec(g.findTestScript(c,'@common test code'))

Hmm. I doubt this is really written up properly, although I know for
sure at the time I did write it up on leo-editor. It's a very cool
pattern.

Now here is the new trick. It *should* work, but actually does *not*
work yet. Boo hoo. Are you ready?

- @mark-for-unit-tests
- @settings
- @enabled-plugins

Alas, this doesn't work because the test code runs dynamicUnitTest.py,
which runs dynamicUnitTest.leo **using the leoBridge module**. And,
as I have just verified, the leoBridge module doesn't load *any*
plugins. Nor does it init any other settings.

So, no matter *what* @enabled-plugins contains, your code will start
off with no plugins at all. Thus, the following unit test will always
pass when run with Alt-5::

@test loaded-plugins

pc = g.app.pluginsController
aList = pc.getLoadedPlugins()
assert not aList,aList

However, your tests *can* load plugins if they like, provided they
don't through an exception when loaded with nullGui. For example, the
following test passes::

@test load backlink plugin

pc = g.app.pluginsController
pc.loadOnePlugin('backlink.py')
aList = pc.getLoadedPlugins()
assert 'leo.plugins.backlink' in aList

A similar trick could probably be used to set c.config settings from
the @settings tree, but at present your code would have to do all the
work itself: the leoBridge module does **not** initialize settings.
That is, c.config.getBool(x) returns None for all (strings) x.

I hope this gives you a sense of what is going on. For full details,
consult leoTest.py.

It has been fun tracking this down. I'll fix the documentation immediately.

Edward

[Edward K. Ream]

On Oct 15, 6:15 pm, "Edward K. Ream" <edream...@gmail.com> wrote:

> It has been fun tracking this down.  I'll fix the documentation immediately.

Well, "immediately" now has a new meaning :-) For the last several day
I have been busy creating action items from leo-editor and email.
Still not done yet, but I just added the notes from this conversation
to the to-do list. It may be a week or more before I get to this.

Edward