Discuss: abandon TravisCI?

[Edward K. Ream]

Imo, using TravisCI provides no significant testing benefits, slows the commit process,  and creates unnecessary (and confusing) distribution files.

Instead, devs need only ensure that Leo's test-all command succeeds before each commit.

Two recent issues have severely tested my tolerance for TravisCI:

1. TravisCI creates leo.exe, which imo should be hidden but somehow got documented here. See #2370.

2. Within the last several days, TravisCI has suddenly started rejecting all commits, even those based on master, which hasn't changed in over a month.  See #2421.

Folks, I think it's time to do without TravisCI. Your comments, please.

Edward

[tbp1...@gmail.com]

Trying this out for the first time:

Ran 812 tests in 18.881s
OK (skipped=10)

What does the "skipped" tell us and which ones are they?  Is this test routine what Travis runs?  And how will we be able to verify that everything works on the lowest supported version of Python, since most devs won't be running, say, Python 3.6?  Who will be adding new tests and making sure they work as expected?

I suggest that the output should include the Leo version, branch, changeset, OS, and date.  Then the devs can easily copy and paste it somewhere useful.

I'm inclined to think that routine hand-off testing is valuable.  If the current invocation of Travis can't be straightened out, maybe there is another workflow available?

[Edward K. Ream]

On Wed, Feb 16, 2022 at 8:20 AM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

Trying this out for the first time:

Ran 812 tests in 18.881s
OK (skipped=10)

What does the "skipped" tell us and which ones are they? 

"skipped" tells us that the test called self.skipTest().  The argument to skipTest tells why.

Is this test routine what Travis runs? 

I'm not sure.

And how will we be able to verify that everything works on the lowest supported version of Python, since most devs won't be running, say, Python 3.6?  Who will be adding new tests and making sure they work as expected?

I run test-all with python 3.6 from time to time.

I suggest that the output should include the Leo version, branch, changeset, OS, and date.  Then the devs can easily copy and paste it somewhere useful.

That could be done in a unit test. It might be worth doing.

I'm inclined to think that routine hand-off testing is valuable.  If the current invocation of Travis can't be straightened out, maybe there is another workflow available?

I am quickly going crazy trying to disable the automatic python 3.6 test.  I've revoked various TravisCI permissions, and TravisCI is still running tests and failing.  I guess it's time for a break :-)

Edward

[Edward K. Ream]

On Wednesday, February 16, 2022 at 9:14:03 AM UTC-6 Edward K. Ream wrote:

I am quickly going crazy trying to disable the automatic python 3.6 test.  I've revoked various TravisCI permissions, and TravisCI is still running tests and failing.  I guess it's time for a break :-)

Removing the leo-editor/.github/workflows directory disables all checks. (I did the checks in a throw-away PR and branch).

So this is progress :-)

BTW, TravisCI moved from .org to .com, and that caused problems signing in.  It's been a frustrating day.

One more note: the branch protection rules specify whether all tests must pass. This whole area is a nightmarish web of "protections" and "authorizations".  For now, I think it may be possible to avoid failing tests just by deleting the the leo-editor/.github/workflows directory in devel, but I'll cool off before making any more changes...

Edward

[Edward K. Ream]

On Wednesday, February 16, 2022 at 7:53:19 AM UTC-6 Edward K. Ream wrote:

Imo, using TravisCI provides no significant testing benefits, slows the commit process,  and creates unnecessary (and confusing) distribution files.

Imo, TravisCI is a form of featuritis. Indeed, devs need only do the following:

- Run test-all when committing.

- Run pylint on all files periodically.

 I have already revoked the permissions for TravisCI. The next steps:

- Remove the TravisCI files from both master and devel (using PR's), thereby disabling unit tests on all commits.

- Disable (convert @clean to @@clean) the corresponding nodes in leoDist.leo.

This will happen later today or early tomorrow.

Edward

[Edward K. Ream]

On Wednesday, February 16, 2022 at 11:34:47 AM UTC-6 Edward K. Ream wrote:

I have already revoked the permissions for TravisCI. The next steps:

- Remove the TravisCI files from both master and devel (using PR's), thereby disabling unit tests on all commits.

- Disable (convert @clean to @@clean) the corresponding nodes in leoDist.leo.

Done directly in master at rev 58f2dd9.  Done in devel via PR #2429.  leoDist.leo contains @@@clean nodes for the deleted files, so it should be straightforward to restore the status quo ante should that seem desirable.

So now Thomas and I are responsible for running tests manually.  To run pylint on all files, execute the following:

cd C:\leo.repo\leo-editor

python -m pylint-leo %*

 To run Leo's test-all command, execute the following in the leo-editor directory:

cd C:\leo.repo\leo-editor
python -m unittest %*

That's easy enough! The scripts shown above are for Windows .cmd files. Linux scripts would be similar.

Edward

[tbp1...@gmail.com]

c:\Tom\git\leo-editor>py -m unittest %*
E
======================================================================
ERROR: %* (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: %*
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\unittest\loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
ModuleNotFoundError: No module named '%*'


----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

[tbp1...@gmail.com]

Sorry, sent too soon by mistake.

Leo 6.6b2-devel, devel branch, build 513f9493ec
2022-02-16 14:35:19 -0600
Python 3.9.9, PyQt version 6.2.3
Windows 10 AMD64 (build 10.0.19043)
      home: C:\Users\tom
leo-editor: C:\Tom\git\leo-editor

[tbp1...@gmail.com]

All right, now I've got it.  The "%*" on the command line is a mistake.  When there are no command line parameters, then unittest tries to find tests in the directory tree.  So just plain

python3 -m unittest

[tbp1...@gmail.com]

If you want to include the Leo version in the output, as I suggested earlier, then:

python3 -m unittest  & python3 -m leo.core.runLeo -v

[Edward K. Ream]

On Wed, Feb 16, 2022 at 10:18 PM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

All right, now I've got it.  The "%*" on the command line is a mistake. 

My apologies for the confusion. The "%*" syntax is for use only in Windows .cmd files. I have two such .cmd files, one to run the test-all command, and one to run the pylint command on all of Leo's files.

Edward

[tbp1...@gmail.com]

I knew that "%*" was Windows command file syntax, of course, but I - since that's what you had written - supposed that the unittest module had some use for it.  Reflecting on that I don't see how it could, so put it down to a mind blip of mine.

[Edward K. Ream]

On Thu, Feb 17, 2022 at 7:04 AM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

I knew that "%*" was Windows command file syntax, of course, but I - since that's what you had written - supposed that the unittest module had some use for it.  Reflecting on that I don't see how it could, so put it down to a mind blip of mine.

No problem :-) My explanation could have been clearer.

Edward

[Edward K. Ream]

On Wednesday, February 16, 2022 at 11:34:47 AM UTC-6 Edward K. Ream wrote:

Imo, TravisCI is a form of featuritis. Indeed, devs need only do the following:

- Run test-all when committing.

- Run pylint on all files periodically.

I want to emphasize the following:

1. Running pylint on all files is too slow to be done in TravisCI, and the old TravisCI workflow never enforced any pylint tests.

2. I almost always run my test-all script after merging PR's. So TravisCI adds almost nothing.

3. Most importantly, the distribution checklist, in leoDist.leo, contains multiple items involving running unit tests and pylint tests.

In short, even without TravisCI, there is virtually no chance of significant bugs remaining after PR's or in official releases.

Edward