New pyflakes and (soon) flake8 commands

[Edward K. Ream]

pyflakes is a much faster (20x or more) version of pylint, and in some ways does a better job.  However, it is not configurable at all which is bizarre.  Happily, flake8 wraps and provides configuration for pyflakes and the pep8 tool.

I don't remember whether pyflakes or flake8 come with Anaconda, but the usual pip install is all that is needed.

Documentation for both pyflakes and flake8 is poor, even by geek standards. The configuration-related code is way too complex, which has caused me considerable effort.

Anyway, Leo now has a pyflakes command which will use the flake8 configuration file if flake8 has been installed.  Leo looks for the file "flake8" (no file extension) or (an addition for Leo) "flake8.txt".  Leo looks in ~, then ~/.leo and then leo/test for these files.

Coming asap:

- Add a reference copy of flake8.txt to leo/test that disable complaints about Leo's default coding style.
- Create two separate commands: flake8 and pyflakes, so the user can choose between them explicitly.
  At present, the pyflakes command actually runs flake8 if flake8 is installed.
- Configure the flake8 options.statistics, options.total_errors and options.benchmark settings from flake8.txt.
  - Should be easy.  It's not.  I may have to ask for help.
  - At present, these settings are set to True, True and False respectively.
- Optionally use flake8 to check external files on save, just as Leo presently checks their syntax.
  flake8 is easily fast enough to make this feasible.
- Add leo/flake8-leo.py, so that people can run flake8 on Leo's sources, just as with pylint-leo.py.

Edward

[Edward K. Ream]

On Tuesday, May 17, 2016 at 1:19:46 PM UTC-5, Edward K. Ream wrote:

pyflakes is a much faster (20x or more) version of pylint, and in some ways does a better job. 

pyflakes is more like 100x faster than pylint, and 10x faster than flake8. pyflakes is the natural tool to run when saving .leo files.  flake8 is enough slower that it wouldn't be so useful.

So it is worth considerable effort to eliminate all pyflakes complaints. This is usually possible with a bit of effort.  For example, warnings about unused code can be silenced with an assert.  Ugly, but feasible.

As another example, warnings that raw_input is undefined (in Python 3) can be suppressed by "fetching" it from the builtins module:

f = builtins.input if g.isPython3 else builtins.raw_input
    # Suppress pyflakes complaint.
leoid = f('LeoID: ')

Again, not pretty.  A pylint-style suppression comment would be better, imo.  I may complain to the pyflakes people, but I don't have a lot of hope...

Done:

- Added a reference copy of flake8.txt to leo/test.
- Created two separate commands: flake8 and pyflakes.

Still to do:

[Edward K. Ream]

On Tuesday, May 17, 2016 at 2:47:29 PM UTC-5, Edward K. Ream wrote:

 

So it is worth considerable effort to eliminate all pyflakes complaints. This is usually possible with a bit of effort...warnings that raw_input is undefined (in Python 3) can be suppressed by "fetching" it from the builtins module:

f = builtins.input if g.isPython3 else builtins.raw_input
    # Suppress pyflakes complaint.
leoid = f('LeoID: ')

Again, not pretty. 

Not just ugly, but buggy.  On Linux, importing builtins will fail in Python 2. So my code that passed all tests on Windows has just failed several unit tests on Linux.

This illustrates the perils of coding to linters. Complete testing is required for all such workarounds.  A "solution" on Linux would be to import a module called "future", but that would create installation problems. This also illustrates how odious the pyflakes refusal to inhibit messages can be.  Bugs, courtesy of pyflakes.

It's unbearable to see pyflakes messages that can't be suppressed, so something safe and effective must be done. It doesn't have to be pretty, however ;-)

Edward

[Terry Brown]

On Tue, 17 May 2016 13:46:12 -0700 (PDT)
"Edward K. Ream" <edre...@gmail.com> wrote:

> It's unbearable to see pyflakes messages that can't be suppressed, so
> something safe and effective must be done. It doesn't have to be
> pretty, however ;-)

Filter the messages?

Make messages that look like:

File foo.py, line 123: Used * or ** magic
s = a(*b, **c) # no-flakes

go away because of the comment?

Cheers -Terry

[Edward K. Ream]

On Tuesday, May 17, 2016 at 3:46:13 PM UTC-5, Edward K. Ream wrote:

It's unbearable to see pyflakes messages that can't be suppressed, so something safe and effective must be done. It doesn't have to be pretty, however ;-)

The following works:

try:
    import __builtin__ as builtins
    print(builtins.raw_input) # Python 2
except ImportError:
    import builtins
    print(builtins.input) # Python 3

I'll have to give some thought about encapsulating this.  Perhaps g.input and g.raw_input...

This also illustrates why Python 3 hasn't exactly been an overnight success.  Python 3 is the cost that keeps on costing.

EKR

[Edward K. Ream]

On Tuesday, May 17, 2016 at 3:59:43 PM UTC-5, Edward K. Ream wrote

It's unbearable to see pyflakes messages that can't be suppressed, so something safe and effective must be done. It doesn't have to be pretty, however ;-)

The following works:

try:
    import __builtin__ as builtins
    print(builtins.raw_input) # Python 2
except ImportError:
    import builtins
    print(builtins.input) # Python 3

I'll have to give some thought about encapsulating this.  Perhaps g.input and g.raw_input...

g.input, etc. would be a wretched "solution", because g.input would shadow the actual input builtin. Instead, the solution is to define builtin​s as follows in the <<imports>> section everywhere needed:

try:
    import builtins # Python 3
except ImportError:
    import __builtin__ as builtins # Python 2

This illustrates something I never thought about, namely that the Don't Repeat Yourself (DRY) principle doesn't apply to Python imports.

With this import in place, we can use the previous workaround, without worrying about what builtins actually is:

f = builtins.input if g.isPython3 else builtins.raw_input
    # Suppress pyflakes complaint.
leoid = f('LeoID: ')

So this problem looks solved.  We'll see :-)

EKR

[Edward K. Ream]

On Tuesday, May 17, 2016 at 1:19:46 PM UTC-5, Edward K. Ream wrote:

pyflakes is a much faster (20x or more) version of pylint, and in some ways does a better job.

Leo now sports three stand-alone linters: leo-pylint.py, leo-pyflakes.py and leo-flake8.py, (collectively, leo-x.py) all in the top-level leo directory.  You can use the -h command-line option to see available command-line arguments.  As with leo-pylint.py, the typical usage is: "python2 leo-x.py args", which I invoke with a .bat/.sh file.

As mentioned before, leo-flake8.py requires a configuration file, flake8 or flake8.txt in either ~, ~/.leo or leo/test.  leo-pyflakes.py uses no configuration file.

leo-pyflakes.py is super fast: 4.3 sec on my new, fast machine to check all 229 files specified with the -a option.

Still to do:

- Remove all pyflakes errors. All are minor, but most are valid.
- Remove/suppress all flake8 errors. Many are irksome, but it's not hard to suppress unwanted messages.  The settings to do so will be added to
leo/test/flake8.txt.

Edward

P.S. Several cleanups:

1. Finally, I see how to recreate the equivalent of g.app.loadDir from outside of Leo:

loadDir = g.os_path_finalize_join(g.__file__, '..', '..')

This allows the leo-x.py files to compute the full paths of all files under test. This eliminates the need for "helper" code in the leoGlobals module. Previously, this helper code ran in a separate process, which caused problems with timing statistics on Linux. Now, the code can run within the leo-x.py files as normal code.

leo-flake8.py and leo-pyflakes.py already use this technique.  I'll retrofit leo-pylint.py with the new code asap.

2. At present, all three leo-x.py files contain exactly the same code to compute the list of files to be checked with the various command-line option.  I'll soon move an encapsulated version of this code into leo.core.leoTest.py.

EKR