ENB: Curses prototype: phase 2 design and process

[Edward K. Ream]

This post discuss the next phase of curses gui prototype. This is an Engineering Notebook post, of interest only to devs.

This second phase is a crucial intermediate step. It should be doable in a few hours.

Phase 1 summary

1. cursesGui2.py contains three functional classes CursesGui, CursesFrame and CursesMenu, and one disabled class, CursesLog.

2. The cursesGui ctor set self.consoleOnly = True, which tells g.trace to use print immediately for traces instead of queuing up traces for Leo's log window.  Today's work will start with making things work when g.app.gui.consoleOnly is False. This will queue traces for later.

2. GG.runMainLoop is just barely functional:

    def runMainLoop(self):
        '''The curses gui main loop.'''
        w = curses.initscr()
        w.addstr('enter characters: x quits')
        while 1:
            i = w.getch() # Returns an int.
            ch = chr(i)
            if ch == 'x': break
        sys.exit(0)

The call to curses.initscr() erases the entire console, including all traces generated by g.trace.

Phase 2 design and process

1. CG.runMainLoop should implement Leo's log pane in a (scrollable?) curses text widget of some kind. The initial contents of this widget will be the queued log messages generated during startup. This step will enable and revise the CursesLog class, which is presently commented out. This class started life as a copy of the LeoQtLog class.

2. CG.runMainLoop should translate key w.getch key codes to LeoKeyEvent objects and call c.k.masterKeyHandler, where c is the first loaded outline.  I'm guessing c will be available as g.app.log.c.  In effect, this step rewrites LeoQtEventFilter.

And that's it.  The only knowledge needed was/will be:

A. Remembering yesterday's consoleOnly workaround.
B. Knowing that we need to rewrite the keystroke portion LeoQtEventFilter so it calls k.masterKeyHandler.
C. Knowing that the LoadManager class is the proper place to start single-stepping through code to see why log messages get eaten, as they will when g.app.gui.consoleOnly gets set back to False :-)

Summary

Phase two of the curses prototype is a crucial intermediate step.  It will capture all log messages in a curses text widget so that they aren't lost after calling curses.initscr(). More importantly, CG.runMainLoop will call k.masterKeyHandler.  This will allow Leo to execute commands. We will see the effects of those commands only via enabled traces, but those traces will show up in the new curses log widget.

I expect to finish phase two today. It should be straightforward, provided a proper curses text widget can be found. I am hoping that npyscreen widgets will suffice. We shall see...

The third and final phase of this project will be to implement most of Leo's gui elements using curses/npyscreen widgets. Leo's menu, icon bar and status areas are optional, as are most subsidiary tabs of the log widget. This last phase is still a big deal. It remains to be seen how useful npyscreen will be.

Edward

[Edward K. Ream]

On Thursday, April 20, 2017 at 3:26:31 AM UTC-5, Edward K. Ream wrote:

2. The cursesGui ctor set self.consoleOnly = True, which tells g.trace to use print immediately for traces instead of queuing up traces for Leo's log window.  Today's work will start with making things work when g.app.gui.consoleOnly is False. This will queue traces for later.

Done at 243740. Notes:

1. Leo does not queue g.trace messages. Leo only queues log messages.
2. It was useful not to define a __getattr__ method in the CursesLog class.  This ensured quick tracebacks.
3. The CursesLog class is not a subclass of the LeoLog class, at least for now. Only two methods were copied from LeoLog.
4. CG.runMainLoop just calls sys.exit().  It is only a test of the CursesLog class.

So now we have gone as far as we can without a curses log widget.  There are two parts left of this phase of the prototype:

Part A.  Implement a curses log widget and re-enable curses in CG.runMainLoop.
Part B.  Call k.masterKeyHandler to handle keystrokes.

Each part will like take an hour or three.

Edward

[Edward K. Ream]

On Thursday, April 20, 2017 at 4:43:26 AM UTC-5, Edward K. Ream wrote:

> There are two parts left of this phase of the prototype:

Part A.  Implement a curses log widget and re-enable curses in CG.runMainLoop.
Part B.  Call k.masterKeyHandler to handle keystrokes.

Rev e1e4e48 completes part A using an npyscreen.NPSApp class and associated MultiLineEditableBoxed widget. It looks like npyscreen has what we need, including a tree widget.

Queued log messages appear in the text widget, and subsequent calls to g.es adds to that text.

Remember, to run the app, use the --gui=curses command-line option. To quit the app, click on the ok button in the lower right corner, then hit return.

Edward

[Edward K. Ream]

On Thursday, April 20, 2017 at 6:21:05 AM UTC-5, Edward K. Ream wrote:

Rev e1e4e48 completes part A using an npyscreen.NPSApp class and associated MultiLineEditableBoxed widget. It looks like npyscreen has what we need, including a tree widget.

One more improvement: the various __getattr__ methods now use both g.trace and g.es, so that their output is retained in the curses log widget after the original console's contents is overwritten.

On to part B!

Edward

[Edward K. Ream]

On Thursday, April 20, 2017 at 6:21:05 AM UTC-5, Edward K. Ream wrote:

Remember, to run the app, use the --gui=curses command-line option. To quit the app, click on the ok button in the lower right corner, then hit return.

Also, when not editing, tab moves between widgets, and shift-tab moves in the opposite direction.

The npyscreen docs are weak.  Reading the code (or trial and error) is more helpful.

Edward

[john lunzer]

I realize you're doing this as an exercise but if you're getting frustrated with npyscreen (I also think the docs are a little sparse) there is always urwid, which I think has much better documentation. It's also got all the widgets necessary for Leo.

Also, again, a good example of a full screen application which has a layout somewhat similar to what you'd expect in Leo is pudb. It's got a tree widget which lists variables and a code widget which shows source code. It's also got a list widget that lists the frame stack.

I've mentioned it before but there is also a full fledged code editor name xo (exofrills) which is also written in urwid.

My plan, when I magically found time, was to study both pudb and xo, because I think between the two there is enough there to put together a full fledged Leo curses GUI.

[Edward K. Ream]

On Thu, Apr 20, 2017 at 8:12 AM, john lunzer <lun...@gmail.com> wrote:

I realize you're doing this as an exercise but if you're getting frustrated with npyscreen (I also think the docs are a little sparse) there is always urwid, which I think has much better documentation. It's also got all the widgets necessary for Leo.

​Thanks for the reminder. Now would be a good time to investigate urwid.

I'm having trouble getting control of npyscreen keystrokes in a useful way. The biggest challenge is debugging. Both prints (sys.stdout.write) and g.pdb scroll the npyscreen stuff off the console!

Edward

[Kent Tenney]

possibly logging would help?
https://pypi.python.org/pypi/devpy

provides logging with one line of code, maybe the useful message gets

logged before the terminal is lost ...

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscribe@googlegroups.com.
To post to this group, send email to leo-e...@googlegroups.com.
Visit this group at https://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

[Edward K. Ream]

On Thu, Apr 20, 2017 at 11:31 AM, Kent Tenney <kte...@gmail.com> wrote:

possibly logging would help?
https://pypi.python.org/pypi/devpy

​I doubt it. The solution, imo, is to use pudb (linux) or winpdb (windows). I got sidetracked with other issues.  I'll attempt one or the other working now...

Edward

[Terry Brown]

On Thu, 20 Apr 2017 17:24:19 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

In linux at least there are plenty of ways of separating the trace
output from the screen output. As Kent points out, you could use
logging to log to a file, and then to tail -f on the file in a
different terminal, separating tty display and logging completely.
Maybe you don't even need logging, just points g.* outputs at a file
rather than sys.stderr would do it.

Cheers -Terry

[Edward K. Ream]

On Thursday, April 20, 2017 at 6:09:19 PM UTC-5, Terry Brown wrote:

In linux at least there are plenty of ways of separating the trace
output from the screen output.  

Yes, I know that. On Linux, but not windows, the console is restored after curses exits, so any traces are available later, just as with logging or printing to a file.

But I also need to single-step through code.  g.pdb uses the console, so it's out. winpdb runs on Linux, so I'll come up to speed with that.

Edward

[Edward K. Ream]

Yesterday was, mostly, a long digression into portability, installation and debugging issues. Not fun. These diversions are a great way to kill enthusiasm.

I'll be investigating the portability (windows/linux and python 2/3) and usability of urwid next.  The "extra" test between python 2/3 roughly doubles my work. Alas, there is no way to avoid it.  Python 2 suffers about as many problems as 3.

One excellent (unexpected!) development was that defining just a few new ivars in the curses-related classes sufficed to eliminate all the __getattr__ methods!  I discovered this mostly by accident. Strange details omitted.

In hindsight, another way of getting to the present state might have been to base classes on the various "Null" classes in leoPy.leo#Code-->Gui base classes. But the __getattr__ traces have a great virtue.  They showed exactly what attributes Leo's startup code accessed and where the the startup code the accesses happened. Null classes could not have shown that.

Edward

[Edward K. Ream]

On Friday, April 21, 2017 at 3:58:49 AM UTC-5, Edward K. Ream wrote:

Yesterday was, mostly, a long digression into portability, installation and debugging issues. Not fun. These diversions are a great way to kill enthusiasm.

Importing urwid works in windows with 2/3.  On Linux, urwid is installed in home/local.../site-packages (python 2) which is on the path, and on /usr/local.../dist-packages (python 3), which isn't on my path.  This kind of thing is wearing me out.

Also, on Linux, pip is screaming at me to upgrade to 9.0.1, even after I have done so.

So I would like to try urwid on Windows, but neither pudb nor winpdb seem to work on Windows, due to problems with fctrl on Windows.  Can nobody simulate fctrl (pipes) on Windows?? Some days I am just fed up with it all...

For now, the way forward will be to duplicate the npyscreen code in urwid on Windows.  I don't need a debugger for that.  Then we'll see about dashed Linux install paths...

Edward

[Edward K. Ream]

On Friday, April 21, 2017 at 4:19:43 AM UTC-5, Edward K. Ream wrote:

> For now, the way forward will be to duplicate the npyscreen code in urwid on Windows.

The following code fails with both Python 2 and 3 on Windows:

    import urwid
    txt = urwid.Text(u"Hello World")
    fill = urwid.Filler(txt, 'top')
    loop = urwid.MainLoop(fill)
    loop.run() # On Windows, fails differently on 2 and 3

The failures on Windows involve either missing termios or fctrl, depending on python version.

So urwid is not going to get the job done on Windows.  A stack-overflow item recommends using cygwin on windows. That's conceivable, but for now I need a break from all of this. We can discuss options at the sprint...

Edward

[Edward K. Ream]

On Friday, April 21, 2017 at 4:19:43 AM UTC-5, Edward K. Ream wrote:

So I would like to try urwid on Windows, but neither pudb nor winpdb seem to work on Windows, due to problems with fctrl on Windows.

Alright.  Some real progress. winpdb works well on Leo as follows:

1.  In a console, do:

    rpdb2 -d -r launchLeo.py --gui=curses

This creates a server process which controls execution of the script.  Do not specify the python executable!

2. In a separate console, do:

    winpdb

3. Choose File:Attach in winpdb. This attaches winpdb to the server created by rpdb2.

Now you are ready to execute pdb-like commands from the command line in winpdb.  I just did g (go), and the first console executed as usual.

Actually, the first time I ran the go command there was a "window too small exception" which winpdb caught.  After detaching, the traceback appeared in the first (rpdb2) window.

This is pretty cool.  I actually would prefer not to use the winpdb gui.  At present, I haven't figured out how I might use rpdb2 in the second console.

Edward

[Edward K. Ream]

On Friday, April 21, 2017 at 5:47:56 AM UTC-5, Edward K. Ream wrote:

This is pretty cool.  I actually would prefer not to use the winpdb gui.  At present, I haven't figured out how I might use rpdb2 in the second console.

Ok.  Misleading 'help attach' and error message confused me for awhile.  At first, the proper form of attach is 'attach <password>', where the password is the one given in the first console.

attach <password> yields a list process id's.  Then attach <pid> attaches to the first console.  Now you can use the second rpdb2 pretty much like pdb to debug the script in the first console.  It's excellent.

Edward

[Edward K. Ream]

On Friday, April 21, 2017 at 5:56:29 AM UTC-5, Edward K. Ream wrote:

Ok.  Misleading 'help attach' and error message confused me for awhile.

Sorry, I mis-remembered what I had just done!

In the second console, first do:

    password <the password from the first console>

Then do:

    attach

to get a list of pid's. Then do:

    attach <pid>

to actually attach to the first console.

This does work, but alas rpdb2 doesn't print anything useful after single-stepping, so I'm always having to type l (list).  So using the winpdb gui version is probably best after all.

I've just tried to use rpdb2 in both consoles on Windows.  There is a problem connecting to the newly-created server on localhost.  Something about a timeout.  Not sure if this is a programming problem (fctrl?) or a firewall problem.  However, there is no great need to do debugging on Windows.

In short, urwid doesn't seem to work on Windows, but npyscreen runs anywhere and can be debugged on Linux with rpdb2 and winpdb.  I'd say this is a good enough resolution to the problem of what curses widgets to use.

Edward

[john lunzer]

Ahh, I'm sorry to have suggested urwid. I totally forgot about the Linux only limitation. 

[Kent Tenney]

It might be worth biting the bullet, buying a commercial IDE like
https://wingware.com/

[Kent Tenney]

or https://www.jetbrains.com/pycharm/

[john lunzer]

Blasphemy! Haha, jk. I too probably would have recommended pycharm if I were going to go with anything "pro". As much as I try not to give commercial software kudos they've got a good piece of software.

It's a shame the debugging experience on Windows isn't more fluid. I'm not sure how long I'd survive with Python on Linux without pudb. At least it wouldn't be nearly as fun. Another reason to lament urwid not working on Windows.

On Friday, April 21, 2017 at 8:14:26 AM UTC-4, Kent Tenney wrote:

or https://www.jetbrains.com/pycharm/

On Fri, Apr 21, 2017 at 7:11 AM, Kent Tenney <kte...@gmail.com> wrote:

It might be worth biting the bullet, buying a commercial IDE like
https://wingware.com/

On Fri, Apr 21, 2017 at 6:51 AM, john lunzer <lun...@gmail.com> wrote:

Ahh, I'm sorry to have suggested urwid. I totally forgot about the Linux only limitation. 

On Friday, April 21, 2017 at 7:24:11 AM UTC-4, Edward K. Ream wrote:

On Friday, April 21, 2017 at 5:56:29 AM UTC-5, Edward K. Ream wrote:

Ok.  Misleading 'help attach' and error message confused me for awhile.

Sorry, I mis-remembered what I had just done!

In the second console, first do:

    password <the password from the first console>

Then do:

    attach

to get a list of pid's. Then do:

    attach <pid>

to actually attach to the first console.

This does work, but alas rpdb2 doesn't print anything useful after single-stepping, so I'm always having to type l (list).  So using the winpdb gui version is probably best after all.

I've just tried to use rpdb2 in both consoles on Windows.  There is a problem connecting to the newly-created server on localhost.  Something about a timeout.  Not sure if this is a programming problem (fctrl?) or a firewall problem.  However, there is no great need to do debugging on Windows.

In short, urwid doesn't seem to work on Windows, but npyscreen runs anywhere and can be debugged on Linux with rpdb2 and winpdb.  I'd say this is a good enough resolution to the problem of what curses widgets to use.

Edward

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.

To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.

[Edward K. Ream]

On Friday, April 21, 2017 at 6:24:11 AM UTC-5, Edward K. Ream wrote:

In short, urwid doesn't seem to work on Windows, but npyscreen runs anywhere and can be debugged on Linux with rpdb2 and winpdb.  I'd say this is a good enough resolution to the problem of what curses widgets to use.

At last I see. There is a much easier shortcut to using rpdb2 or winpdb.  Just invoke them without arguments:

    rpdb2 launchLeo.py --gui=curses

or

   winpdb launchLeo.py --gui=curses

This launches rpdb2 or winpdb in the original console, and creates a second console to run the launchLeo.py. The server in the second console connects automatically with the client/debugger in the original console.  So everything "just works".

This is more than good enough. No need for commercial alternatives.

Edward

[Edward K. Ream]

On Fri, Apr 21, 2017 at 6:51 AM, john lunzer <lun...@gmail.com> wrote:

Ahh, I'm sorry to have suggested urwid. I totally forgot about the Linux only limitation. 

​Not a problem.  Getting up to speed with winpdb/rpdb2 makes everything worthwhile.

Edward