Where have you been? Stuck at the command line.

[john lunzer]

My job has me primarily stuck SSHing into workstations remotes over not so great connections, so -X forwarding is not really an option. A lot of this is manually processing data with in-house tools and workflow automation tasks.

As a result I've been primarily been stuck with CLI only editors. My obvious choice is between vim and emacs. I used to use vim but I feel like I've recently groked emacs so I'm using that now.

Regardless. I am desperate and saddened by not having Leo. I so badly desire to walk through code in a proper outline. As many know emacs has Org mode but it is too "bulky" for trying to emulate what Leo does. I have some intuition that built-in to the core of emacs are all the tools necessary to create a "Leo-mode" but I certainly don't have the time right now to investigate. Long term, I'm trying to contemplate if it would be easier to write a limited CLI front-end for Leo or if it would be easier to write a limited Leo style outlining mode in emacs. They both seem daunting.

I'm not trying to manipulate anyone into somehow getting Leo onto the command line. I'm just reporting in from deep on the front lines as somebody who loves and appreciates Leo and misses it sorely. 

[Edward K. Ream]

On Mon, Apr 17, 2017 at 2:16 PM, john lunzer <lun...@gmail.com> wrote:

> ​stuck SSHing into workstations remotes...I am desperate and saddened by not having Leo. I so badly desire to walk through code in a proper outline.

Interesting problem.  Let's see what we can do...

I've never done any SSHing, so correct me if I'm wrong.  Is the problem that your external files are on another machine and it would be too expensive to copy them over the link to a machine with Leo on it?

Edward

[Terry Brown]

On Mon, 17 Apr 2017 12:16:46 -0700 (PDT)
john lunzer <lun...@gmail.com> wrote:

> I'm not trying to manipulate anyone into somehow getting Leo onto the
> command line.

If you can manipulate someone into doing this, please do ;-) I'd love
to have command line Leo too. I work on marginal connections too -
just quick enough to make putting up with laggy X11 forwarding doable,
but tty would be nice.

> I'm just reporting in from deep on the front lines as
> somebody who loves and appreciates Leo and misses it sorely.

You've probably thought of these things, and they're much easier if
your local machine is linux, not Windows, but you could sshfs mount the
remote machines and use Leo locally. Or look at Jake's @sftp plugin,
which is a sort of push / pull approach.

I think the "what does it take to write alternative interfaces for Leo"
discussion is worthwhile. Command line, web based, the edit pane
component I've been working on... all variations on that theme.

I think the tricky bits are things like key handling and abbreviations
and so on. But not everything has to be supported everywhere.

Cheers -Terry

[Edward K. Ream]

On Monday, April 17, 2017 at 2:30:55 PM UTC-5, Terry Brown wrote:

I think the "what does it take to write alternative interfaces for Leo"
discussion is worthwhile.

Another topic for Ashland, perhaps.

Edward

[lewis]

Not sure this is relevant but there is a cursesGui.py

Many of the menu options are not fully functional.

Lewis

[Edward K. Ream]

On Mon, Apr 17, 2017 at 8:51 PM, lewis <lewi...@operamail.com> wrote:

Not sure this is relevant but there is a cursesGui.py

Many of the menu options are not fully functional.

​That would be where I would start, but a proper job would be a lot more work.

Edward

[Edward K. Ream]

The wikipedia page for text guis has a lot of interesting stuff.  I'll play around with it a bit today.

Edward

[Edward K. Ream]

On Tuesday, April 18, 2017 at 5:57:38 AM UTC-5, Edward K. Ream wrote:

> The wikipedia page for text guis has a lot of interesting stuff.

More importantly, Python has a curses module (Linux only). This should suffice for the present task. Actually creating the gui is another matter, but I'll see if I can do a hello world app today.

Edward

[Edward K. Ream]

On Tuesday, April 18, 2017 at 6:10:19 AM UTC-5, Edward K. Ream wrote:

> I'll see if I can do a hello world app today.

'''hello_curses.py: Hello world in curses.'''
import curses as c
import time
w = c.initscr()
w.insstr('Hello World')
w.refresh()
time.sleep(2)
c.endwin()

To repeat, this would be a major project.  I'm not going down this rabbit hole any time soon.

Edward

[john lunzer]

For about half of my SSH work my files are on what is called the "primary network", which is accessible to all windows workstations (where I sit) and a few authorized more powerful and capable Linux workstation/servers. When I'm accessing these particular remote Linux workstations through PuTTY, I do use Leo because my machine has access to the same network shares that the Linux servers do.

In many large corporations it is difficult to get a new piece of hardware blessed to join the "primary network", but the pace of business is such that we have to use new hardware far before jumping through the hoops of getting something hardened by security. In these cases we will set up an "offline" network (usually in labs) which connects to workstation already on the "primary network". The idea is that if the other workstations do not have direct access to the internet and they're only being remotely accessed through hardware that's already been locked down by IT. 

A lot of my recent work has been on Linux workstations/servers on these offline networks. These require two levels of SSH to pass through. SSH to secured computer on corporate network, then SSH to offline workstations. In fact I usually have several PuTTY terminals open at once, one of them is always the node the "offline" computers are connected to. I then painstakingly haul over everything I need by means of scp/rsync.

So the whole thing is a bit ugly and setting up network file sharing to the "offline" hardware with NFS/sshfs starts to get sketchy from a security standpoint. 

Initially I attempted to do all my editing on files on the "primary" network and then scp/rsync them over to the "offline" workstations but I quickly lost the discipline due to the need for quick edits, instead opting for terminal based editing facilities.

That's where I'm at, one of the only solutions I can think of longterm is having a localized (offline) terminal based outlining editing tool. I think there are only two candidates, Leo + full terminal UI or emacs + leo-mode. I think they both involve a lot of work. 

I think ideally Leo would gain a full terminal UI. I'm personally waiting for Jonathan Slenders to finish python-prompt-toolkit 2.0 because he is rewriting huge portions of it to make it easier to write full screen terminal applications. He had already the 0.x and 1.x versions to write PyMux and PyVim but really he was the only one capable of coercing python-prompt-toolkit to producing full screen terminal applications.

[john lunzer]

Of course Edward, I tried to make it clear in original post that in now way was I trying to manipulate/trick you into taking this on. It's a huge project. I think you've got important core work to do that doesn't involve writing GUIs. Terry alluded to this but the only thing I would say, perhaps as a discussion for your sprint, is simply that you consider the current interface for writing front-ends in Leo. 

I think if I were going to try to coerce you into anything it would be in reviewing the boundary between Leo's core and Leo's Qt interface to ensure that it is fully generic/decoupled so that it becomes as easy as possible for others beside you to write front-ends for whatever comes along. From my limited attempts to understand this interface it looked like the line might have blurred a bit over the years, but that might also be me just not having the expertise with either Leo or Qt to know where the line is exactly. Blurry line or blurry vision? Not sure.

Despite you being a constant with Leo for decades I think we all know all good things must come to an end at some point. My only wish, when you leave, is that Leo's abstraction layers are clear, clean, concise. That way when you're gone perhaps a team of 5 or 6 of us can hobble enough effort together to keep Leo alive (by writing new parsers for new languages and new GUIs for whatever popular framework is available).

[john lunzer]

I will investigate sshfs and the sftp plugin. Given my two levels of SSH I'm not sure if either would be workable but I will check.

[Terry Brown]

On Tue, 18 Apr 2017 05:06:08 -0700 (PDT)
john lunzer <lun...@gmail.com> wrote:

> I think if I *were* going to try to coerce you into anything it would

> be in reviewing the boundary between Leo's core and Leo's Qt
> interface to ensure that it is fully generic/decoupled so that it

> becomes as easy as possible for *others beside you* to write

> front-ends for whatever comes along. From my limited attempts to
> understand this interface it looked like the line might have blurred
> a bit over the years, but that might also be me just not having the
> expertise with either Leo or Qt to know where the line is exactly.
> Blurry line or blurry vision? Not sure.

Very well put - like you I'm not sure, when I struggle with issues in
this area, whether it's the structure or my understanding of it that
makes some things seem challenging.

Cheers -Terry

[Edward K. Ream]

On Tuesday, April 18, 2017 at 7:06:08 AM UTC-5, john lunzer wrote:

I think if I were going to try to coerce you into anything it would be in reviewing the boundary between Leo's core and Leo's Qt interface to ensure that it is fully generic/decoupled

Imo, it's decoupled enough.  All, or almost all, of Leo's core interacts with the gui via the high-level interface.

See leoPy.leo#Code-->Gui base classes-->@file leoFrame.py-->API classes-->class WrapperAPI

and

leoPy.leo#Code-->Gui base classes-->@file leoFrame.py-->API classes-->class TreeAPI

A curses gui for Leo that implemented these APIs would be pretty complete.  Of course, there are other issues, namely implementing some kind of event structure.

The redraw_after methods may be the biggest hack. They are a performance optimization, and like many optimizations can be difficult to understand.  I have no opinion about whether they all could be replaced by a full redraw.

Edward

[Edward K. Ream]

​​

On Tue, Apr 18, 2017 at 7:28 AM, Terry Brown <terry...@gmail.com> wrote:

Very well put - like you I'm not sure, when I struggle with issues in
this area, whether it's the structure or my understanding of it that
makes some things seem challenging.

​An interesting conversation.  Thinking about this is a welcome break from fixing seemingly endless fit-and-finish problems. In particular, I think it would be interesting to design an event handling mechanism.  Terry said this earlier:

"I think the tricky bits are things like key handling and abbreviations and so on.  But not everything has to be supported everywhere.

"

If done properly, abbreviations should "just work", because they use the high-level interface.  Key handling starts in the gui code and then passes a LeoKeyEvent to the rest of Leo. The initial Qt key-handling code is very complex because it's hard/impossible to recover the intended keyboard key given an Qt key event.  Perhaps the curses key handling is simpler.

But once the gui creates a LeoKeyEvent and passes it to k.masterKeyHandler, the rest of Leo's key-handling machinery should "just work".  That's the power of Leo's existing gui interface.

I've brought up new guis several times now.  The MO I use is simply to define a new gui and then fire Leo up. It will crash.  Python will report what the problem is. Fix the crash. Repeat until no more crashes.

There are a limited number of wrapper functions, so this is actually a reasonable thing to do.  It's fine to define lots of do-nothing placeholders at first.

Just now I realized that there is an other possible path: define curses replacements for all Qt widgets.  To do this, one would "trick" the code in leoQt.py so it "redirects" to what could be called QtCurses modules. The advantage is that one could use most or all the Qt gui code unchanged. The big drawback is that one would have to simulate all the Qt calls.  Probably only useful as a thought experiment :-)

Either way, it's a big job.  For instance, syntax coloring the body pane would require much of the "smarts" of the behind-the-scenes QSyntaxHighlighter code.  The attic contains an implementation of this class in Python. (It's a long story). But even this code assumes it is living in the Qt world.

Finally, there is something really important I want to emphasize. Leo has stable API's and interfaces, but even the best API's and interfaces only slightly mitigate the inherent complexity of porting a large program like Leo.  What matters more than prissy "standards" are powerful tools.  Happily, this project has two incredibly powerful tools at hand.

1. Python.  Python catches almost all errors instantly. Fixing crashes until there aren't any is a real candidate for the quickest way to get a new gui up and running.

2. Leo. In particular, the cff (clone-find-flattened) command is unique in all the known universe for managing (not just searching for) all the details that are going to be involved.  cff, when applied to leoPy.leo, is a grep over all of Leo's sources.  Once in awhile I have to do cff in leoPlugins.leo, but that probably won't be necessary when porting a gui.

I'll be demonstrating cff at the sprint. There is no tool like it.  It's a snap to find and organize all calls to x (cff def x) , all assignments to  (regex cff x\s*=) and anything else you want. The recent cleanups to leoAtFile.py could not have been done without cff.  cff is the truth.  Documentation, standards, and API's are mostly half truths.  Imo, cff is Leo's most compelling feature.

Edward

[Edward K. Ream]

On Tuesday, April 18, 2017 at 6:32:27 AM UTC-5, Edward K. Ream wrote:

'''hello_curses.py: Hello world in curses.'''
import curses as c
import time
w = c.initscr()
w.insstr('Hello World')
w.refresh()
time.sleep(2)
c.endwin()

This also works on Windows 10, provided one installs the proper binary from this page. You can get to this page by googling "python _curses" which takes you to this stack overflow page.

Edward

[Eric S. Johansson]

On Tuesday, April 18, 2017 at 7:50:30 AM UTC-4, john lunzer wrote:

A lot of my recent work has been on Linux workstations/servers on these offline networks. These require two levels of SSH to pass through. SSH to secured computer on corporate network, then SSH to offline workstations. In fact I usually have several PuTTY terminals open at once, one of them is always the node the "offline" computers are connected to. I then painstakingly haul over everything I need by means of scp/rsync.

I have a similar need for different reason. Because of my disability and use of speech recognition, I'm bound to Windows. However, I do all my work on Linux systems. What I want is for Leo to trigger an rsync to a remote host and then, somehow trigger an install script to update the production environment such as a Web server. The second stage is usually required to cross security perimeters, fix permissions, ownership etc. 

The first stage would require a plug-in that implements "I saved in this directory, this directory means that machine target, synchronize to that machine target".

The second stage requires an inotify reaction routine that knows how to do the appropriate transformations on a file or directory to a given location. 

Obviously, there would be a lag in copying and synchronization that hopefully wouldn't be too bad. I may take a stab at the second stage soon because this would make my development life easier. The place to start is a way of representing the file/directory and destination PUG (permission, user, group). I did something like this years ago I don't know if I still have the code.

John, I suspect you could use something similar except you would need a second and third stage. Where the second stage simply replicates (another rsync).

[john lunzer]

What I've done in the short term is rather than using rsync I use fossil because it is cleaner and gives me reliable two way updating capabilities. 

I set up a fossil repo in a "online" location, where ever I'm keeping my files I want to keep synced. Then I clone the repo using fossil's ssh clone capabilities individually on the "offline" machines. fossil saves the "remote url" of where the clone came from. 

A "fossil sync" command at any cloned repo will request the most up to date versions of the files (as well as new files added to the repo) from where they were cloned from and also attempt to merge in any local changes. 

I still have to cascade updates between the second and third level. But I suppose if the main repo was on the node with access to both the "online" and "offline" network I wouldn't have to do the cascade. I'll toy with that soon. Each clone of the repo has a "remote-url" setting which allows you to point each clone to another clone to affect the topology of the repo distribution. The "master" clone has an empty "remote-url".

Using RSA keys it should be possible to set this up so that the machines don't ask each other for passwords, but in keeping with security policies I've opted to simply input my password every time I update.

fossil is cross platform to it shouldn't be an issue to set this up between linux and windows workstations.

I've also been thinking of ways to mix entr (linux only) into this to try to automate things further.

[Terry Brown]

On Tue, 18 Apr 2017 05:17:56 -0700 (PDT)
john lunzer <lun...@gmail.com> wrote:

> I will investigate sshfs and the sftp plugin. Given my two levels of
> SSH I'm not sure if either would be workable but I will check.

Not sure if you've looked into ssh's port forwarding capabilities, but
that will usually take care of any multi-hop issues.

I.e to get to C from A via B, tell your A->B session to forward port
2222 on A to port 22 on C. Then you can scp / sshfs stuff from A to C
on port 2222. I can give examples if useful.

A couple of other technologies I've found useful in restrictive
environments, shellinabox, an app you run on a server to provide
terminal login (ssh) from a browser, and Apache Guacamole, browser
based VNC / RDP server / client. Might not be relevant to you though.

Cheers -Terry