Embedding An External Program In Leo - Proof Of Principle

[Thomas Passin]

I have been able to embed an external (non-Leo, non-Python) application in a Leo window. The attached screen shot shows an HP-42 Reverse Polish Calculator program running in Leo's main splitter. This program uses an image of real HP-42 hand-help calculator, as you can see. The calculator program is not my own work; it has been around for a long time. If I had known how to embed it, I might not have adapted the RPCalc code to use with Leo.  The HP-42 is much better.

This is a proof of principle and everything isn't quite right yet.  The calculator responds to mouse presses but it isn't getting the focus and so doesn't get keypresses as yet.

I'm not sure what kinds of external programs will be useful this way. A spreadsheet might  be interesting but they usually need to have a large window so it wouldn't work to confine it in part of Leo's window.  The calculator is useful because I sometimes want to calculate some numbers and it's convenient to do it within Leo.  You could do an image viewer but it would be tricky to get it to work with Leo's data. And there are one or more Leo scripts or plugins for viewing images already, including VR3 if you want to write down a bit of RestructuredText or Markdown. Or you can paste the full path to an image file into the body and CTRL-Click it. So there are many ways to handle viewing images already.

A small-size paint or drawing program might be interesting.

Anyway, I hope this post will stimulate some ideas.

[Thomas Passin]

I have solved the problem of the calculator not getting keypresses, and let me tell you it was tricky.  The problem is that Windows does not know that Qt thinks that *it* (Qt) is managing the focus.  Instead, The OS feeds mouse events directly to the calculator when it is visibly on top.  But it doesn't do that for keypresses. Here is how I worked around the problem:

When the calculator native window gets embedded into a Qt container object, we then overlay a transparent Qt widget over it. When Qt detects a mouse click on the overlay, it notifies the OS to focus the calculator window.  When the Qt focus switches away, Qt notifies the OS that the calculator window no longer has focus.

This actually works in my test program. I could not have done this without the help of ChatGPT, or at least it would have taken way longer and have been very frustrating. I had already learned that there are some incompatibilities that affected key handling between Qt-managed windows and OS-managed ones and that some people had been struggling with them.  ChatGPT knew about the OS/Qt focus incompatibility and it knew how to call the right Windows functions from within the program.  But it needed a lot of guidance from me, including the idea of the transparent overlay.  The whole story is a saga that I will post about separately.

I don't know how hard it will be to adapt the approach to Linux.  It probably won't be too hard for the X11 system, but more and more distros are changing over to Wayland and that is very different.  I'm not sure I will tackle it. Getting this HP calculator, which is Windows-only,  into Leo satisfies my immediate desire.

[jkn]

OT-ish: that view of the HP42s ('Free42' or whatever) looks an awful lot better there than it does on my android phone, for some reason...

[Thomas Passin]

Here is a screen shot of the corrected version of the embedded calculator program.  Notice that the calculator's native menu is now available, which is a pleasant improvement. In this version, the calculator program only receives mouse clicks and key presses when it has been focused by clicking on it. That was the tricky part to work out.

You could embed other programs just by changing the program window's display name and the path to its executable.  One restriction, which could be lifted, is that the external program's window must be non-resizable, as this calculator's is.

Because of good refactoring, you can see in the image that the main script is very simple.  The magic happens in the ForeignWindowEmbedder class.  

The calculator can be removed by running the Layout Demo's command layout-restore-default.

[Edward K. Ream]

On Sun, Sep 8, 2024 at 8:19 AM Thomas Passin <tbp1...@gmail.com> wrote:

I have been able to embed an external (non-Leo, non-Python) application in a Leo window.

Thanks for this. I'll say more about this project later.

Edward

[Edward K. Ream]

On Sun, Sep 8, 2024 at 8:19 AM Thomas Passin <tbp1...@gmail.com> wrote:

I have been able to embed an external (non-Leo, non-Python) application in a Leo window.

Thanks for this work! It's way cool.

I'm not sure what kinds of external programs will be useful this way.

Wouldn't it be great if Leo could communicate with NeoVim. If we could replace Leo's body pane with the NeoVim pane we might be able to get rid of Leo's Vim mode entirely!

Edward

[Thomas Passin]

I don't know NeoVim at all.  If it's a program that runs in its own window it can probably be encapsulated into a Qt widget.  The issue will be to get communication between Leo and the encapsulated window. The calculator program I used didn't need to communicate with Leo at all.

[Edward K. Ream]

On Thu, Oct 10, 2024 at 8:41 AM Thomas Passin <tbp1...@gmail.com> wrote:

I don't know NeoVim at all.  If it's a program that runs in its own window it can probably be encapsulated into a Qt widget.  The issue will be to get communication between Leo and the encapsulated window. The calculator program I used didn't need to communicate with Leo at all.

Exactly.

Edward

[Thomas Passin]

There is a NeoVim-Qt so someone has figured it out -

NeoVim-Qt

[Thomas Passin]

NeoVim has a NeoVim Python API. I could picture embedding an instance like I did with the calculator program and have Leo use its Python API.

[Edward K. Ream]

On Thu, Oct 10, 2024 at 1:24 PM Thomas Passin <tbp1...@gmail.com> wrote:

There is a NeoVim-Qt so someone has figured it out -

NeoVim-Qt

Cool. Maybe Vim-in-Leo is feasible! But it appears to require Qt5.

Edward

[Edward K. Ream]

On Thu, Oct 10, 2024 at 2:07 PM Thomas Passin <tbp1...@gmail.com> wrote:

NeoVim has a NeoVim Python API. I could picture embedding an instance like I did with the calculator program and have Leo use its Python API.

Care to have a go? Do you think Qt5 compatibility will be an issue?

Edward

[Thomas Passin]

I think they got it working with Qt6 too.  I haven't been able to tell from their web site. There could be another way forward, too.  NeoVim has a Python API, and it also has a remote API system.  If we were to embed a stock NeoVim window into a Qt widget, we could communicate with it over this interface.  We could do anything their scripting system will do, which means darn near anything.  It could be wrapped much as a LeoTextBrowser is wrapped.  From NeoVim docs:

"Nvim Python scripting is performed by an external host process implemented in ~2k lines of Python"

You can look at  some of the UI docs.

I've never been a VIM user, and whenever I look at it all I see is a lot of complication with little benefit to the way I use editors. So I may not be a good person to work on it.

[Edward K. Ream]

On Thu, Oct 10, 2024 at 3:37 PM Thomas Passin <tbp1...@gmail.com> wrote:

I think they got it working with Qt6 too.  I haven't been able to tell from their web site. There could be another way forward, too.  NeoVim has a Python API, and it also has a remote API system.  If we were to embed a stock NeoVim window into a Qt widget, we could communicate with it over this interface.  We could do anything their scripting system will do, which means darn near anything.  It could be wrapped much as a LeoTextBrowser is wrapped.  From NeoVim docs:

"Nvim Python scripting is performed by an external host process implemented in ~2k lines of Python"

You can look at  some of the UI docs.

I've never been a VIM user, and whenever I look at it all I see is a lot of complication with little benefit to the way I use editors. So I may not be a good person to work on it.

Thanks for this update.

I have little interest in the project either, but I might consider it if I get bored in my retirement :-)

Edward

[HaveF HaveF]

On Thursday, September 12, 2024 at 7:30:31 AM UTC+8 tbp1...@gmail.com wrote:

Here is a screen shot of the corrected version of the embedded calculator program.  Notice that the calculator's native menu is now available, which is a pleasant improvement. In this version, the calculator program only receives mouse clicks and key presses when it has been focused by clicking on it. That was the tricky part to work out.

You could embed other programs just by changing the program window's display name and the path to its executable.  One restriction, which could be lifted, is that the external program's window must be non-resizable, as this calculator's is.

Hi Thomas, I feel like I'm not following your thoughts, is there any benefit to embedding them this way, rather than putting them side by side directly?

[Thomas Passin]

By "embedding" here, I mean that we take a non-Qt program window and embed it into a certain kind of Qt widget.  Then we can put that widget where ever we want in our Qt program.  Side by side, whatever.  With a Qt widget A, we can attach it to another one, B, like a splitter, by making B be the "parent" of A.  If A has no parent, it will be a free-floating window.  With a non-Qt window like that calculator, we get its window ID from Windows, and use that to make a child of B. Now we can put it into our own program.  We haven't done anything yet to connect its signaling to the Qt system.

This is probably more than you care about, but in case it's interesting to someone, here's the core of how it's done.  I wrote a class called ForeignWindowEmbedder.  Here's the method that gets our calculator window and re-parents it into a Qt widget:

          # Embed the foreign window using its title
    def embed_foreign_window(self):
        self.hwnd = find_window_by_title(self.title)
        if self.hwnd:
            win32gui.SetParent(self.hwnd, int(self.winId()))

            # Modify the foreign window's style
            style = win32gui.GetWindowLong(self.hwnd,
                            win32con.GWL_STYLE)
            # Remove title bar (caption)
            new_style = style & ~win32con.WS_CAPTION
            win32gui.SetWindowLong(self.hwnd,
                            win32con.GWL_STYLE, new_style)

            # Move the foreign window to the 0, 0 corner of self
            win32gui.SetWindowPos(self.hwnd, None, 0, 0, 0, 0,
                    win32con.SWP_NOSIZE | win32con.SWP_NOZORDER)

self.winId()is the Windows ID of our  ForeignWindowEmbedder widget. With a Qt Widget, we would set its parent widget with setParent().  With a raw Windows object we use win32gui.SetParent()instead.

[HaveF HaveF]

On Sat, Oct 12, 2024 at 10:07 AM Thomas Passin <tbp1...@gmail.com> wrote:

By "embedding" here, I mean that we take a non-Qt program window and embed it into a certain kind of Qt widget.  Then we can put that widget where ever we want in our Qt program.  Side by side, whatever.  With a Qt widget A, we can attach it to another one, B, like a splitter, by making B be the "parent" of A.  If A has no parent, it will be a free-floating window.  With a non-Qt window like that calculator, we get its window ID from Windows, and use that to make a child of B. Now we can put it into our own program.  We haven't done anything yet to connect its signaling to the Qt system.

I see. Thanks for your input, Thomas!