Images In Leo Nodes?

[Thomas Passin]

It would be very useful if an image could be embedded into a Leo body.  Probably no one would want to do this in a program or script, but for documentation it could be very helpful.  You can put an image into a ReST, MD, or Asciidoc node and view it with VR3 or by creating Sphinx output.  But even better would be to have an image in an ordinary text page.

In theory it should be possible to put an image into a Leo node.  Qt has a way to do it, and I have written a little test Qt program that can put one into a QTextBrowser or a QTextEdit widget at the cursor location.

The way Qt does it is to insert a special Unicode character:

"Inline images are represented by a Unicode value U+FFFC (OBJECT REPLACEMENT CHARACTER) which has an associated QTextImageFormat."

I *think* that Qt converts an image file to a data: url and stores it in that character.  Aside from the issue of how to implement commands to let a user work with an image (have text reflow around it, delete it, etc), there is the question of how to save and restore the image  data along with the rest of the outline.

One possibility is that saving the file will simply include the data: url's data (which would be encoded as base64, so it should not be a problem to save), and the file load will simply restore it with no extra effort.  This is probably too much to hope for.

If that turns out not to happen, maybe it would not be too hard to save the data as a UA, and restore it from the UA.  The problem, I think, is to get the data into that special character in the right place.  Another possibility would be to write Leo code to restore the image from the original image file.  That sounds harder to me, but I'm not sure.  It would be better not to have to depend on an external image file if possible.

I'm very interested in this, and I hope we can have a good discussion about it!

[Jacob Peck]

Not what you’re really looking for, but Leo already supports @svg nodes.  Of course, those are just the image in a node, viewable in VR — no text around them in the same node, to my knowledge.

I’ve done kinda-sorta similar things with one of my personal use ‘leo apps’.  In that use case, I had a new pane that was effectively a rendered-to-html view of a running markdown document that would be appended to as various actions happened — including embedding images.  It included a button to swap out the webview widget for a textedit widget that contained the markdown source.  So kind of like a livecoding-styled viewrendered panel.  It was overkill but it had data that needed to live ‘outside’ the outline it was in.

Anyway, there are many approaches to this.  It would be interesting to see, but I’m not sure it’s really something that core Leo needs to support.  A plug-in would be pretty neat though.

Jake

On Mar 28, 2023, at 6:04 PM, Thomas Passin <tbp1...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/7723eb4d-8495-4107-8c6a-dd79180479a7n%40googlegroups.com.

[Thomas Passin]

On Tuesday, March 28, 2023 at 8:59:01 PM UTC-4 gates...@gmail.com wrote:

Not what you’re really looking for, but Leo already supports @svg nodes.  Of course, those are just the image in a node, viewable in VR — no text around them in the same node, to my knowledge.

Right, and VR3 has even more capability for mixing images with text.  You have to be willing to view the material as ReST, MD, etc, or export the rendered view to a browser.  I'm thinking of basic documentation.  Right now, I'm writing some documentation or a Leo app I've written. I would like to embed images, especially screen shots, in a way that doesn't require a reader to do anything special.  Just read from one node to the next.

I’ve done kinda-sorta similar things with one of my personal use ‘leo apps’.  In that use case, I had a new pane that was effectively a rendered-to-html view of a running markdown document that would be appended to as various actions happened — including embedding images.  It included a button to swap out the webview widget for a textedit widget that contained the markdown source.  So kind of like a livecoding-styled viewrendered panel.  It was overkill but it had data that needed to live ‘outside’ the outline it was in.

I know what you mean.  The Freewin plugin does something very similar.  It can switch between a live TextEdit view of a node and a rendered ReST view.  Each Freewin window is node-locked, which yours probably was not.

 

Anyway, there are many approaches to this.  It would be interesting to see, but I’m not sure it’s really something that core Leo needs to support.  A plug-in would be pretty neat though.

I'd be willing - I think - to go with a plugin, but either way there are things to work out.

[Edward K. Ream]

On Tue, Mar 28, 2023 at 5:04 PM Thomas Passin <tbp1...@gmail.com> wrote:

It would be very useful if an image could be embedded into a Leo body.  Probably no one would want to do this in a program or script, but for documentation it could be very helpful.

... 

In theory it should be possible to put an image into a Leo node.  Qt has a way to do it, and I have written a little test Qt program that can put one into a QTextBrowser or a QTextEdit widget at the cursor location.

The way Qt does it is to insert a special Unicode character:

I played around with this a year (or 5?) ago. I kinda worked, but I seem to remember there were problems. Probably something to do with reloading the images when reloading Leo or re-visiting nodes.  Naively I would think those problems could be solved just by searching for the special character. I have no idea why I gave up.

In short, I think this would be a great addition to Leo. Let's give it a go.

Edward

[Thomas Passin]

In an internet search I found that saving would probably best be done by walking through all the blocks on a page, of which the image would be one.  During restoring a body, I suppose that one would insert the blocks one by one, though I am less clear about how to do that.  It seems to me that if an image were stored as a data: url, the image bytes could be stored as text in a UA, so everything could be included in the Leo outline.

So it may be feasible!

I have attached a small package with my demo experiment.  Unzip the zip file, open the Leo outline, and run its external file in a console.  Click in the Qt window that opens - nothing will seem to happen but it will locate the cursor in the frame.  Then click the pushbutton.  The image will get inserted at that point.  You can do this more than once to insert at different places.  When the image is inserted, the program will print the contents of the widget to the console.  You will notice a weird character - that is the special unicode character that can contain the image data.

The program requires PyQt6, but you could edit to use PyQt5 if that's what you have installed.

[Thomas Passin]

I've made a little progress.  That unicode character  - 0xfffc - is only a marker and does not carry any information about the image.  But behind the scenes, Qt keeps the image data and associates it with that marker.  The actual image bits are stored in Qt's "resources" for the page, and identified by a string.  One way to get the image data is to have the QTextEditor/QTextBrowser give you the contents as markdown or html, then parse that for image identifiers.  With the identifiers we can get the images from the resource system (though I don't know all the details for doing that yet).  The image data can be converted to a base64 string, and we could store that string in the node's UAs.  When the outline is loaded, new code could look through the UA data to see what image data to get and where to put it, then insert the images.  We should probably include a flag so that we can avoid spending time checking nodes that don't have any image data.

One drawback would be that if an external file were edited outside of Leo, the images might end up in the wrong places or even get deleted by mistake.  But my guess is that the kind of files that might get edited outside of Leo won't have any images anyway.

[Edward K. Ream]

On Sat, Apr 22, 2023 at 3:26 PM Thomas Passin <tbp1...@gmail.com> wrote:

I've made a little progress.  That unicode character  - 0xfffc - is only a marker and does not carry any information about the image. 

Iirc, the lack of identifying data with the 0xfffc character was why I abandoned this project years ago. It would be natural to allow body text to contain more than one image. Ordinary editing operations could change the order in which those images appear (or, as you say, delete images).

You're correct, Leo could use uAs to associate images with 0xfffc characters initially, but I see no good (easy) way of maintaining those associations after all of Leo's editing operations.

Edward

[Thomas Passin]

In principle, there could be a way, but I don't know how practical it would be.  We have the widget give us the md (or html) version and search through it for all the image tags.  This gives us the resource identifiers and the 0xffc characters in serial order through the document.  Back in the text view, we can find them all and insert the images in order where their markers are.  There ought to be a less hacky way to do this, but I haven't found anything yet on stack overflow etc, or in the api docs.  It's probably there somewhere if I can grasp what it all means.

[Edward K. Ream]

On Saturday, April 22, 2023 at 4:51:13 PM UTC-5 Edward K. Ream wrote:

Iirc, the lack of identifying data with the 0xfffc character was why I abandoned this project years ago. It would be natural to allow body text to contain more than one image. Ordinary editing operations could change the order in which those images appear (or, as you say, delete images).

You're correct, Leo could use uAs to associate images with 0xfffc characters initially, but I see no good (easy) way of maintaining those associations after all of Leo's editing operations.

I've just reread Thomas's comment in #3256. It suggests using the @image directive. Leo already supports this directive. Instead, Leo could support a new @embedded-image directive. This directive would specify a path to an image file in the usual way (relative to the outline, with support for @path, etc.).

This plan should work:

- When loading/reloading the body pane, Leo would scan for @embedded-image directives and add the "magic byte" for Qt.

- Leo's onBodyChanged event handler would do the same.

In short, Leo could easily update the correspondence between magic bytes and images. There is no need for uAs.

Edward

[Thomas Passin]

uAs would avoid the need to put an image file in another directory, where it could get separated from the outline.  This separation (and a need for uAs) could be avoided if we had an archive format for Leo outlines.  Maybe images could be cached so they would reload faster while Leo has the outline open.

As for the magic byte, Qt is going to put it there when the image is inserted, and that byte is going to be associated with a private bitmap stored invisibly by Qt outside of the text content.  So these bytes will need to be managed on every reload.

We would also want to make sure that the image does not get re-inserted unless the magic byte becomes moved away from the position of the @embedded-image directive.  Otherwise I suspect that the latency after a few keystrokes would become annoying, especially if a node contained several images.

[Edward K. Ream]

On Sun, Apr 23, 2023 at 7:19 AM Thomas Passin <tbp1...@gmail.com> wrote:

> uAs would avoid the need to put an image file in another directory, where it could get separated from the outline.  This separation (and a need for uAs) could be avoided if we had an archive format for Leo outlines.  Maybe images could be cached so they would reload faster while Leo has the outline open.

I think this is a separate issue.

> As for the magic byte, Qt is going to put it there when the image is inserted, and that byte is going to be associated with a private bitmap stored invisibly by Qt outside of the text content.  So these bytes will need to be managed on every reload.

I agree.

> We would also want to make sure that the image does not get re-inserted unless the magic byte becomes moved away from the position of the @embedded-image directive.  Otherwise I suspect that the latency after a few keystrokes would become annoying, especially if a node contained several images.

A straightforward prototype should reveal whether there are any such problems.

Edward

[Thomas Passin]

Once we get embedded images working well, we can go after tables.

[Edward K. Ream]

On Sun, Apr 23, 2023 at 9:07 AM Thomas Passin <tbp1...@gmail.com> wrote:

Once we get embedded images working well, we can go after tables.

Yes.

Edward