Delayed loading of large nodes

[Edward K. Ream]

More background for git bug 28: https://github.com/leo-editor/leo-editor/issues/28

There is no way around it: loading more than 10K lines of text into the body pane will be slow.

However, when moving around the outline, we often want just to move *past* a large node. This suggests the following strategy. 

1. Let w be a LeoQTextEditWidget. w.setAllText will load small text as usual.  For large text, w.setAllText will do the following:

- Put 'Loading...' in the body pane.
- Lock/disable w, preventing user edits.
- set w.leo_loading = True
- set w.leo_loading_p = w.c.p.copy()
- queue up w.load_text at idle time.

This has the potential for big problems.  Happily this scheme can probably be made safe as follows:

2. Define w.getAllText as follows:

    def getAllText(self):
        w = self.widget
        if w.leo_loading:
            return w.leo_loading_p.b
        else:
            return g.u(w.toPlainText())

3. At idle time, w.load_text will loads the text, but *only* if w.c.p == w.leo_loading.

In any case, w.load_text will clear w.leo_loading and w.leo_loading_p.

===== Conclusion

This looks like a viable strategy, but it will require substantial testing. There will have to be a delayed_load switch to enable/disable this feature.

Your comments, please.

Edward

[Edward K. Ream]

On Thu, Aug 28, 2014 at 8:20 AM, Edward K. Ream <edre...@gmail.com> wrote:
> More background for git bug 28:

...

> However, when moving around the outline, we often want just to move *past* a large node. This suggests the following strategy.

Important: this strategy conflicts with implementing w.getTextLength
using a textChanged event handler. In general, any textChanged
handler is going to complicate the way-too-complicated node switching
code. Best not to go there.

EKR

[Fidel N]

Edward, yes, the worst part of those kind of nodes is waiting for it to load when you are actually going to other node, or when you want to run a script on them but not read them.

Actually, just some short of "loading paused" button instead of a node body, such as when you click it, the node body will load, would be enough.

This way, you could still select given node to run scripts on it, but it wouldn't load the body/text unless you click the button.

If, on top of that solution, there is a "send to clipboard" option too, it would make mostly unnecessary even to load given nodes into Leo text widget, and you would still have most of any desired functionality (that I can think of). 

Because with those two described options, you can quickly edit them with your scripts, or copy and paste them in another editor to edit/view them. And you still can use Leo to keep / organize your information.

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To post to this group, send email to leo-e...@googlegroups.com.
Visit this group at http://groups.google.com/group/leo-editor.
For more options, visit https://groups.google.com/d/optout.

[Edward K. Ream]

On Thu, Aug 28, 2014 at 11:49 AM, Fidel N <fidel...@gmail.com> wrote:

> Edward, yes, the worst part of those kind of nodes is waiting for it to load
> when you are actually going to other node, or when you want to run a script
> on them but not read them.

If all you want to do is run a script on them, the script can just
find the position p *without* selecting it. Then just use p.b.

> Actually, just some short of "loading paused" button instead of a node body,
> such as when you click it, the node body will load, would be enough.

Great idea. Leo's body pane is a QTextBrowser, so "filling" the node
with one or two buttons should be possible. I'll prototype that
"today"(tm).

The crucial thing, from an internals point of view, remains that
w.getAllText returns p.b for "paused" nodes and not the (empty) text
that never got loaded ;-)

Edward

[Edward K. Ream]

On Thursday, August 28, 2014 12:07:36 PM UTC-5, Edward K. Ream wrote:

On Thu, Aug 28, 2014 at 11:49 AM, Fidel N <fidel...@gmail.com> wrote:

> Actually, just some short of "loading paused" button instead of a node body,
> such as when you click it, the node body will load, would be enough.

Great idea.  Leo's body pane is a QTextBrowser, so "filling" the node
with one or two buttons should be possible.  I'll prototype that
"today"(tm).

Well, "today" actually came today.

Here is a *prototype* that you can run from any Leo node.  Please, as an extra precaution, save your leo file before running this ;-)  And then, when in doubt, do *not* save the file after running the script.  Clear?

Running the script will hide the edit widget and show a "Load Text" button.  Click the button to restore things as they were.

Important: the button will remain until you click it, even if you change nodes.  But if you do change nodes, you will see the correct text for the new node under the button.

Regardless of the current node, clicking the button will take you back to the original node, and will restore the proper text there.  But don't try deleting the original node.  Hehe.  There are limits to the prototype!

----- start ----
from leo.core.leoQt import QtGui
edit = c.frame.body.widget.widget # A LeoQTextBrowser.
frame = edit.parent() # A QWidget
layout = frame.layout()

# Clear the text without clearing p.b
# The selecting lockout prevents setText from changing p.b.
s = p.b
c.frame.tree.selecting = True
edit.setText('')
c.frame.tree.selecting = False
assert p.b == s

b = QtGui.QPushButton('Load Text')

def onClicked(checked=False,p=p.copy(),s=s):
    c.selectPosition(p)
    edit.setPlainText(s)
    layout.addWidget(edit)
    layout.removeWidget(b)
    b.deleteLater()
    c.recolor_now()

b.clicked.connect(onClicked)
layout.addWidget(b)
layout.removeWidget(edit) # Put button in middle.
----- end -----

Edward

[Fidel N]

Wow, this was quicker than I would have expected.

Here is a question: 

The button will appear automatically instead of the text when the body contains 10k+ lines (or 5k maybe), right?

This solution is great!

Also, please consider adding also this button:

b = QtGui.QPushButton('Copy body to clipboard')

def onClicked(checked=False,p=p.copy(),s=s):
    c.gui.replaceClipboardWith(p.b)

b.clicked.connect(onClicked)
layout.addWidget(b)

[Edward K. Ream]

On Fri, Aug 29, 2014 at 2:44 AM, Fidel N <fidel...@gmail.com> wrote:

> The button will appear automatically instead of the text when the body
> contains 10k+ lines (or 5k maybe), right?

Correct. The number of characters (not lines) in the text that will
trigger this behavior will be a user-configurable option. Using
characters allows the test to use len(p.b) rather than p.b.count('\n')
which could be significant ;-)

> Also, please consider adding also this button:
>
> b = QtGui.QPushButton('Copy body to clipboard')

Thanks for the reminder. Will do. I used one button in the prototype
just to reduce the work. It took about an hour to arrive at the
final, relatively simple code. Hehe.

Edward

[Fidel N]

I have just been working with "semi-large" nodes, and noticed the increase in performance very much.

Now its not a pain at all to work with such nodes, thanks Edward :)

Also, didn't seem to notice the buttons yet. That is a work in progress, or should we see those already?

[Edward K. Ream]

On Sun, Sep 7, 2014 at 10:51 AM, Fidel N <fidel...@gmail.com> wrote:
> I have just been working with "semi-large" nodes, and noticed the increase
> in performance very much.
> Now its not a pain at all to work with such nodes, thanks Edward :)

You're welcome.

> Also, didn't seem to notice the buttons yet. That is a work in progress, or
> should we see those already?

I disabled it, maybe because I was bummed about QTextEdit generally.
I'll re-enable it now so you can play with it. Be *sure*, though, to
wait until all text is loaded. Otherwise any unloaded text will
disappear when you save the file.

I'll have more to say about working around the horrible QTextEdit bug
in another post, perhaps tomorrow.

Edward

[Edward K. Ream]

On Sun, Sep 7, 2014 at 2:34 PM, Edward K. Ream <edre...@gmail.com> wrote:

>> Also, didn't seem to notice the buttons yet. That is a work in progress, or
>> should we see those already?

Rev 6adccac now corrects some problems with the button code.

I consider this alpha code. The code is disabled by default.

Here is the checkin log:

QQQ
Fixed some problems with the big-text buttons.

To enable, set big_text_buttons = True at the start of leoFrame.py

Leo build: 20140907145404
QQQ

For testing, I use @int max-pre-loaded-body-chars = 1000

Let me know how this works for you.

Edward

[Fidel N]

Great, thanks Edward.

Going to be away for a few days, this is the first thing I'll do when Im back

Edward

[Edward K. Ream]

On Sunday, September 7, 2014 3:00:00 PM UTC-5, Edward K. Ream wrote:

> Rev 6adccac now corrects some problems with the button code. I consider this alpha code.  The code is disabled by default.

The code is still alpha-quality, but I have just enabled it.  It's time to test it before releasing 5.0 a1.

Here is the checkin log for 63578c0:

QQQ
Enabled "big-text" buttons.  These buttons appear in the body pane when the body text exceeds the limit given by the @int max-pre-loaded-body-chars

To disable these buttons, set @int max-pre-loaded-body-chars to zero.

Leo build: 20140917142108
QQQ

Please report any problems immediately.

Edward

[Fidel N]

it works great here, thanks for this Edward

With the update, also noticed the big blue square surrounding the selected widget, its also great. Those things improve usability big time IMO :D

--

[Edward K. Ream]

On Wed, Sep 17, 2014 at 3:03 PM, Fidel N <fidel...@gmail.com> wrote:
> it works great here, thanks for this Edward

Thanks for the report. Maybe the code is beta quality now ;-)

> With the update, also noticed the big blue square surrounding the selected
> widget, its also great. Those things improve usability big time IMO :D

I assume you mean the selected tree node. It's a new setting::

@color tree-item-selected-bg = @LightSteelBlue1

You can change the color by changing this setting in myLeoSettings.leo.

About colors: LightSteelBlue1 is defined in leoSettings.leo. As
mentioned in leoSettings.leo, you can see the list of available Qt
colors (case ignored by Qt) by running this script:

from leo.core.leoQt import QtGui
aList = sorted([g.u(z) for z in QtGui.QColor().colorNames()])
print('\n'.join(aList))

I'm not sure whether these colors depend on platform. It would be
good if Leo color settings automatically knew about the colors in
Leo's color database in leoColor.py, but at present they do not. Not
sure when that will happen...

Edward

[Edward K. Ream]

On Wednesday, September 17, 2014 3:03:27 PM UTC-5, Fidel N wrote:

it works great here, thanks for this Edward

Oops.  It appears to be impossible to cut or delete text from large nodes.

It's on my urgent list, but in the meantime I recommend setting::

    @int max-pre-loaded-body-chars = 0

to disable the big buttons feature.

Edward

[Zoltan Benedek]

Hi,

Unfortunately I have an issue with delayed loading:

https://github.com/leo-editor/leo-editor/issues/58

and I could not disable it by:

@int max-pre-loaded-body-chars = 0

[Edward K. Ream]

On Friday, September 19, 2014 10:05:47 AM UTC-5, Edward K. Ream wrote:

> Oops.  It appears to be impossible to cut or delete text from large nodes.

Fixed at rev 80e5693, Leo build: 20140919143910

The fix was simple, setting w.leo_big_text = None in two places, so perhaps the code is solid now.

Please let me know if it isn't...

Edward

[Zoltan Benedek]

Now the github 58 related issues are solved.

Thanks for the prompt fix.