Leo Node Move Error

[k-hen]

I'm getting an error moving a node in Leo. It's a very large/deep node if that matters.

This large node, X, is/was a root. I added a new root node, Y, then tried to move X under Y which resulted in the attached error.

I'm using the sqlite DB back-end if that matters.

This same behaviour has happened twice. The first time it deleted X and all descendants the undo wasn't able to bring it back :-(

I reverted to a backup and tried it again and am getting the same error.

Kevin

[Edward K. Ream]

On Wed, Sep 2, 2020 at 7:38 AM k-hen <percepti...@gmail.com> wrote:

I'm getting an error moving a node in Leo. It's a very large/deep node if that matters.

Thanks for this report. I have just created #1639 for this issue.

This large node, X, is/was a root. I added a new root node, Y, then tried to move X under Y which resulted in the attached error.

How exactly did you add Y? Did you use insert-node-before?

Usually, adding Y (say with Ctrl-I) would not create a new root. After creating Y, you would then have to move Y to the root, which in this case would likely init Leo's data structures properly.

Edward

[percepti...@gmail.com]

I had root nodes like the following, I selected node B and inserted a new node below, i.e. using insert-node Y.  

Then I attempted to move X under Y. Note that I repeated this and also tried to move deep nodes within X at lower levels which also resulted in the error.

Node X currently has ~6,700 descendant nodes.

A

B

X

D

Kevin

(I'll also add this to the Git ticket)

--
You received this message because you are subscribed to a topic in the Google Groups "leo-editor" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/leo-editor/sANduRjZDVI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAMF8tS1j42g_U6rs5b0sUO9kAaVG7ys998R-uN1yma%3D6t4tejg%40mail.gmail.com.

[Edward K. Ream]

On Wed, Sep 2, 2020 at 8:14 AM <percepti...@gmail.com> wrote:

I had root nodes like the following, I selected node B and inserted a new node below, i.e. using insert-node Y.  

Then I attempted to move X under Y. Note that I repeated this and also tried to move deep nodes within X at lower levels which also resulted in the error.

Node X currently has ~6,700 descendant nodes.

Thanks for the clarification. A note of terminology. The "root" node is the first top-level node. I think you mean "top-level" node instead of root node.

If you can, please send me the smallest node X that demonstrates the bug.

Edward

[percepti...@gmail.com]

Thanks re: the clarification, yes they are 'top-level' then.  

Unfortunately I'm unable to send this _particular_ outline due to client confidentiality, 

but am happy to run whatever code or tests that you'd like me too.

--

You received this message because you are subscribed to a topic in the Google Groups "leo-editor" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/leo-editor/sANduRjZDVI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to leo-editor+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAMF8tS3MiyXu6xx3nvaU3gzfYRRoK2WhyV6tO3ni85NPCSUK-w%40mail.gmail.com.

[Thomas Passin]

I would have moved it by collapsing the node, then <CTRL-u> until it was under the intended parent, then <CTRL-r> to make it a child of the target.  I'm not saying you did it "wrong", just that this is the sequence that seems simplest and has always worked well for me.

On Wednesday, September 2, 2020 at 9:14:14 AM UTC-4, k-hen wrote:

I had root nodes like the following, I selected node B and inserted a new node below, i.e. using insert-node Y.  

Then I attempted to move X under Y. Note that I repeated this and also tried to move deep nodes within X at lower levels which also resulted in the error.

Node X currently has ~6,700 descendant nodes.

A

B

X

D

Kevin

(I'll also add this to the Git ticket)

On Wed, 2020-09-02 at 08:08 -0500, Edward K. Ream wrote:

On Wed, Sep 2, 2020 at 7:38 AM k-hen <percepti...@gmail.com> wrote:

I'm getting an error moving a node in Leo. It's a very large/deep node if that matters.

Thanks for this report. I have just created #1639 for this issue.

This large node, X, is/was a root. I added a new root node, Y, then tried to move X under Y which resulted in the attached error.

How exactly did you add Y? Did you use insert-node-before?

Usually, adding Y (say with Ctrl-I) would not create a new root. After creating Y, you would then have to move Y to the root, which in this case would likely init Leo's data structures properly.

Edward

--
You received this message because you are subscribed to a topic in the Google Groups "leo-editor" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/leo-editor/sANduRjZDVI/unsubscribe.

To unsubscribe from this group and all its topics, send an email to leo-e...@googlegroups.com.

[percepti...@gmail.com]

To be clear, these are just the top level nodes and everything is collapsed. 

Y is a new top-level node created right above X, then X is indented right under its parent.

I think we're saying the same thing.

[Edward K. Ream]

On Wed, Sep 2, 2020 at 8:35 AM <percepti...@gmail.com> wrote:

Thanks re: the clarification, yes they are 'top-level' then.  

Unfortunately I'm unable to send this _particular_ outline due to client confidentiality, 

but am happy to run whatever code or tests that you'd like me too.

It would help if you would run a script to remove all body text (it doesn't matter) and replace all headlines with something innocuous. That way you can send me the problematic outline without compromising privacy.

I'll write this script soon and publish it here. You can then run the script on a copy of you file, and send the result to me.

Edward

[Edward K. Ream]

On Wednesday, September 2, 2020 at 9:04:18 AM UTC-5, Edward K. Ream wrote:

> It would help if you would run a script to remove all body text (it doesn't matter) and replace all headlines with something innocuous. That way you can send me the problematic outline without compromising privacy.

Here is the tested script:

"""Remove all sensitive material in this outline, except for this node."""
# Warning: This script is dangerous! Use on a copy of your outline.
n = 1
v0 = c.p.v
for p in c.all_unique_positions():
    if p.v != v0:
        p.h = f"node {n}"
        p.b = ''
        n += 1
c.redraw()

Please run this script inside a copy of you file, and send the result to me.

Edward

[percepti...@gmail.com]

Ok, got it, will need a bit of time, but will get back to you soon.

Thanks! :-) 

Kevin

--

You received this message because you are subscribed to a topic in the Google Groups "leo-editor" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/leo-editor/sANduRjZDVI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to leo-editor+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/3d6a2dc6-d63f-493f-9554-e70eb1223124o%40googlegroups.com.

[Edward K. Ream]

On Wednesday, September 2, 2020 at 3:24:45 PM UTC-5, k-hen wrote:

Ok, got it, will need a bit of time, but will get back to you soon.

Something strange is going on.  Here is what I think I know:

The unpacked original file is leo_error_test_01 - cleared - Copy.db. It is about 28.1 MB in size.

Opening the file with Leo does work, but the two "node 5660" are not shown as clones of each other. Yet they have the same gnx and id(v) is the same for each. Imo, this is likely the ultimate cause of the crash.

Saving this .db as a .leo file creates a valid .leo file in which the two "node 5660" are indeed true clones. The bug never appears in the .leo file.

Saving the .leo file as a .leo.db file creates a much smaller file, about 1.2 MB. However, in all other respects it appears to be identical to the original .db file. The two "node 5660" are not shown as clones of each other, and the bug does manifest itself.

My tentative conclusion is that there may be a bug in fc.retrieveVnodesFromDb. However, there are no special cases in this code, so what the bug could possibly be is a big mystery. Iirc, Vitalije wrote this code. Any insights would be appreciated.

Finally, I note that the .leo files are smaller than the corresponding .db files, so one workaround would be to use the .leo files instead.  Presumably, however, there are reasons for using the .db files.

Edward

[percepti...@gmail.com]

Hmm ... well at least it seems reproduceable.  I did run across at least one set of nodse after filing this that were behaving as clones but not flagged as clones, and it's probably the same ones. Not sure if it matters but the box icon was grey instead of black, I thought that was unusual as well, but not sure what that indicates 

I'm using the sqlite version because I assumed it could offer better performance for very large outlines (e.g. potentially 50K+ nodes). 

Also, because I like SQL and can connect to it with DBVisualizer/alternate tools and write queries (read only ones for now) and/or use ETL workflows with it. 

One thing that _might_ be happening is that I have a 'code' top-level tree that syncs with other developers using Git.

It's using @file's and so there are now comments scattered throughout the code base. 

Then subnodes within this tree are cloned into @nosent  'documentation' trees which contains other notes/commentary outputs.

This is mostly doing what I want, but importing the nodes sometimes results in errors/broken outlines/etc, which I have to repair.

I close Leo, do the pull, then open Leo again and let it import. 

Then I fix any issues (mostly nodes losing their parents and appearing at the root).

I also keep an eye on Git to ensure that I'm not making any unwanted changes after doing this and can revert if necessary.

It's possible that other developers are copying/moving blocks of code that include the Leo comments (I've asked them not to do this).

We're making a lot of structural changes now too as we're refactoring so this should settle down later.

For now, perhaps I can save as .leo and then save back as .leo.db to potentially repair any issues.

Is it perhaps possible to detect and/or repair these broken clones using a script?

Thanks so much for all your help.

Kevin

--
You received this message because you are subscribed to a topic in the Google Groups "leo-editor" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/leo-editor/sANduRjZDVI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to leo-editor+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/9afd5fe4-7fd3-4e68-91d7-283f02367601o%40googlegroups.com.

[k-hen]

As a follow up:

I figured out the grey box, it's just the unsaved/dirty nodes, so it's not that.

Regarding the file size for the DB, I assume this is because the version I sent you doesn't have any content.

So I think this is a legitimate size for the doc, but it probably needs to be vacuumed/cleaned up to reduce it back.

https://sqlite.org/lang_vacuum.html

The Leo version is 16MB which is a bit smaller but not quite as dramatic.

Kevin

[vitalije]

My tentative conclusion is that there may be a bug in fc.retrieveVnodesFromDb. However, there are no special cases in this code, so what the bug could possibly be is a big mystery. Iirc, Vitalije wrote this code. Any insights would be appreciated.

I am almost 100% sure that the fc.retrieveVnodesFromDb works correctly. It just restores outline from the rows retrieved from db. Saving and loading from db work in tandem. If the outline being saved has broken links, then the outline loaded afterwards will have broken links too.

Most likely outline got broken links during some modification and later was saved as such.

Here is a script that can fix broken links:

from collections import defaultdict
def relink_outline(c):
    '''Normalizes outline fixing broken links'''
    parDict = defaultdict(list)
    for p in c.all_positions():
        parDict[p.gnx].append(p._parentVnode())
    gnx2v = c.fileCommands.gnxDict
    for gnx, parents in parDict.items():
        v = gnx2v[gnx]
        v.parents = parents
    c.checkOutline()
relink_outline(c)

When saving to xml information about parent links is omitted. When loading from xml file, correct parent links are infered again. Perhaps I can change writing to db to omitt parent links and just infer them when loading from db.

It is still a mistery how k-hen got his outline to this invalid state. My guess is that it was caused during the updating clones in both @file files and @nosent files. That would explain why we haven't noticed this bug before. It is not very common to have shared clones in both @nosent and @file files.

Vitalije

[Edward K. Ream]

On Fri, Sep 4, 2020 at 4:52 AM vitalije <vita...@gmail.com> wrote:

My tentative conclusion is that there may be a bug in fc.retrieveVnodesFromDb. However, there are no special cases in this code, so what the bug could possibly be is a big mystery. Iirc, Vitalije wrote this code. Any insights would be appreciated.

I am almost 100% sure that the fc.retrieveVnodesFromDb works correctly. It just restores outline from the rows retrieved from db. Saving and loading from db work in tandem. If the outline being saved has broken links, then the outline loaded afterwards will have broken links too.

This seems like the most likely explanation to me as well.

Most likely outline got broken links during some modification and later was saved as such.

Here is a script that can fix broken links:

from collections import defaultdict
def relink_outline(c):
    '''Normalizes outline fixing broken links'''
    parDict = defaultdict(list)
    for p in c.all_positions():
        parDict[p.gnx].append(p._parentVnode())
    gnx2v = c.fileCommands.gnxDict
    for gnx, parents in parDict.items():
        v = gnx2v[gnx]
        v.parents = parents
    c.checkOutline()
relink_outline(c)

Thanks for this script.   It should be possible for c.checkOutline to catch the kinds of problems we have been discussing.  It could even compare the links before and after running relink_outline!

When saving to xml information about parent links is omitted. When loading from xml file, correct parent links are infered again. Perhaps I can change writing to db to omitt parent links and just infer them when loading from db.

Interesting. Please let us know what you conclude.

It is still a mistery how k-hen got his outline to this invalid state. My guess is that it was caused during the updating clones in both @file files and @nosent files. That would explain why we haven't noticed this bug before. It is not very common to have shared clones in both @nosent and @file files.

Yes, it's still a mystery. Many thanks for your helpful and insightful comments.

Edward

[percepti...@gmail.com]

Thank you for this :-)

I realize that @nosent is nontraditional. I'd be happy to use @adoc or @md but I currently don't know of a way to identify nodes that aren't section headlines.

Once the outline is too deep then the doc fails  :-/  Nosent doesn't write the headlines so it makes this possible but I agree that it's not very elegant. 

--
You received this message because you are subscribed to a topic in the Google Groups "leo-editor" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/leo-editor/sANduRjZDVI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to leo-editor+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/56aa7952-a09d-4da5-b176-206916448e26o%40googlegroups.com.

[percepti...@gmail.com]

For the record, unfortunately this script didn't fix the issue - it still wasn't happy.

I had to convert to .leo xml and then back to .db. 

I think there's still a lot of value in fixing the DB format, but hopefully in a way that won't limit it's potential performance.

On Fri, 2020-09-04 at 02:52 -0700, vitalije wrote:

--
You received this message because you are subscribed to a topic in the Google Groups "leo-editor" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/leo-editor/sANduRjZDVI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to leo-editor+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/56aa7952-a09d-4da5-b176-206916448e26o%40googlegroups.com.

[Edward K. Ream]

On Tue, Sep 8, 2020 at 10:35 AM <percepti...@gmail.com> wrote:

For the record, unfortunately this script didn't fix the issue - it still wasn't happy.

I had to convert to .leo xml and then back to .db. 

I think there's still a lot of value in fixing the DB format, but hopefully in a way that won't limit it's potential performance.

Thanks for this report.

Edward