@auto x.json and @auto-json now work

[Edward K. Ream]

2e8d9df4 completes #225, but see below.  This was straightforward, given the excellent prototype code.  Not sure whom to thank for it.

The .json format is a perfect match for Leo's uA's.  It takes just a few lines of code for the import/export code to save and restore uA's and gnx's.  Hmm. Speaking of gnx's, as I write this I see that the import code doesn't relink clones properly. I'll fix this soon.

Please report any other problems.

Edward

[Kent Tenney]

Leo Log Window
Leo 5.4-devel, build 20160412153848, Tue Apr 12 15:38:48 CDT 2016
Git repo info: branch = master, commit = 2e8d9df4bb39
Python 3.5.1, PyQt version 5.5.1
linux

Tried loading a large json file:
@auto data/lappy.lshw.json

Rclick -> Refresh from disk

reading: @auto data/albert.lshw.json
Exception running JSON_Scanner
Traceback (most recent call last):
File "/opt/fetching/leo-editor/leo/core/leoImport.py", line 665, in
scanner_for_ext_cb
return scanner.run(s, parent, prepass=prepass)
File "/opt/fetching/leo-editor/leo/plugins/importers/json.py", line 56, in run
root.setDirty(setDescendentsDirty=False)
NameError: name 'root' is not defined

> --
> You received this message because you are subscribed to the Google Groups
> "leo-editor" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to leo-editor+...@googlegroups.com.
> To post to this group, send email to leo-e...@googlegroups.com.
> Visit this group at https://groups.google.com/group/leo-editor.
> For more options, visit https://groups.google.com/d/optout.

[Edward K. Ream]

On Wed, May 4, 2016 at 10:49 AM, Kent Tenney <kte...@gmail.com> wrote:

Leo Log Window
Leo 5.4-devel, build 20160412153848, Tue Apr 12 15:38:48 CDT 2016
Git repo info: branch = master, commit = 2e8d9df4bb39
Python 3.5.1, PyQt version 5.5.1
linux

Tried loading a large json file:
@auto data/lappy.lshw.json

​...​

 

NameError: name 'root' is not defined

​Should be fixed at 0b0369d7e, but this probably won't help you.

At present, @auto x.json will only restore properly when the @auto tree was first written with Leo.  So it's one-way initially: from Leo to .json.​

 

​After that, @auto will work.

I see no way to fix this: json is not an outline format.  How is Leo supposed to know where to break things, or for that matter, what things mean?  Where is the AI to come from?

Edward

[Kent Tenney]

On Wed, May 4, 2016 at 11:43 AM, Edward K. Ream <edre...@gmail.com> wrote:
>
>
> On Wed, May 4, 2016 at 10:49 AM, Kent Tenney <kte...@gmail.com> wrote:
>>
>> Leo Log Window
>> Leo 5.4-devel, build 20160412153848, Tue Apr 12 15:38:48 CDT 2016
>> Git repo info: branch = master, commit = 2e8d9df4bb39
>> Python 3.5.1, PyQt version 5.5.1
>> linux
>>
>> Tried loading a large json file:
>> @auto data/lappy.lshw.json
>
>
> ...
>
>>
>> NameError: name 'root' is not defined
>
>
> Should be fixed at 0b0369d7e, but this probably won't help you.
>
> At present, @auto x.json will only restore properly when the @auto tree was
> first written with Leo. So it's one-way initially: from Leo to .json.

Ah, didn't know that. I value Leo so much as a tool to aid in making
sense of complexity via it's capability to parse on import, I was
hoping the same would be available here.

>
> After that, @auto will work.
>
> I see no way to fix this: json is not an outline format. How is Leo
> supposed to know where to break things, or for that matter, what things
> mean? Where is the AI to come from?

Naively:
As I understand json's relationship to Python:
json.loads(file.read()) returns a dict consisting of dicts, lists,
strings, bools

I thought maybe these could be translated to nodes and children somehow,
@key as organizer nodes, lists as child nodes representing @string,
@bool and @key

As I said, *naive* :-]

I do have interest in json, and have made use of the kind of logic
discussed here
http://stackoverflow.com/questions/12507206/python-recommended-way-to-walk-complex-dictionary-structures-imported-from-json

I'm using it to help extract interesting components from the json
files I'm dealing with,
interesting if not relevant.

Thanks,
Kent

[Edward K. Ream]

On Wed, May 4, 2016 at 10:38 AM, Edward K. Ream <edre...@gmail.com> wrote:

2e8d9df4 completes #225, but see below.  This was straightforward, given the excellent prototype code.  Not sure whom to thank for it.

​...​

as I write this I see that the import code doesn't relink clones properly. I'll fix this soon.

​Fixed at 0b0369d7e.

EKR

[Edward K. Ream]

​On Wed, May 4, 2016 at 12:34 PM, Kent Tenney <kte...@gmail.com> wrote:

> At present, @auto x.json will only restore properly when the @auto tree was
> first written with Leo.  So it's one-way initially: from Leo to .json.

Ah, didn't know that. I value Leo so much as a tool to aid in making
sense of complexity via it's capability to parse on import, I was
hoping the same would be available here.

​It won't be available automatically, that is, via static commands or (probably) @auto.  However, it would be straightforward to write a script that would create an outline based on outline structure, per your desires.

Furthermore, it would be possible to create a customizable class that would make writing such scripts a simple matter of filling in some vars/settings.

> json.loads(file.read()) returns a dict consisting of dicts, lists, strings, bools.

Correct. Programming is a snap because  all the parsing has already been done. It's also easy to create Leo nodes on the fly.  For example, is a simplified version (ignoring clones/traces) of JSON_Scanner.create_nodes, in leo/importers/json.py:

def create_nodes(self, parent, parent_d):
    '''Create the tree of nodes rooted in parent.'''
    c, d = self.c, self.gnx_dict
    for child_gnx in parent_d.get('children'):
        d2 = d.get(child_gnx)
        child = parent.insertAsLastChild()
        child.h = d2.get('h') or '<**no h**>'
        child.b = d2.get('b') or g.u('')
        if d2.get('gnx'):
            child.v.findIndex = d2.get('gnx')
        if d2.get('ua'):
            child.u = d2.get('ua')
        self.create_nodes(child, d2)

Parent is the parent position, parent_d is a dict. As you can see, the programming is just a matter of handling dict contents.

The more general case would be more tedious because we have to discover whether values are dicts, lists or something else. And unless we are willing to have all lists and dicts to produce children, we will want to specify which dictionary keys or list items are to create descendants.  A little messy, but clearly doable.

Here is where the helper class comes in.  It would contain all the general code, asking the "user" to tell it what elements are to create outline structure, which elements should be ignored, and which elements are to create headline text, body text, or even may uA's.

Kent, let's try this.  Take a .json file that interests you, and consider how you would like to have it imported into a Leo file. No need to spend lots of time on this--just give a shot.  Then send me the file and your thoughts about how you would like it imported.  I'll write a JSON_Importer class to make writing such a script easier.  It will be kinda like the class RecursiveImportController class.

Edward

[Kent Tenney]

That's a very good fit with a project I'm working on,
currently I'm overwhelmed at work to near-panic levels,
it won't be right away.

Thanks,
Kent

> child.v.findIndex = gnx = d2.get('gnx')

> if d2.get('ua'):
> child.u = d2.get('ua')
> self.create_nodes(child, d2)
>
> Parent is the parent position, parent_d is a dict. As you can see, the
> programming is just a matter of handling dict contents.
>
> The more general case would be more tedious because we have to discover
> whether values are dicts, lists or something else. And unless we are willing
> to have all lists and dicts to produce children, we will want to specify
> which dictionary keys or list items are to create descendants. A little
> messy, but clearly doable.
>
> Here is where the helper class comes in. It would contain all the general
> code, asking the "user" to tell it what elements are to create outline
> structure, which elements should be ignored, and which elements are to
> create headline text, body text, or even may uA's.
>
> Kent, let's try this. Take a .json file that interests you, and consider
> how you would like to have it imported into a Leo file. No need to spend
> lots of time on this--just give a shot. Then send me the file and your
> thoughts about how you would like it imported. I'll write a JSON_Importer
> class to make writing such a script easier. It will be kinda like the class
> RecursiveImportController class.
>
> Edward
>

[Edward K. Ream]

On Wed, May 4, 2016 at 3:32 PM, Kent Tenney <kte...@gmail.com> wrote:

That's a very good fit with a project I'm working on,
currently I'm overwhelmed at work to near-panic levels,
it won't be right away.

​No problem.  I'll be doing my own researches.

EKR

[Jose Gonzalez]

A possibility could be to specify a schema for a given json file indicating how to structure the document. That would expand the possibilities for Leo to handle flat-file hierarchical datasets for applications like bookmarks manager, bibliographies, etc. A similar concept is implemented in http://treeline.bellz.org, I believe using (less human readable) XML files. Just an idea...

[Edward K. Ream]

On Wed, May 4, 2016 at 10:07 PM, Jose Gonzalez <de0...@gmail.com> wrote:

A possibility could be to specify a schema for a given json file indicating how to structure the document.

​An interesting idea. ​

 

​Perhaps there are already tools that import json based on json schema.  I'll look into this.​

That would expand the possibilities for Leo to handle flat-file hierarchical datasets for applications like bookmarks manager, bibliographies, etc.

​I'm less interested in this. We don't have problems with data once we get it into Leo ;-)

EKR

[Edward K. Ream]

On Thursday, May 5, 2016 at 4:09:06 AM UTC-5, Edward K. Ream wrote:

On Wed, May 4, 2016 at 10:07 PM, Jose Gonzalez <de0...@gmail.com> wrote:

A possibility could be to specify a schema for a given json file indicating how to structure the document.

​An interesting idea. ​

 

​Perhaps there are already tools that import json based on json schema.  I'll look into this.​

After a quick check, I don't think json schema are what we want.  Instead, we want a description of json formats as they apply to Leo. That is, we want to tell Leo which fields should create children, and similar information. Naturally, existing json schema tools have no knowledge of Leo, and so can't be used.

Yes, we could represent the Leonine information using json, but that's wonky and unnecessary. Instead, my original idea of providing data to a class that would help import json files seems like the simplest thing that could possibly work.

EKR