New writer plugins in leo/plugins/writers

[Edward K. Ream]

By analogy with the plugin importers in leo/plugins/importers, Leo will soon support plugin write code in leo/plugins/writers.

There are writer plugins for .md (markdown), .org, .otl and .rst files.  They are remarkably simple.

These plugins are untested, and not yet hooked into Leo's core. Coming later today, perhaps...

**Important**: there is no need for separate writer plugins for most kinds of files.  The standard @file write code will suffice because Leo knows about the comment delimiters, etc. for each language.  At present, such data is hard-coded in tables in leoApp.py, but Leo will soon build those tables from something like:

    leo/plugins/data/language_data.py.

Users will be able to tell Leo about new languages simply by adding a new dictionary to language_data.py.

Edward

[Terry Brown]

On Sat, 26 Jul 2014 08:32:10 -0700 (PDT)
"Edward K. Ream" <edre...@gmail.com> wrote:

> By analogy with the plugin importers in leo/plugins/importers, Leo
> will soon support plugin write code in leo/plugins/writers.
>
> There are writer plugins for .md (markdown), .org, .otl and .rst
> files. They are remarkably simple.
>
> These plugins are untested, and not yet hooked into Leo's core.
> Coming later today, perhaps...
>
> **Important**: there is no need for separate writer plugins for most
> kinds of files. The standard @file write code will suffice because
> Leo knows about the comment delimiters, etc. for each language. At
> present, such data is hard-coded in tables in leoApp.py, but Leo will
> soon build those tables from something like:

Interesting, until this post I hadn't realized the modules you were
making weren't both readers and writers.

I thought I had a use case where I wanted language specific reading and
writing, e.g. writing comments like

### setOfLetters #################################################

in the output and having those disappear into Leo node names in Leo,
but get written out that way again on write. But now the use case
isn't clear in my mind - if the node names are just "function" names
(or whatever named units the language in question defines), then it
seems you're right, Leo only needs to know the language on read, write
is trivial but for the edge cases in whitespace and indentation of
course.

So it's only organizer nodes that need special consideration, not sure
where they fall for other languages / @persistance / @shadow etc.

I guess the use case I was thinking of is the situation you see in R
statistical analysis scripts sometimes where there's a long string of
steps and it's nice to organize them into a *flat* list of named nodes,
which could be written out / read in in a comment format like the
above, but there isn't a language element carrying the names.
Sort of non-hierarchical organizer nodes.

Having said all that, there are writer modules where needed, so, best
of both worlds :-)

Cheers -Terry

[Edward K. Ream]

On Sat, Jul 26, 2014 at 11:19 AM, 'Terry Brown' via leo-editor
<leo-e...@googlegroups.com> wrote:

> Interesting, until this post I hadn't realized the modules you were
> making weren't both readers and writers.

I guess I was thinking in a code-oriented way. files in
leo/plugins/importers replace code in leoImport.py, while writers (and
readers?) replace code in leoAtFile.py. As a practical matter, I think
the distinction must remain.

> Having said all that, there are writer modules where needed, so, best
> of both worlds :-)

I think so, but this is a tricky area, so we'll have to see.

It remains to be seen just how easy it will be to hook up readers(?)
and writers. Haven't done that yet, and I strongly suspect that ugly
implementation details will complicate things. Stay tuned.

Edward

[Edward K. Ream]

On Sunday, July 27, 2014 8:58:40 AM UTC-5, Edward K. Ream wrote:

On Sat, Jul 26, 2014 at 11:19 AM, 'Terry Brown' via leo-editor
<leo-e...@googlegroups.com> wrote:

> Having said all that, there are writer modules where needed, so, best
> of both worlds :-)

I think so, but this is a tricky area, so we'll have to see.

It turns out that the simplest, most useful things to do are following:

1. There will be no readers, just importers.

Leo's import code is now *fully* driven by tables created at startup using the data in the importers_dict dictionary in each .py file in leo/plugins/importers. The basescanner.py file has no such dict because it only defines the BaseScanner class used by all the other importers.

2. For now, only the existing writers are needed.

**Important**: There are many more importers than writers because each importer must parse the incoming text.  In contrast, we only need writers for languages that define their own outline structures: we can use the "default" @auto *write* code for all other language, because the default @auto write code needs to know only the language delims in effect for the language.

In short, for now and for the foreseeable future, the importers and writers will support just @auto.

Limiting importers/writers plugins to @auto greatly simplifies the task of supporting the plugins.  In particular, only at.writeOneAtAutoNode needs to "dispatch" writers based on the various flavors of @auto that it is trying to write.  Crucially, writer plugins can be blissfully *unaware* of all the horribly complex "setup" and "teardown" code in at.writeOneAtAutoNode. Furthermore, writers just call to at.os and at.nl!  (The setup code inits the file-like object used by these two methods.)  It seems we have stumbled upon an amazingly simple interface for writers...

In principle, one could imaging replacing *all* of Leo's @<file> write code in leoAtFile.py by plugins, but that would *hugely* complicate each writer's job, without accomplishing much. How likely is it that a user will want (or be able!) to revise Leo's write code for @file, @shadow, @nosent etc.?  Even if we wanted this "flexibility", the chance of going horribly wrong would be great.  If I were designing this code from scratch, I might be tempted to go this route, but not now. I am far from confident that I could refactor the code without introducing subtle time-bombs in the read/write code.  Keeping the write code *exactly* as it is (in its effect) is much more important than making the code look cleaner.  This is so important.  Understand?

Edward

P.S.  This post will be pre-writing for readme.txt and howto.txt files to be placed in the plugins/importers and plugins/writers folders.  The howto.txt files will explain in detail how to create new importers and writers.

EKR

[Terry Brown]

On Sat, 26 Jul 2014 08:32:10 -0700 (PDT)
"Edward K. Ream" <edre...@gmail.com> wrote:

> By analogy with the plugin importers in leo/plugins/importers, Leo
> will soon support plugin write code in leo/plugins/writers.

This all looks very good.

readme.txt and howto.txt -> README.txt and HOWTO.txt would move them
helpfully to the start of the directory listing.

The code should probably search ~/.leo/plugins/importers and
~/.leo/plugins/writers as well, if it doesn't already.

The writers seem good, i.e. the simplest thing that could possibly
work (TM) :-)

leo/plugins/importers/basecanner.py isn't in the repo. at the moment,
which makes it hard to evaluate ;-) but I will anyway :-)

It looks like it's defining a set of variables (ivars) which define how
to import something. In addition to that, I think there should be an
altnerative level in the API where the signature is something like:


def build_tree(root, filename):
"""build_tree - Import the structured data in ``filename`` to a tree
starting in node ``root``, which will be called `@auto-??? filename`.

Raises LeoImporterFail Exception if it fails, returns nothing.

:Parameters:
- `root`: vnode root of tree
- `filename`: filename (full path) containing structured data.

Note: some data is stored in multiple files (foo.par, foo.xll) so
passing the path to the "core" file rather than a file object makes
it easier for the delegate code to calculate additional paths and
access additional data.
"""

<|code|>


Two languages which seem like they'd be easier to handle this was are
LaTeX and ctext. ctext is a language I just made up but I'm excited to
use :-) it's only structural element is a comment line

### Node name #######################################################

where the comment is always 75 chars long, and indentation level
is indicated by the number of # at the start, three being an
indentation level of zero. This would be great for writing R and SQL
scripts which are essentially long lists of steps, but being able to
manage them better in Leo, and still share them with others without
weird sentinels.

LaTeX of course uses [[[sub]sub]sub]section to indicate structure.

Also, while I suspect no one but me uses it, xml_edit.py can actually
round trip data between XML and a Leo outline in a highly structured
manner, and it would be very nice to hook it up to @auto-xml instead of
having to manually use its import and export commands.

Maybe it's just lack of familiarity on my part, but "use python" seems
to be an easier way to write an importer in these cases than setting up
the base_scanner ivars.

Anyway, it think the new @auto-??? importer/writer selection approach
is going to be great for letting Leo quickly adapt to specific use
cases, perhaps leading to situations where people will be saying "the
easiest way to edit these ??? files is to install Leo and use its
@auto-??? mode".

Cheers -Terry

[Terry Brown]

On Mon, 28 Jul 2014 10:50:14 -0500

"'Terry Brown' via leo-editor" <leo-e...@googlegroups.com> wrote:

> Note: some data is stored in multiple files (foo.par, foo.xll) so
> passing the path to the "core" file rather than a file object
> makes it easier for the delegate code to calculate additional paths
> and access additional data.

Of course (a) this applies to the writer too, and (b) the path
information could just be made available through inherited methods /
ivars in the importer / writer subclasses, so passing a file object
would be fine.

Cheers -Terry

[Edward K. Ream]

On Mon, Jul 28, 2014 at 10:50 AM, 'Terry Brown' via leo-editor
<leo-e...@googlegroups.com> wrote:

> readme.txt and howto.txt -> README.txt and HOWTO.txt would move them
> helpfully to the start of the directory listing.

Ha. So that's why the weird capitalization...

> The code should probably search ~/.leo/plugins/importers and
> ~/.leo/plugins/writers as well, if it doesn't already.

Good idea!

> leo/plugins/importers/basecanner.py isn't in the repo. at the moment,
> which makes it hard to evaluate ;-) but I will anyway :-)

It is now. It used to be in leoImport.py.

> It looks like it's defining a set of variables (ivars) which define how
> to import something.

These vars are really just hacks that allow subclasses to override
various behaviors without having to override large, complex methods.

> In addition to that, I think there should be an alternative level in the API where the signature is something like:

I'm not really following you here.

> Two languages which seem like they'd be easier to handle this way are
> LaTeX and ctext. [snip]

You are welcome to override BaseScanner.scan (or any of its helpers)
if you like ;-) That way you have complete control over the scanning
process.

However, the real "excitement" comes in the code generation phase.
This code creates the Leo outline. It's complex because it must be
very careful about indentation and whitespace.

The BaseScanner class also has way-too-complex checking code.

You could override any part of scanning or code generation or checking
if you think there is an easier way. If you find such a way, please
let me know. Heh, heh.

> Anyway, it think the new @auto-??? importer/writer selection approach
> is going to be great for letting Leo quickly adapt to specific use
> cases, perhaps leading to situations where people will be saying "the
> easiest way to edit these ??? files is to install Leo and use its
> @auto-??? mode".

I'm glad you think so.

Edward

[Terry Brown]

On Mon, 28 Jul 2014 15:46:27 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

> On Mon, Jul 28, 2014 at 10:50 AM, 'Terry Brown' via leo-editor
> <leo-e...@googlegroups.com> wrote:

[snip]

> > In addition to that, I think there should be an alternative level
> > in the API where the signature is something like:
>
> I'm not really following you here.
>
> > Two languages which seem like they'd be easier to handle this way
> > are LaTeX and ctext. [snip]
>
> You are welcome to override BaseScanner.scan (or any of its helpers)
> if you like ;-) That way you have complete control over the scanning
> process.
>
> However, the real "excitement" comes in the code generation phase.
> This code creates the Leo outline. It's complex because it must be
> very careful about indentation and whitespace.

I'm not looking for excitement ;-) I'm looking for Leo to say "here's
the filename, here's the node to stick the subtree in, now make the
subtree". Well, ok, I guess that would be exciting for the flexibility
it offers :-)

I thought maybe just over-riding .run() would do it, but I'm getting
something wrong:

.../leo/git/leo-editor/leo/plugins/importers/test.py:

from basescanner import BaseScanner

class TestScanner(BaseScanner):
def __init__(self, *args, **kwargs):
print 'ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ'
return BaseScanner.__init__(self, *args, **kwargs)
def run(self,s,parent,parse_body=False,prepass=False):
print 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
return True
def scan (self,s,parent,parse_body=False):
print 'YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY'
return True

importer_dict = {
'@auto': ['@auto-test',],
'class': TestScanner,
'extensions': ['.txt',],
}


the modules being loaded because if I add a syntax error it dies, but
none of the XYZ strings are ever printed. My test .leo has a
@auto-test ~/r/foo.txt node, but it behaves just like an @auto node,
the file does get loaded, if the code above was active, I assume it
wouldn't be loaded. I tried .txxt as an extension in both places to,
same result (in case .txt was conflicting with something).

Cheers -Terry

[Edward K. Ream]

On Mon, Jul 28, 2014 at 10:26 PM, 'Terry Brown' via leo-editor
<leo-e...@googlegroups.com> wrote:

> I'm not looking for excitement ;-) I'm looking for Leo to say "here's
> the filename, here's the node to stick the subtree in, now make the
> subtree". Well, ok, I guess that would be exciting for the flexibility
> it offers :-)

Try this:

ic = c.importCommands
ic.createOutline(fileName,parent,
atAuto=True,atShadow=False,s=None,ext=None)

Note 100% sure about the atAuto arg. If s is given, it will be taken
as the contents of the file. The ext arg can be used to force a given
importer.

When in doubt, look at the sources for createOutline. I've recently
refactored them, so they should be more understandable.

Edward

[Terry Brown]

On Tue, 29 Jul 2014 04:52:57 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

> On Mon, Jul 28, 2014 at 10:26 PM, 'Terry Brown' via leo-editor
> <leo-e...@googlegroups.com> wrote:
>
> > I'm not looking for excitement ;-) I'm looking for Leo to say
> > "here's the filename, here's the node to stick the subtree in, now
> > make the subtree". Well, ok, I guess that would be exciting for
> > the flexibility it offers :-)
>
> Try this:
>
> ic = c.importCommands
> ic.createOutline(fileName,parent,
> atAuto=True,atShadow=False,s=None,ext=None)

Ok, so that forced it to use my importer, but it failed like this:

File "/mnt/usr1/usr1/home/tbrown/Package/leo/git/leo-editor/leo/core/leoCommands.py", line 2473, in executeScript
c.executeScriptHelper(args,define_g,define_name,namespace,script)

File "/mnt/usr1/usr1/home/tbrown/Package/leo/git/leo-editor/leo/core/leoCommands.py", line 2511, in executeScriptHelper
execfile(scriptFile,d)

File "/home/tbrown/.leo/scriptFile.py", line 7, in <module>
atAuto=True,atShadow=False,s='testtest',ext=None)

File "/mnt/usr1/usr1/home/tbrown/Package/leo/git/leo-editor/leo/core/leoImport.py", line 560, in createOutline
func(atAuto=atAuto,parent=p,s=s)

File "/mnt/usr1/usr1/home/tbrown/Package/leo/git/leo-editor/leo/core/leoImport.py", line 611, in scanner_for_at_auto_cb
scanner = aClass(importCommands=self,atAuto=atAuto)

File "/mnt/usr1/usr1/home/tbrown/Package/leo/git/leo-editor/leo/plugins/importers/test.py", line 6, in __init__
return BaseScanner.__init__(self, *args, **kwargs)

TypeError: __init__() takes at least 4 arguments (3 given)

Cheers -Terry

[Edward K. Ream]

On Tue, Jul 29, 2014 at 8:50 AM, 'Terry Brown' via leo-editor
<leo-e...@googlegroups.com> wrote:

> File "/mnt/usr1/usr1/home/tbrown/Package/leo/git/leo-editor/leo/plugins/importers/test.py", line 6, in __init__
> return BaseScanner.__init__(self, *args, **kwargs)

You must not return a value from a ctor.

EKR

[Edward K. Ream]

On Tue, Jul 29, 2014 at 9:13 AM, Edward K. Ream <edre...@gmail.com> wrote:

> You must not return a value from a ctor.

leoPluginsRef.leo contains the sources for all importers & writers.
Following those patterns may help.

EKR

[Terry Brown]

On Tue, 29 Jul 2014 09:13:29 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

Whoops, true of course. So now my importer is:

from basescanner import BaseScanner
print 'TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT'
class TestScanner(BaseScanner):
#def __init__(self, *args, **kwargs):
# print 'ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ'
# BaseScanner.__init__(self, *args, **kwargs)

def run(self,s,parent,parse_body=False,prepass=False):
print 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
return True
def scan (self,s,parent,parse_body=False):
print 'YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY'
return True

importer_dict = {
'@auto': ['@auto-test',],
'class': TestScanner,
'extensions': ['.txt',],
}

and I'm still getting

Traceback (most recent call last):

File "/mnt/usr1/usr1/home/tbrown/Package/leo/git/leo-editor/leo/core/leoCommands.py", line 2473, in executeScript
c.executeScriptHelper(args,define_g,define_name,namespace,script)

File "/mnt/usr1/usr1/home/tbrown/Package/leo/git/leo-editor/leo/core/leoCommands.py", line 2511, in executeScriptHelper
execfile(scriptFile,d)

File "/home/tbrown/.leo/scriptFile.py", line 5, in <module>
atAuto=True,atShadow=False,s=None,ext=None)

File "/mnt/usr1/usr1/home/tbrown/Package/leo/git/leo-editor/leo/core/leoImport.py", line 560, in createOutline
func(atAuto=atAuto,parent=p,s=s)

File "/mnt/usr1/usr1/home/tbrown/Package/leo/git/leo-editor/leo/core/leoImport.py", line 611, in scanner_for_at_auto_cb
scanner = aClass(importCommands=self,atAuto=atAuto)

TypeError: __init__() takes at least 4 arguments (3 given)

when I force it to run with

ic = c.importCommands
ic.createOutline('~/r/foo.txt',p,
atAuto=True,atShadow=False,s=None,ext=None)


although I still expect it to run simply because I have a
`@auto-test ~/r/foo.txt` node in the outline I'm opening (in a separate
Leo instance).

Cheers -Terry

[Edward K. Ream]

On Tue, Jul 29, 2014 at 9:37 AM, 'Terry Brown' via leo-editor
<leo-e...@googlegroups.com> wrote:


> Whoops, true of course. So now my importer is:

Ok. I'll have a look.

Edward

[Terry Brown]

On Tue, 29 Jul 2014 09:29:44 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

> On Tue, Jul 29, 2014 at 9:13 AM, Edward K. Ream <edre...@gmail.com>
> wrote:
>
> > You must not return a value from a ctor.
>
> leoPluginsRef.leo contains the sources for all importers & writers.
> Following those patterns may help.

I think what's throwing me off is the BaseScanner ctor is:

def __init__ (self,importCommands,atAuto,language,alternate_language=None):
'''ctor for BaseScanner.'''

and

line 611, in scanner_for_at_auto_cb does
scanner = aClass(importCommands=self,atAuto=atAuto)

which doesn't match the BaseScanner ctor signature. I don't have
anything to say to BaseScanner, so I didn't expect to have to
re-implement __init__.

So I re-implement it throwing in "En" for language, just to get it to
run, and what I expect to happen happens, my re-implemented .run()
runs, which allows me to build a tree under `parent` from `s` however I
want without getting tangled up in all this "scan" voodoo ;-) My
re-implemented .scan() is never called, which makes sense given
my .run()

So that's good, now I'm just back to wondering why this doesn't happen
automatically during Leo launch / outline loading when the `@auto-test`
node is encountered. My importer is never instantiated.

Also, context menu refresh from disk doesn't show up for my `@auto-test`
node.

Cheers -Terry

[Edward K. Ream]

On Tue, Jul 29, 2014 at 9:59 AM, 'Terry Brown' via leo-editor
<leo-e...@googlegroups.com> wrote:

> I'm just back to wondering why this doesn't happen
> automatically during Leo launch / outline loading when the `@auto-test`
> node is encountered. My importer is never instantiated.

> Also, context menu refresh from disk doesn't show up for my `@auto-test`
> node.

I'll look into it. Having your code helps ;-)

Edward

[Edward K. Ream]

On Tue, Jul 29, 2014 at 2:05 PM, Edward K. Ream <edre...@gmail.com> wrote:
> On Tue, Jul 29, 2014 at 9:59 AM, 'Terry Brown' via leo-editor
> <leo-e...@googlegroups.com> wrote:
>
>> I'm just back to wondering why this doesn't happen
>> automatically during Leo launch / outline loading when the `@auto-test`
>> node is encountered. My importer is never instantiated.
>
>> Also, context menu refresh from disk doesn't show up for my `@auto-test`
>> node.

It looks like the problem is that the list of @auto nodes is too
static. In fact, it is located inside at.anyAtFileNodeName.
Naturally, @auto-test is not part of the list.

The solution is to make the list visible and extensible, and have
ic.parse_importer_dict add entries to it. Probably something similar
for writers.

Thanks, Terry, for getting the ball rolling. I'll have a fix later
today or tomorrow.

Edward

[Edward K. Ream]

On Tue, Jul 29, 2014 at 2:32 PM, Edward K. Ream <edre...@gmail.com> wrote:

> It looks like the problem is that the list of @auto nodes is too
> static. In fact, it is located inside at.anyAtFileNodeName.
> Naturally, @auto-test is not part of the list.
>
> The solution is to make the list visible and extensible, and have
> ic.parse_importer_dict add entries to it. Probably something similar
> for writers.
>
> Thanks, Terry, for getting the ball rolling. I'll have a fix later
> today or tomorrow.

Here is my version of modes/test.py:

QQQQQ
'''The @auto importer for .xyzzy files and @auto-test nodes.'''
import leo.core.leoGlobals as g
import leo.plugins.importers.basescanner as basescanner
tag = '(test)'
print('%s importing test.py' % ('='*20))
class TestScanner(basescanner.BaseScanner):
def __init__(self,importCommands,atAuto):
g.trace(tag)
basescanner.BaseScanner.__init__(
self,importCommands,atAuto=atAuto,language='test')
def run(self,s,parent,parse_body=False,prepass=False):
g.trace(tag)

return True
def scan (self,s,parent,parse_body=False):

g.trace(tag)

return True

importer_dict = {
'@auto': ['@auto-test',],
'class': TestScanner,

'extensions': ['.xyzzy',],
}
QQQQQ

This code will be executed for @auto whatever.xyzzy nodes. Naturally,
the importer doesn't populate the tree properly.

This shows, I think that the only remaining problem is the recognition
of @auto-test nodes.

Edward

P.S. As of rev e8eec77...Leo now guarded the calls to the TestScanner
methods in the dispatch methods in leoImport.py and leoAtFile.py.
Previously, syntax or run-time errors in an importer or writer would
crash Leo.

EKR

[Terry Brown]

On Tue, 29 Jul 2014 14:58:50 -0500
"Edward K. Ream" <edre...@gmail.com> wrote:

> This shows, I think that the only remaining problem is the recognition
> of @auto-test nodes.

Sounds good. Could the signature of BaseScanner.__init__ go from

def __init__ (self,importCommands,atAuto,language,alternate_language=None):

to

def __init__ (self,importCommands,atAuto,language='unnamed',alternate_language=None):

just so there's no need to make a do nothing __init__?

Also you probably have it on a todo list somewhere, but importers /
writers in ~/.leo/plugins/[importers|writers] :-)

Cheers -Terry

[Edward K. Ream]

On Tue, Jul 29, 2014 at 3:05 PM, 'Terry Brown' via leo-editor
<leo-e...@googlegroups.com> wrote:
> On Tue, 29 Jul 2014 14:58:50 -0500
> "Edward K. Ream" <edre...@gmail.com> wrote:
>
>> This shows, I think that the only remaining problem is the recognition
>> of @auto-test nodes.
>

> Sounds good. Could the signature of BaseScanner.__init__ [change]

Done at rev 7ccb6b8...

More importantly, a previous rev ad66ffa... now supports dynamic
spellings of @auto nodes. This was driven by the desire to support
@auto-test without any explicit changes to the code.

This is quite a collaboration we have going. I took your
importers/test.py plugin and created an analogous writers/test.py
plugin.

Oh drat. The writers/test.py is being call by @auto-test, but
something recent is preventing importers/test.py from being called.
It used to work, I swear. Hehe. I'll make it work again asap.

Anyway, the point is that supporting dynamic @auto spellings was
surprisingly tricky (and apparently still is!). The two dirt-simple
test.py plugins are a perfect test bed.

Edward

[Edward K. Ream]

On Tuesday, July 29, 2014 7:53:29 PM UTC-5, Edward K. Ream wrote:

Oh drat.  The writers/test.py is being call by @auto-test, but
something recent is preventing importers/test.py from being called.
It used to work, I swear.  Hehe. I'll make it work again asap.

Actually, there wasn't any problem ;-)  The reason why importers/test.py wasn't called was because the file was empty.

This is probably the most reasonable way, but I changed the code so that the relevant importer is called even if the file is empty, as long as the file actually exists.  So now, if there is an @auto-test node,  ic.createOutline will always call importers/test.py.

In short, afaik, importers and writers can create their own kinds of @auto nodes, as long as they have unique spellings.

This is pretty much the end of the project for me, except for removing a few vestiges of the fixed, unchanging @auto spellings.

Edward

[Terry Brown]

On Tue, 29 Jul 2014 18:20:16 -0700 (PDT)

"Edward K. Ream" <edre...@gmail.com> wrote:

> In short, afaik, importers and writers can create their own kinds of
> @auto nodes, as long as they have unique spellings.

Yep, just overriding run() and write() makes it very easy to do
whatever you want when @auto-thingy is loaded or saved, with all the
usual triggers for loading or saving - very useful - thanks :)

Cheers -Terry

[Edward K. Ream]

You're welcome :-) To repeat, we have a great collaboration going.
The auto-registration of @auto-names wouldn't have happened without
your work yesterday.

Only one thing left on the programming list for this project: look in
~/.leo/plugins for importers and writers directory. Note that the
spelling is .leo, not leo.

Oh yes, one more thing: I'll probably comment out the no-longer-used
ctors in the subclasses of BaseScanner and BaseWriter.

A couple of things to document:

1. As you know, importers and writers can register either @auto-names
or file extensions. I have just verified that @auto x.xyzzy will use
both the importer and the writer for the .xyzzy extension, that is,
importers/test.py and writers/test.py. So, for *unique* extensions,
there is no need to use a separate @auto name, you can just use @auto.

2. Nah: ic.createOutline won't call an importer for empty files.
That's just silly. However, if you set trace = True in
ic.createOutline, the traces will tell you whether the read failed
(file did not exist) or returned an empty file. This should eliminate
confusion, especially if your test file isn't empty!

Edward

[Terry Brown]

Well... generally I think you're right, but just for testing last night
I made an importer / writer pair that just added "Opened 2014-2x-xx" and
"Closed 2014-2x-xx" timestamps to the body... hmm, that's not really
illustrating my point.

My point is that @auto-weather for example might read the weather
observations recorded in the file and append the current observation to
the file... so potentially it should be called even on an empty file.

It seems to me that calling the importer and writer on empty files
should have so little cost in cases where it's unnecessary that it's
worth doing it just to enable all the "who would have thought of that"
applications these super cheap load/save hooks provide.

Cheers -Terry

[Edward K. Ream]

On Wed, Jul 30, 2014 at 9:14 AM, 'Terry Brown' via leo-editor
<leo-e...@googlegroups.com> wrote:

> It seems to me that calling the importer and writer on empty files
> should have so little cost in cases where it's unnecessary that it's
> worth doing it just to enable all the "who would have thought of that"
> applications these super cheap load/save hooks provide.

Ok. ic.createOutline does now call importers for empty files.

Unless I am mistaken, writers are always called.

I'm working on optional loading from ~/.leo right now. It will be
ready sometime today.

Edward

[Kent Tenney]

Yeah, I haven't been keeping up with details, but it seems
@auto is gaining superpowers, wherein I can equip the
node with custom capabilities

blah blah

nevermind, Edward's message just arrived: empty ok

> --
> You received this message because you are subscribed to the Google Groups "leo-editor" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
> To post to this group, send email to leo-e...@googlegroups.com.
> Visit this group at http://groups.google.com/group/leo-editor.
> For more options, visit https://groups.google.com/d/optout.

[Terry Brown]

On Wed, 30 Jul 2014 09:38:19 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

> On Wed, Jul 30, 2014 at 9:14 AM, 'Terry Brown' via leo-editor
> <leo-e...@googlegroups.com> wrote:
>
> > It seems to me that calling the importer and writer on empty files
> > should have so little cost in cases where it's unnecessary that it's
> > worth doing it just to enable all the "who would have thought of
> > that" applications these super cheap load/save hooks provide.
>
> Ok. ic.createOutline does now call importers for empty files.
>
> Unless I am mistaken, writers are always called.

Only if the file's tree is dirty. Probably best it stay that
way I guess, calling it regardless would be inconsistent. You could
add a do nothing .write_always() which is called when the tree's not
dirty so the user can call their .write() from that if they want to do
something regardless of dirtiness.

> I'm working on optional loading from ~/.leo right now. It will be
> ready sometime today.

Great.

Can you confirm that @auto-explicit always trumps extension? I would
have assumed that, but tests indicate otherwise. I.e.
@auto-ctext ~/r/foo.txt doesn't work, @auto-ctext ~/r/foo.txxt does,
the ctext importer isn't run on the .txt version.

Cheers -Terry


> Edward
>

[Edward K. Ream]

On Wed, Jul 30, 2014 at 11:06 AM, 'Terry Brown' via leo-editor
<leo-e...@googlegroups.com> wrote:
> On Wed, 30 Jul 2014 09:38:19 -0500
> "Edward K. Ream" <edre...@gmail.com> wrote:

>> Unless I am mistaken, writers are always called.
>
> Only if the file's tree is dirty.

Er, right. That's what I meant :-)

> Can you confirm that @auto-explicit always trumps extension?

That was the intention. I'll look into it later today.

EKR

[Edward K. Ream]

On Wed, Jul 30, 2014 at 11:11 AM, Edward K. Ream <edre...@gmail.com> wrote:
>> Can you confirm that @auto-explicit always trumps extension?

Works for me. Example: @auto-test xyzzy.txt works.

But notice, I wrote xyzzy.txt with and @edit node. We probably have to
be careful when working with "dummy" plugins. If the writer doesn't
actually write anything the to-be-imported file doesn't exist, and
then there isn't anything that ic.createOutline can do. It surely
won't call any importer on a non-existent file.

EKR

[Terry Brown]

On Wed, 30 Jul 2014 12:59:30 -0500

"Edward K. Ream" <edre...@gmail.com> wrote:

> On Wed, Jul 30, 2014 at 11:11 AM, Edward K. Ream
> <edre...@gmail.com> wrote:
> >> Can you confirm that @auto-explicit always trumps extension?
>
> Works for me. Example: @auto-test xyzzy.txt works.

Indeed, I have three copies of the same file:

md5sum:
b48242021d35479828664d6879864907 /home/tbrown/r/foo_good.txt
b48242021d35479828664d6879864907 /home/tbrown/r/foo.txt
b48242021d35479828664d6879864907 /home/tbrown/r/foo.txxt

and import works on foo.txxt and foo_good.txt but not foo.txt.

...and when I move my testing to a new .leo outline everything works,
I'll just assume the outline I grabbed for testing was haunted /
cursed / possessed and move on.

It was completely reproducible in the bad outline though:

OK @auto-ctext foo_good.txt
OK @auto-ctext foo.txxt

FAIL @auto-ctext foo.txt

Cheers -Terry

[Edward K. Ream]

On Wed, Jul 30, 2014 at 1:59 PM, 'Terry Brown' via leo-editor
<leo-e...@googlegroups.com> wrote:

> I'll just assume the outline I grabbed for testing was haunted /
> cursed / possessed and move on.

When testing I always use --no-cache. Otherwise caching will shortcut
things in very confusing ways.

Edward