Auto-export of all @file nodes as @nosent without modifying the outline

[zpcspm]

Having a lot of @file nodes pointing to derived files in an outline
that need to be exported without sentinels (I've found in the docs
that @file must be changed with @nosent), how would one automate the
export process? Playing with the File->Export->Remove sentinels
doesn't do what I need. Any hints are welcome. A plug-in that is able
to perform the task (it is possible that I've overlooked it, but if
Leo doesn't have such a feature yet, consider this a feature request
please) would be probably better than a general advice to write one,
since I'm just starting to use Leo and I lack a lot of knowledge.

[thyrsus]

I'm afraid I don't understand the question. If I create a node
"@nosent foo"
it gets written out verbatim without sentinels, except if it contains
indicators that
subnodes should be included, they are replaced with the contents of
the subnodes
(without sentinels) as intended. E.g., if "@nosent foo" contains

This is the contents of foo.

...that is exactly what the derived file contains (after you tell leo
to save). If the
node contains

This is the contents of foo
@others

and there is a node right indented under that node containing

This is the contents of the subnode.

Then the file contains, after a save

This is the contents of foo
This is the contents of the subnode.

without any sentinels. Tell us what you were expecting, and perhaps
we can
get from that to how to accomplish it.

[zpcspm]

On Mar 29, 2:04 am, thyrsus <sschae...@acm.org> wrote:
> I'm afraid I don't understand the question.

Pardon for being obscure. Here's a simple example:

--- cut here ---

<?xml version="1.0" encoding="utf-8"?>
<?xml-stylesheet ekr_test?>
<leo_file>
<leo_header file_format="2" tnodes="0" max_tnode_index="0"
clone_windows="0"/>
<globals body_outline_ratio="0.5">
<global_window_position top="0" left="0" height="679" width="938"/>
<global_log_window_position top="0" left="0" height="0" width="0"/>
</globals>
<preferences/>
<find_panel_settings/>
<vnodes>
<v t="shadow.20080329095632"><vh>@chapters</vh></v>
<v t="shadow.20080329095703" a="E" tnodeList="shadow.
20080329095703,shadow.20080329095859"><vh>@file /tmp/demo/run.py</vh>
<v t="shadow.20080329095859"><vh>&lt;&lt; imports &gt;&gt;</vh></v>
</v>
<v t="shadow.20080329095723" a="EV" tnodeList="shadow.
20080329095723,shadow.20080329095759,shadow.20080329095839"><vh>@file /
tmp/demo/mymodule.py</vh>
<v t="shadow.20080329095759" a="E"><vh>class HelloWorld</vh>
<v t="shadow.20080329095839"><vh>hello_world</vh></v>
</v>
</v>
</vnodes>
<tnodes>
<t tx="shadow.20080329095632"></t>
<t tx="shadow.20080329095703">@language python
&lt;&lt; imports &gt;&gt;

if __name__ == '__main__':
hello = mymodule.HelloWorld()
hello.hello_world()</t>
<t tx="shadow.20080329095723">@language python
@others</t>
<t tx="shadow.20080329095759">class HelloWorld:
@others</t>
<t tx="shadow.20080329095839">def hello_world(self):
print 'Hello, world!'</t>
<t tx="shadow.20080329095859">import mymodule</t>
</tnodes>
</leo_file>

--- cut here ---

I have two @file nodes in this outline. If I want to export the clean
(with sentinels removed) derived files to /tmp/demo/clean, I can make
the following changes:

"@file / tmp/demo/run.py" -> "@nosent / tmp/demo/clean/run.py"
"@file / tmp/demo/mymodule.py" -> "@nosent / tmp/demo/clean/
mymodule.py"

and save the outline.

So my question is: is it possible to get the clean derived files
without having to convert manually each @file node in the outline to a
@nosent node? The reason for this is that I can have hundreds of
derived files and I might need clean snapshots often.

[Edward K. Ream]

On Sat, Mar 29, 2008 at 3:22 AM, zpcspm <zpc...@gmail.com> wrote:

So my question is: is it possible to get the clean derived files
without having to convert manually each @file node in the outline to a
@nosent node? The reason for this is that I can have hundreds of
derived files and I might need clean snapshots often. 

The answer must be 'yes', because Leo is highly scriptable.  Let's see how to do this.  I'll write the script in easy steps, so this should be a good example.

1. Run Leo from the console.  I like to run Leo from a console at all times, so I can see the output of print statements.

2. Create an @button node in your outline.

The headline is @button write-cleaned-files.  The body text will contain the script.  While testing, I'll execute the script with control-b (Leo's execute-script command).  We can also execute the script by hitting the yellow 'script-button' button in the icon bar.  This will create a new script button called write-cleaned-file (the title gets truncated).  Creating a script button also creates a corresponding command, which we can execute with <alt-x>write-cleaned-files<return> or more easily, using tab completion, <alt-x>write-cl<tab><return>

BTW, when you reload the outline later, Leo will create a write-cleaned-file button automatically.

2. Turn on autocompletion and calltips.

Type Alt-1 and Alt-2.  This saves typing and can be useful to see what methods exists.

3.  Put the following script in the body pane of the @button node:

@others

for p in c.allNodes_iter():
    if p.isAnyAtFileNode():
        print p.headString()

With autocompletion on, you will actually only have to type:

for p in c.allN():
    if p.isA():
        print p.he():

Run this script by typing control-b, or by executing the script button as discussed above.  You will see a list of all the @thin, @auto, @file, etc nodes in the outline.

4. Create a helper function, called clean_node, in a child node.

The headline will be clean_file, and the body text will contain the function.

For now, but the following in body:

def clean_file(p):
    g.trace(p)

In order for the script to access this function, add the @others directive to the @button node.  We'll also call clean_node, so the body of the @button node will be:

@others

for p in c.allNodes_iter():
    if p.isAnyAtFileNode():
        clean_file(p):

Now comes the interesting part.  We want to use Leo's write code in leoAtFile.py to write the derived file, but we want to modify that code in two ways: we want to disable writing of sentinels, and we want write the cleaned file to a 'cleaned' directory.

The leoAtFile.write method, in leoPy.leo in the node

Code-->Core classes...-->@thin leoAtFile.py-->Writing...-->Writing (top level)-->Don't override in plugins-->write

is the general-purpose method that writes (most) derived files.

In the body of the clean_file, type:

c.atFileCommands.write(

(because of autocompletion, you actually won't have to type all this).  You will see

c.atFileCommands.write(root, nosentinels=False, thinFile=False, scriptWrite=False, toString=False, write_strips_blank_lines=None

Clear the selection, and add a closing ')'.

Looking at the source code, we see that the write method 'writes' the file to the .stringOutput ivar.  So here is our preliminary code:

def clean_file(p):

    at = c.atFileCommands

    at.write(
        root=p,nosentinels=True,
        thinFile=False,
        scriptWrite=True,
        toString=True,
        write_strips_blank_lines=False)

    g.trace(p.headString(),len(at.stringOutput))

If we execute this script in test.leo, we get the following output:

clean_file: @thin gtkOutlineDemo.py 27847
clean_file: @file server.py 666
clean_file: @file hello.html 1098
clean_file: @file cgi-bin/edward.py 3089
clean_file: @thin cgi-bin/leo.js 3684
clean_file: @thin leoBridgeTest.py 1174
clean_file: @thin leoDynamicTest.py 991

We still must write the result.  We use p.anyAtFileNodeName() to get the 'raw' file name.  We can use the .default_directory ivar (inited by at.write) to get the proper directory.  This directory is set by at.scanAllDirectives, so @path and other details are handled properly.

So we add the following lines:

    fileName = g.os_path_normpath(
        g.os_path_join(
            at.default_directory,'clean',p.anyAtFileNodeName()))

    g.trace(p.headString(),len(at.stringOutput),fileName)

The output of executing the script is now:

clean_file: @thin gtkOutlineDemo.py 27847 C:\leo.repo\leo-editor\trunk\leo\test\clean\gtkOutlineDemo.py
clean_file: @file server.py 666 C:\leo.repo\leo-editor\trunk\leo\test\clean\server.py
clean_file: @file hello.html 1098 C:\leo.repo\leo-editor\trunk\leo\test\clean\hello.html
clean_file: @file cgi-bin/edward.py 3089 C:\leo.repo\leo-editor\trunk\leo\test\clean\cgi-bin\edward.py
clean_file: @thin cgi-bin/leo.js 3684 C:\leo.repo\leo-editor\trunk\leo\test\clean\cgi-bin\leo.js
clean_file: @thin leoBridgeTest.py 1174 C:\leo.repo\leo-editor\trunk\leo\test\clean\leoBridgeTest.py
clean_file: @thin leoDynamicTest.py 991 C:\leo.repo\leo-editor\trunk\leo\test\clean\leoDynamicTest.py

So the last piece of the puzzle is to actually write at.stringOutput to fileName.

BTW, I never remember the names of ivars, or indeed much of anything except the highest-level details.  I found the name of the .default_directory ivar from the at.openFileForWritingHelper.

Looking at at.openFileForWritingHelper, we see that it has code to create the 'clean' directory if it does not exist.  We could duplicate that code in our script, or simply issue an error message if the directory does not exist.

Let's assume that the 'clean' directory does exist, and issue an error otherwise.  So the final version of clean_file is:

def clean_file(p):

    at = c.atFileCommands

    at.write(
        root=p,nosentinels=True,
        thinFile=False,
        scriptWrite=True,
        toString=True,
        write_strips_blank_lines=False)

    fileName = g.os_path_normpath(
        g.os_path_join(
            at.default_directory,'clean',p.anyAtFileNodeName()))

    # g.trace(p.headString(),len(at.stringOutput),fileName)

    # Adapted from at.openFileForWritingHelper
    path = g.os_path_dirname(fileName)
    if not g.os_path_exists(path):
        g.es('clean directory does not exist',path)
        return
    try:
        f = file(fileName,'w')
        f.write(at.stringOutput)
        f.close()
        g.es_print('wrote',fileName)
    except IOError:
        g.es_print('can not write',fileName,color='red')
        g.es_exception()

So that was pretty easy :-)  I'll push test.leo containing this script to the trunk shortly.

HTH.

Edward

[Edward K. Ream]

On Sat, Mar 29, 2008 at 5:35 AM, Edward K. Ream <edre...@gmail.com> wrote:

> So that was pretty easy :-)

Oops.  There may be a problem.  Do **not** use this script until I have tested it more.

Edward
--------------------------------------------------------------------
Edward K. Ream email: edre...@gmail.com
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------

[Edward K. Ream]

On Mar 29, 5:42 am, "Edward K. Ream" <edream...@gmail.com> wrote:

> Oops. There may be a problem. Do **not** use this script until I have
> tested it more.

It looks like writing any kind of node @thin, @nosent, etc, works,
**except** @file nodes. It appears that the so-called tnode-list gets
cleared by the script. This might be called a bug in at.write (when
toString is True). In any case, I'll have to look into it. Again,
don't use this script as it is if you have @file nodes.

Edward

[Edward K. Ream]

Here is the correct def for clean_file:

def clean_file(p):

at = c.atFileCommands

if hasattr(p.v.t,'tnodeList'):
has_list = True
old_list = p.v.t.tnodeList[:]
else:
has_list = False

at.write(
root=p,nosentinels=True,
thinFile=False,
scriptWrite=True,
toString=True,
write_strips_blank_lines=False)

if has_list:
p.v.t.tnodeList = old_list

fileName = g.os_path_normpath(
g.os_path_join(
at.default_directory,'clean',p.anyAtFileNodeName()))

# g.trace(p.headString(),len(at.stringOutput),fileName)

# Adapted from at.openFileForWritingHelper
path = g.os_path_dirname(fileName)
if not g.os_path_exists(path):
g.es('clean directory does not exist',path)
return

try:
f = file(fileName,'w')
f.write(at.stringOutput)
f.close()
g.es_print('wrote',fileName)
except IOError:
g.es_print('can not write',fileName,color='red')
g.es_exception()

This will appear in test.leo and scripts.leo shortly. The node is
called
"Write cleaned files to 'clean' directory" in scripts.leo.

It doesn't actually have to be an @button node because all nodes are
scanned, regardless of the selected position when the script is
invoked.

Edward

[zpcspm]

On Mar 29, 4:28 pm, "Edward K. Ream" <edream...@gmail.com> wrote:
> Here is the correct def for clean_file:

I had to change

fileName = g.os_path_normpath(
g.os_path_join(
at.default_directory,'clean',p.anyAtFileNodeName()))

in the original version to

fileName = g.os_path_normpath(
g.os_path_join(
at.default_directory, 'clean',

g.os_path_basename(p.anyAtFileNodeName())))

and it worked as expected.

However, I can see that this script works just for a particular case
(when all derived files belong to the same directory). A more general
scenario (the derived files are stored in a hierarchy of directories)
would probably require collecting all file paths, finding a common
prefix (let's call it root, that's usually the src directory of a
project), addinng the 'clean' part to it, then adding the remaining
part of the absolute file path (that is relative to the root).
Thank you very much for the feedback.

[Edward K. Ream]

On Sat, Mar 29, 2008 at 1:58 PM, zpcspm <zpc...@gmail.com> wrote:

On Mar 29, 4:28 pm, "Edward K. Ream" <edream...@gmail.com> wrote:
> Here is the correct def for clean_file:

I had to change

fileName = g.os_path_normpath(
       g.os_path_join(
           at.default_directory,'clean',p.anyAtFileNodeName()))

in the original version to

fileName = g.os_path_normpath(
   g.os_path_join(
       at.default_directory, 'clean',

g.os_path_basename(p.anyAtFileNodeName())))

and it worked as expected.

Thanks for this improvement.  I'll put it in scripts.leo.

 

However, I can see that this script works just for a particular case
(when all derived files belong to the same directory).

I think the script looks a directory named 'clean' as a subdirectory of the directory containing each file.  I could be wrong :-)

In any case, this is a minor detail that you can change to suit yourself.

Many other changes are possible.  For example, you could have the script put all the cleaned files in a single, global directory.  The script could just process the selected tree, rather than the entire outline.  The code is left as an exercise--feel free to ask for help.

Edward

[zpcspm]

Here's one more useful (for me at least) improvement:

I've added one more debug line to the script after the main loop that
iterates over nodes. Now it looks like

--- cut here ---

for p in c.allNodes_iter():
if p.isAnyAtFileNode():
clean_file(p)

g.es_print('All files written.', color = 'blue')

--- cut here ---

The reason for this is that saving many files requires some time and
it's not very handy to track script execution by looking at the CPU
load average :-)
This way I see the blue line in the log panel and I know that all
clean files are updated.