viewrendered - file not found

[lewis]

Edward,

Welome back! I just ran build 4446 and noticed Leo now seems to add
the python folder path before the file location. Select an @movie node
alt-x vr
A rendered view window opens:
Movie

file not found: C:\Python32\Lib\site-packages\leo-editor\leo\core
\C:Users\Lewisn\Downloads\introducing-creo_5mb.wmv

No such problem with build 4445. I couldn't find a bug reference for
build 4446 - can you point to what bug was fixed?
Leo Log Window
Leo 4.9 final, build 4411, June 21, 2011
Python 3.2.2, qt version 4.7.3
Windows 6, 0, 6002, 2, Service Pack 2
reading: N:\leo\Media_Leo.leo

Regards
Lewis

[Edward K. Ream]

On Wed, Aug 31, 2011 at 7:28 AM, lewis <lewi...@operamail.com> wrote:
> Edward,
>
> Welome back! I just ran build 4446 and noticed Leo now seems to add
> the python folder path before the file location.

That's the problem with making almost any change to file-resolution
logic: it's going to break some file references.

> No such problem with build 4445. I couldn't find a bug reference for
> build 4446  - can you point to what bug was fixed?

https://bugs.launchpad.net/leo-editor/+bug/835735

Feel free to suggest alternative fixes. For the actual changes, you
can do bzr qlog for rev 4446.

Edward

[lewis]

Thanks for the bug link. I tried to use the examples to make my node
links work again, changing from the simpler:
node > body text
@movie Creo.wmv > N:\Home\PTC_Creo\Creo.wmv
to trying
@movie {{g.app.loadDir}}Creo.wmv > N:\Home\PTC_Creo\Creo.wmv

What am I getting wrong here? I also tried variations with the file
path only on the node side. Do these examples apply to windows?
My impression is that the fix to this bug is more difficult to work
with than the bug. That may be because the problem was NIMBY :)

Regards
Lewis

On Aug 31, 10:36 pm, "Edward K. Ream" <edream...@gmail.com> wrote:

> That's the problem with making almost any change to file-resolution
> logic: it's going to break some file references.
>

> https://bugs.launchpad.net/leo-editor/+bug/835735

[lewis]

Edward,

While I was running leo build 4449 it found the files and worked fine
using the simple @movie Creo.wmv > N:\Home\PTC_Creo\Creo.wmv.
No fiddly {{g.app.loadDir}} required!
If I run build 4451 the problem re-appears. The rendered view window
message example:

Movie

file not found: C:\Python32\Lib\site-packages\leo-editor\leo\core

\N:Home\PTC_Creo\Creo.wmv

Leo Log Window
Leo 4.9 final, build 4411, June 21, 2011
Python 3.2.2, qt version 4.7.3

Windows 6, 1, 7601, 2, Service Pack 1

I've verified this difference by going back to 4449.
Let me know if you need any more information.

[Edward K. Ream]

On Tue, Sep 6, 2011 at 6:46 AM, lewis <lewi...@operamail.com> wrote:
> Edward,
>
> While I was running leo build 4449 it found the files and worked fine
> using the simple @movie Creo.wmv   >   N:\Home\PTC_Creo\Creo.wmv.
> No fiddly {{g.app.loadDir}} required!
> If I run build 4451 the problem re-appears.

Is this still a problem with recent builds?

Edward

[lewis]

Yes, same problem exists right through to build 4671

Lewis

On Oct 28, 6:21 am, "Edward K. Ream" <edream...@gmail.com> wrote:

[Edward K. Ream]

On Fri, Oct 28, 2011 at 7:44 AM, lewis <lewi...@operamail.com> wrote:
> Yes, same problem exists right through to build 4671

Thanks for this confirmation. I've just put it on the list to investigate asap.

Edward

[Edward K. Ream]

Just did so. My guess is that the issue is confined to the get_fn
method in viewrendered.py.

Oops. I just now took a good look at the error:

file not found: C:\Python32\Lib\site-packages\leo-editor\leo\core

\C:Users\Lewisn\Downloads\introducing-creo_5mb.wmv

There are two instances of \C: here! Obviously, this is the proximate
cause of the failure to open the file: the filename is bad.

Earlier you said:

@movie Creo.wmv > N:\Home\PTC_Creo\Creo.wmv

I suspect that N:\Home is not getting translated properly;
We have another instance of bug 613153:
unable to describe root directory on thumb drive
https://bugs.launchpad.net/leo-editor/+bug/613153

Anyway, let's press on...

Here is the before/after version of the code, slightly reformatted to
make the similarities and differences clearer::

# Before

#@+node:ekr.20110320233639.5776: *5* get_fn
def get_fn (self,s,tag):

pc = self ; p = pc.c.p
fn = s or p.h[len(tag):]
fn = fn.strip()

path = g.os_path_finalize_join(g.app.loadDir,fn)

ok = g.os_path_exists(path)
return ok,path

# After

def get_fn (self,s,tag):

pc = self ; c = pc.c
fn = s or c.p.h[len(tag):]
fn = fn.strip()

# path = g.os_path_finalize_join(g.app.loadDir,fn)
fn = fn.replace('\\','/')
parts = fn.split('/')
args = [g.app.loadDir]
args.extend(parts)
path = g.os_path_finalize_join(*args,c=c)

ok = g.os_path_exists(path)
return ok,pat

In other words, the only difference is replacing::

path = g.os_path_finalize_join(g.app.loadDir,fn)

by::

fn = fn.replace('\\','/')
parts = fn.split('/')
args = [g.app.loadDir]
args.extend(parts)
path = g.os_path_finalize_join(*args,c=c)

Perhaps replacing back slashes with forward slashes may have the
original problem, and then you used a "fancy" path that caused even
more problems.

To find out, please insert the following (tested) trace and report the
output (it will go to the console):

# Put this at the start of the method, before changing fn.
old_path = g.os_path_finalize_join(g.app.loadDir,fn)

# Put this after path has been computed.
g.trace('old_path: %s\nnew_path: %s' % (
repr(old_path),repr(path)))

Please try this with both these nodes::

@movie Creo.wmv
@movie N:\Home\PTC_Creo\Creo.wmv (or something similar)

Thanks.

Edward

[Edward K. Ream]

On Fri, Oct 28, 2011 at 4:42 PM, Edward K. Ream <edre...@gmail.com> wrote:
> I suspect that N:\Home is not getting translated properly;
> We have another instance of bug 613153:

A bad copy-editing bug: I meant to say:

We *may* have another instance of bug 613153.

EKR

[lewis]

Edward,

The filename isn't bad. I used \C:Users\Lewisn\Downloads\introducing-
creo_5mb.wmv as a second example of the problem.
Maybe I misled you by using a different example.
The text "Movie file not found: C:\Python32\Lib\site-packages\leo-
editor\leo\core\"
is the common first section of the error message. Viewrendered simply
doesn't find the file.

Regards
Lewis

On Oct 29, 8:42 am, "Edward K. Ream" <edream...@gmail.com> wrote:
> On Fri, Oct 28, 2011 at 4:15 PM, Edward K. Ream <edream...@gmail.com> wrote:

>
> > On Fri, Oct 28, 2011 at 7:44 AM, lewis <lewisn...@operamail.com> wrote:
> >> Yes, same problem exists right through to build 4671
>
> > Thanks for this confirmation.  I've just put it on the list to investigate asap.
>
> Just did so.  My guess is that the issue is confined to the get_fn
> method in viewrendered.py.
>
> Oops.  I just now took a good look at the error:
>
>  file not found: C:\Python32\Lib\site-packages\leo-editor\leo\core
> \C:Users\Lewisn\Downloads\introducing-creo_5mb.wmv
>
> There are two instances of \C: here!  Obviously, this is the proximate
> cause of the failure to open the file: the filename is bad.
>
> Earlier you said:
>
> @movie Creo.wmv   >   N:\Home\PTC_Creo\Creo.wmv
>
> I suspect that N:\Home is not getting translated properly;
> We have another instance of bug 613153:

> unable to describe root directory on thumb drivehttps://bugs.launchpad.net/leo-editor/+bug/613153

[lewis]

Edward,

I updated viewrendered.py
replaced:
fn = fn.strip()
with
fn = fn.strip()
old_path = g.os_path_finalize_join(g.app.loadDir,fn)

and replaced
ok = g.os_path_exists(path)
with
ok = g.os_path_exists(path)

g.trace('old_path: %s\nnew_path: %s' % (
repr(old_path),repr(path)))

For both the nodes,

@movie Creo.wmv
@movie N:\Home\PTC_Creo\Creo.wmv

The log is the same:

Leo Log Window
Leo 4.9 final, build 4411, June 21, 2011

Python 3.2.2, qt version 4.7.4

Windows 6, 1, 7601, 2, Service Pack 1

reading: N:\leo\Media_Leo.leo
get_fn old_path: 'N:\\Home\\PTC_Creo\\Creo.wmv'
new_path: 'C:\\Python32\\Lib\\site-packages\\leo-editor\\leo\\core\
\N:Home\\PTC_Creo\\Creo.wmv'

The viewrendered window is "Movie

file not found: C:\Python32\Lib\site-packages\leo-editor\leo\core

\N:Home\PTC_Creo\Creo.wmv"

Regards
Lewis

On Oct 29, 8:42 am, "Edward K. Ream" <edream...@gmail.com> wrote:

> On Fri, Oct 28, 2011 at 4:15 PM, Edward K. Ream <edream...@gmail.com> wrote:
> Anyway, let's press on...
>

> .....

[Edward K. Ream]

On Sat, Oct 29, 2011 at 8:01 PM, lewis <lewi...@operamail.com> wrote:
> Edward,
>
> The filename isn't bad.

This is an invalid file name::

C:\Python32\Lib\site-packages\leo-editor\leo\core\N:Home\PTC_Creo\Creo.wmv"

It appears the problem is in g.os_path_finalize_join. I'll investigate.

Edward

[Edward K. Ream]

On Sun, Oct 30, 2011 at 8:50 AM, Edward K. Ream <edre...@gmail.com> wrote:

> It appears the problem is in g.os_path_finalize_join.  I'll investigate.

g.os_path_join works; g.os_path_finalize doesn't. I'll have the fix shortly.

BTW, all investigations now take place in the context of a unit test.
This is just great stuff.

Edward

[Edward K. Ream]

On Thu, Nov 3, 2011 at 9:51 PM, Edward K. Ream <edre...@gmail.com> wrote:
> On Sun, Oct 30, 2011 at 8:50 AM, Edward K. Ream <edre...@gmail.com> wrote:
>
>> It appears the problem is in g.os_path_finalize_join.  I'll investigate.
>
> g.os_path_join works; g.os_path_finalize doesn't.   I'll have the fix shortly.

Here is the unit test that reveals the problem, imo::

path1 = r'C:\Python32\Lib\site-packages\leo-editor\leo\core'
path2 = r'\N:Home\PTC_Creo\Creo.wmv'
path3 = r'N:\Home\PTC_Creo\Creo.wmv'

path12 = os.path.join(path1,path2)
path13 = os.path.join(path1,path3)

print(path12,g.os.path.abspath(path12))
print(path13,g.os.path.abspath(path13))

And here is the output::

\N:Home\PTC_Creo\Creo.wmv c:\N:Home\PTC_Creo\Creo.wmv
N:\Home\PTC_Creo\Creo.wmv N:\Home\PTC_Creo\Creo.wmv

The first line shows an improper filename::

c:\N:Home\PTC_Creo\Creo.wmv

the second line show a proper filename::

N:\Home\PTC_Creo\Creo.wmv

That is, on Windows at least:

os.path.abspath(os.path.join(path1,path2)) # bad
os.path.abspath(os.path.join(path1,path3)) # good

In both cases, os.path.join discards path1, but::

os.path.abspath(path12) # bad
os.path.abspath(path13) # good

Thus, on Windows, the following yields a proper file name::

path3 = r'N:\Home\PTC_Creo\Creo.wmv'

while the following yields an improper file name::

path2 = r'\N:Home\PTC_Creo\Creo.wmv'

Because this is is an issue with os.path.abspath, g.os_path_abspath
can not be to blame, as extensive unit testing has shown.

Obviously, the difference between N:\whatever and \N:whatever is hard
to see, but I think it's pretty clear that one works and the other
doesn't, quite independently of Leo.

Edward

[Edward K. Ream]

On Nov 4, 3:03 am, "Edward K. Ream" <edream...@gmail.com> wrote:

> Obviously, the difference between N:\whatever and \N:whatever is hard
> to see, but I think it's pretty clear that one works and the other
> doesn't, quite independently of Leo.

Apparently, much the same discussion appears at this bug:

unable to describe root directory on thumb drive
https://bugs.launchpad.net/leo-editor/+bug/613153

I've just promoted this bug to high priority so it will be resolved
asap.

Edward

[lewis]

Edward,

'On Nov 4, 1:51 pm, Edward K. Ream <edream...@gmail.com> wrote:

g.os_path_join works; g.os_path_finalize doesn't. I'll have the fix

shortly. '

Interestingly test.leo#Startup-->Buttons-->@ignore Disabled buttons--
>Movie buttons-->@button load video-->run
lists a movie button where the run node (at line 12) uses:
path = g.os_path_finalize_join


This @button works fine and has no trouble with paths to files.

Thanks
Lewis

On Nov 7, 10:55 pm, "Edward K. Ream" <edream...@gmail.com> wrote:
> On Nov 4, 3:03 am, "Edward K. Ream" <edream...@gmail.com> wrote:
>
> > Obviously, the difference between N:\whatever and \N:whatever is hard
> > to see, but I think it's pretty clear that one works and the other
> > doesn't, quite independently of Leo.
>
> Apparently, much the same discussion appears at this bug:
>

> unable to describe root directory on thumb drivehttps://bugs.launchpad.net/leo-editor/+bug/613153

[vitalije]

Maybe Leo could deduce by the first character of path whether it is absolute or relative path. Than if it is absolute it computes relative to root of leo file. In pseudo code:

def root_of_leo_file():
 ____ if os.name == 'posix':
________ return '/'
____ elif os.name == 'nt':
________ return c.fileName()[:3]
____ elif ..other names

def calculate_path(fname):
____ if fname.startswith(("/","\\"):
_______# absolute path
_______return os.path.join(root_of_leo_file(), fname)
____ return os.path.join(os.path.dirname(c.fileName()), fname)

my 2 cents
Vitalije

[Terry Brown]

On Mon, 7 Nov 2011 03:55:15 -0800 (PST)

"Edward K. Ream" <edre...@gmail.com> wrote:

Part of the problem, as mentioned in the thread for that bug, is
http://bugs.python.org/issue1669539 which has escaped resolution for 4
years as is listed as impacting py 3.3.

I think, before trying to fix this, we should have a page in the docs.
which painstakingly spells out how paths are determined, and then just
implement whatever the page says.

Which I think means *not* using os.getcwd() which is basically random.
I think our past consensus was the concept of "process current
directory" makes sense in an interactive shell, but for Leo the correct
context is "node current directory", determined by the effects of all
path related aspects of all parent nodes.

Of course unix has no analog of windows "A:" being the current
directory on the the "A:" drive vs. "A:\" being the root of the "A:"
drive, so ultimately that part's Window's specific.

Also, we should handle shares specified
as //machine/share/path/file.leo or \\machine\share\path\file.leo.
Without confusing them with unix absolute paths.

Cheers -Terry

[Terry Brown]

On Mon, 7 Nov 2011 04:33:46 -0800 (PST)
vitalije <vita...@gmail.com> wrote:

> Maybe Leo could deduce by the first character of path whether it is
> absolute or relative path. Than if it is absolute it computes relative to
> root of leo file.

I think absolute paths should be interpreted according to the host
systems file system conventions, so /etc/hosts is /etc/hosts
not /home/tbrown/.leo/etc/hosts. Perhaps I misunderstand what your
comment.

Cheers -Terry

[Terry Brown]

On Mon, 7 Nov 2011 10:32:34 -0600
Terry Brown <terry_...@yahoo.com> wrote:

> Perhaps I misunderstand what your
> comment.

Is it too late to claim English isn't my first language? :-}

Cheers -Terry

[vitalije]

>Is it too late to claim English isn't my first language?  :-}

neither is my first language.

I didn't think that /etc/hosts should be resolved as /home/.../etc/hosts
My idea was that we should deduce root folder as root of Leo file. On linux machines it would be '/', while on windows it would be <Drive>:\ . Concatenating, that way deduced root with the given absolute path in heading we would get correct absolute path on both windows and linux. I have no idea what would be correct on Macintosh but, AFAIK support for Mac is given up recently. Maybe some Mac users could suggest solution.

I agree that current directory has no meaning in Leo. As a matter of fact executing script could easily change cwd. OTOH folder of Leo file has unique meaning on both systems and root of absolute path to Leo file is something that we could define on:

1.  Windows as <Drive letter>:\
2. Linux as /
   

I did forget network paths like  "\\machine\share\path\file.leo". That could be the 3rd case where root folder is computed as \\machine\share\

On my Ubuntu samba shares are mounted as ~/.gvfs/sharename, so I guess that on linux we should check for special case if absolute path to leo file starts with ~/.gvfs/sharename/  then root should be computed as ~/.gvfs/sharename/

PS: reading again what I've just written I am not sure that I made it understandable at all. If it's not clear I am sorry...

[Terry Brown]

vitalije - if you read all the way to the end, I think we end up in the
same place :-)

On Mon, 7 Nov 2011 13:06:21 -0800 (PST)
vitalije <vita...@gmail.com> wrote:

> >Is it too late to claim English isn't my first language? :-}
> neither is my first language.

The trouble is it is my first, and only :-(, language, so my mangled
sentence was just careless.

> I didn't think that /etc/hosts should be resolved as /home/.../etc/hosts
> My idea was that we should deduce root folder as root of Leo file. On linux
> machines it would be '/', while on windows it would be <Drive>:\ .
> Concatenating, that way deduced root with the given absolute path in
> heading we would get correct absolute path on both windows and linux.

Here's how I think it works now, which is also how I think it should
work in general, and may be the same as what you're saying, if not
perhaps you can point out the differences.

The goal is to work out the path for a node, which may be the path for
a derived file represented by the node.

- The initial directory at the top of the .leo file is the directory
containing the .leo file. (*)

- The path for each node is that directory, with any modifications
made by @path or @<file> (@auto, @file, etc.) directives on the
node itself or its ancestors. Any absolute paths encountered would
eliminate the influence of all path information from any ancestor
nodes.

So for the file /home/tbrown/myproject/project.leo,

@auto projectnotes.txt would be /home/tbrown/myproject/projectnotes.txt

@path docs
@auto projectnotes.txt
would be /home/tbrown/myproject/docs/projectnotes.txt

@path /tmp
@auto projectnotes.txt
or
@auto /tmp/projectnotes.txt

would be /tmp/projectnotes.txt

etc.

(*) - so I realize now maybe this is not what you're suggesting,
currently relative paths are determined relative to the folder
containing the .leo file, and you're suggesting they should be
relative to the root of the filesystem containing the .leo file?

I think such a change would break to many existing outlines, and
is probably less useful overall. But maybe I'm not following your
suggestion properly.

In the specific case of Window's drive specific relative paths, like
A:projectnotes.txt, if the path in effect for the node is on the A:
drive, ignore the A: and treat as a regular relative path. If the
path in effect for the node is *not* on the A: drive, then perhaps
there's no way to make it work consistently. Given a .leo file
C:\myproject\project.leo, what does @auto A:projectnotes.txt mean?

And for Windows paths like @auto \myproject\projectnotes.txt, the "root
of which drive?" question can be answered consistently by following the
above rules to work out which drive you started on, either the one
holding the .leo file, or another specified by an absolute @path
ancestor node.

Ah - and now I think this last point was the point you were making, and
perhaps we agree?

Cheers -Terry

[vitalije]

>Ah - and now I think this last point was the point you were making, and
>perhaps we agree?

I think we were talking the same thing.

> Given a .leo file C:\myproject\project.leo, what does @auto A:projectnotes.txt mean?

IMHO perfectly nothing. I can't imagine a use case where one would like to have path that way defined.

>and you're suggesting they should be relative to the root of the filesystem containing the .leo file,
No, that wasn't my suggestion. I suggested that only absolute paths that have no specified drive should be appended to the "root folder" where root folder has different meaning on different platforms. I wouldn't expect to break any existing outline, because relative paths would still be computed the same way they are computed now.

I hope this was a bit clearer.
Vitalije

PS: When there was discussion in this group about the above bug a while ago, I had this idea but I didn't post it because I was expecting that someone else would solve it anyway. Reading this thread I noticed that the bug is still present and decided to post my suggestion hoping it would help. I don't use windows any more and this bug doesn't affect my work flow  at all.

[Edward K. Ream]

On Mon, Nov 7, 2011 at 4:43 PM, Terry Brown <terry_...@yahoo.com> wrote:

>  - The initial directory at the top of the .leo file is the directory
>   containing the .leo file.  (*)

Yes, and this isn't going to change. That is, for almost all
purposes, Leo does the following when computing paths::

g.os_path_finalize_join(g.app.loadDir,whatever)

We can't change this now: it would break all outlines.

In essence, we are discussing what to do when "whatever" is an absolute path.

>  - The path for each node is that directory, with any modifications
>   made by @path or @<file> (@auto, @file, etc.) directives on the
>   node itself or its ancestors.  Any absolute paths encountered would
>   eliminate the influence of all path information from any ancestor
>   nodes.

Correct.

> So for the file /home/tbrown/myproject/project.leo,
>
> @auto projectnotes.txt would be /home/tbrown/myproject/projectnotes.txt
>
> @path docs
>     @auto projectnotes.txt
>     would be /home/tbrown/myproject/docs/projectnotes.txt
>
> @path /tmp
>     @auto projectnotes.txt
> or
> @auto /tmp/projectnotes.txt
>
> would be /tmp/projectnotes.txt

The above has always been my intention.

> In the specific case of Window's drive specific relative paths, like
> A:projectnotes.txt,

Imo, this has no place in Leo.

> Given a .leo file C:\myproject\project.leo, what does @auto A:projectnotes.txt mean?

I agree that it means nothing. Imo, it should be flagged as an error.

> And for Windows paths like @auto \myproject\projectnotes.txt, the "root
> of which drive?" question can be answered consistently by following the
> above rules to work out which drive you started on, either the one
> holding the .leo file, or another specified by an absolute @path
> ancestor node.

Maybe, but I would prefer to keep the meaning of
g.os_path_finalize_join to be a close as possible to the underlying
os.path methods of which it is composed, namely:

QQQQQ
def os_path_finalize (path,**keys):

'''
Expand '~', then return os.path.normpath, os.path.abspath of the path.

There is no corresponding os.path method'''

c = keys.get('c')

if c: path = g.os_path_expandExpression(path,**keys)

path = g.os_path_expanduser(path)
path = os.path.abspath(path)
path = os.path.normpath(path)
return path

def os_path_finalize_join (*args,**keys):

'''Do os.path.join(*args), then finalize the result.'''

c = keys.get('c')

if c:
args = [g.os_path_expandExpression(z,**keys)
for z in args if z]

return os.path.normpath(os.path.abspath(
g.os_path_join(*args,**keys))) # Handles expanduser
QQQQQ

Unit tests (as expanded by the stupendous Aha) will be helpful, or
even decisive here. A unit test can be a design spec for dozens of
cases.

"Dense" unit tests have the following form::

table = (
case 1, # a tuple, defining (arg1,...,argK,expected)
...
case N,
)

for arg1,...,argK,expected in table:
result = aFunction(arg1,...,argK)
assert result == expected,'expected: %s, got: %s' % (
expected,result)

Note that the tuples in the table may also specify "aFunction".

I have written many such tests, and plan to write many more.

To illuminate the present discussion, "aFunction" will compute the
g.os_path_whatever on a given descendant tree containing @path
directives, @@file nodes (temporarily changed to @file nodes), etc.

Once this test is in place, we can suggest new table entries and argue
about what the expected result should be :-) I'll get started on this
unit test today.

Edward