Discuss: what should path expressions contain?

[Edward K. Ream]

Recent comments have convinced me that Leo should support safe path expressions.

I have renamed and rewritten  #3260.

At minimum, I suppose the following expressions should be valid:

{{~}}  # home

{{/}}  # os.sep

{{..}}  # parent directory.

Anything else come to mind :-)

Edward

[Edward K. Ream]

On Sat, Apr 8, 2023 at 6:39 AM Edward K. Ream <edre...@gmail.com> wrote:

At minimum, I suppose the following expressions should be valid:

{{~}}  # home

{{/}}  # os.sep

{{..}}  # parent directory.

And combinations of the above:

{{../..}}

{{~/}}

Edward

[Edward K. Ream]

On Saturday, April 8, 2023 at 6:43:24 AM UTC-5 Edward K. Ream wrote:

And combinations of the above:

{{../..}}

{{~/}}

And one more, per g.getBaseDirectory:

{{!}}  # g.app.loadDir.

Imo path expressions probably should be allowed only at the start of filenames. I don't see how they could be useful in other positions.

Edward

[Edward K. Ream]

On Saturday, April 8, 2023 at 7:35:43 AM UTC-5 Edward K. Ream wrote:

Imo path expressions probably should be allowed only at the start of filenames. I don't see how they could be useful in other positions.

In the new scheme path expressions are actually directory expressions:

- The default starting directory is os.path.normpath(os.path.dirname(c.fileName())

  An edge case: c.expand_path_expression will fail if the outline doesn't yet have a name.

-  '!' and '~' may appear only at the start of the expression.

  They will set the default starting directory accordingly.

Edward

[Thomas Passin]

Leo already handles "~", so we don't need "{{~}}".  When I say "handles", I mean it adjusts between Windows and Linux.  We don't need {{sep}} if the handling code makes the adjustment for "/" vs "\" automatically.  This would be highly desirable anyway because sometimes the best way to get the right path is to copy it in the OS's file browser and paste it in directly which will insert native path separators.  Changing them all is a pain and I find it to be error prone.  And I have sometimes ended up with a mix of separator types.  Leo should handle this case automatically, adjusting the separators as needed for the OS.

I agree that ".." steps should be handled correctly but I don't see that they need to be enclosed in braces.  We should try for maximum readability and clarity in the path expressions.

In my experience, the biggest need, apart from the separators, is to adjust the path prefix (the entire path up to but not including the file name) from one system to another.  This could also involve changing from an absolute to a relative path.  Here are some use cases:

1. I send a Leo outline along with its external files (perhaps as one of the new Leo archive files!).  The outline uses absolute paths for some historical reason, but the outline location and the desired location of the external files are different for the new user.  The old @file paths may not even be valid on the target system - e.g. c:\tom\devel\leo vs ~/Temp/leo - even after the separators have been changed.

2. Similar to 1) but I need to move an outline from Windows to Linux so I can test my code on both OSs. 

To me it seems that one capability would be to have conditional expressions.  For example:

@clean {windows: c:\tom\devel\llm; linux: ~/Test/leo/llm}/small_llm_1.py

Perhaps we could use Python's string format syntax with a custom formatter.

[Edward K. Ream]

On Sat, Apr 8, 2023 at 8:01 AM Thomas Passin <tbp1...@gmail.com> wrote:

Leo already handles "~", so we don't need "{{~}}". 

Hmm. You are probably right.

We don't need {{sep}} if the handling code makes the adjustment for "/" vs "\" automatically.

This is the dreaded g.os_path_normslashes. We have to be extremely careful about this issue.  The draft PR #3264 supports only '/' within path expressions, and then only to separate '..' items, so I think the draft is good.

  This would be highly desirable anyway because sometimes the best way to get the right path is to copy it in the OS's file browser and paste it in directly which will insert native path separators.  Changing them all is a pain and I find it to be error prone.  And I have sometimes ended up with a mix of separator types.  Leo should handle this case automatically, adjusting the separators as needed for the OS.

The PR would be broken if you had to change any of your existing outlines that don't already use path expressions.

I agree that ".." steps should be handled correctly but I don't see that they need to be enclosed in braces.

Another hmm. It seems like a great idea and I'm not sure exactly what Leo already does.

We should try for maximum readability and clarity in the path expressions.

As long as we don't break existing outlines!

In my experience, the biggest need, apart from the separators, is to adjust the path prefix (the entire path up to but not including the file name) from one system to another. 

Exactly. That's all the PR allows.

To me it seems that one capability would be to have conditional expressions.  For example:

@clean {windows: c:\tom\devel\llm; linux: ~/Test/leo/llm}/small_llm_1.py

Let's get the basics working first.

This kind of hack is why I suggested writing scripts as a workaround.

Thomas, please review the PR.  Thanks.

Edward

[Edward K. Ream]

On Saturday, April 8, 2023 at 8:01:11 AM UTC-5 tbp1...@gmail.com wrote:

Leo already handles "~", so we don't need "{{~}}". 

Now that I've gathered my wits, let me try to make a simpler response:

1. The new code (PR #3264) will be exactly equivalent to the old unless an @file headline contains a path expression.

2. `~' is needed in path expressions unless Leo already handles `~/x` or `~/../y` without path expressions.

   The PR contains a strong new unit test. I'll test Leo's legacy operation using that test and report back.

3. New: withing path expressions, some character, say '*', will evaluate to (say)

  os.environment[LEO_BASE_DIRECTORY],

    eliminating the need for conditional code or other hacks.

Edward

[Thomas Passin]

We definitely need to have a transition strategy so we don't kill anyone's existing outlines.  I see two ways to approach this:

1. Only allow certain g. methods, all of which would be new and only used in path expressions.  Eventually, only these would be allowed.

2. Current path expressions can use curly braces. If there are any places we will need them, change them to be, e.g., square brackets for the new code.   That will let the path expression code know unambiguously when to use the new or the old processing.

Either way, users need to be informed that at some point, the old handling will go away.  Maybe it can be enabled by a setting that initially will have the value True but in later releases defaults to False:

@bool use-old-path-expressions = False

[Edward K. Ream]

On Sat, Apr 8, 2023 at 9:39 AM Thomas Passin <tbp1...@gmail.com> wrote:

We definitely need to have a transition strategy so we don't kill anyone's existing outlines. 

No such strategy is possible.

Maybe it can be enabled by a setting that initially will have the value True but in later releases defaults to False:

@bool use-old-path-expressions = False

Absolutely not. The PR must break outlines containing path expressions.

The result will (usually) be that Leo can't write the external file, and Leo will say so.

Edward

[jkn]

- shurely '~' is not always needed (if that is what you are saying): I can use {{sep}}home{{sep}}myusername{{sep}}tmp{{sep}}filename.txt  , for example

- I really hope you don't choose '*' for such a feature. might something like '~~' work?

[Edward K. Ream]

On Saturday, April 8, 2023 at 10:49:03 AM UTC-5 jkn wrote:

- I really hope you don't choose '*' for such a feature.

What's wrong with *?  I'm willing to use any single character. What would you prefer, and why?

Edward

[Thomas Passin]

"*" is so widely used as a wildcard for any string that it would be confusing to use it in another way.

[Edward K. Ream]

On Saturday, April 8, 2023 at 10:49:03 AM UTC-5 jkn wrote:

I can use {{sep}}home{{sep}}myusername{{sep}}tmp{{sep}}filename.txt

Without understanding  the exactly what you want to do, you probably should be able to specify this somehow, either without using path expressions at all, or with something like the following:

   {{~}}tmp/filename.txt

  {{*}}filename.txt

 

Edward

[Edward K. Ream]

On Saturday, April 8, 2023 at 1:17:31 PM UTC-5 tbp1...@gmail.com wrote:

"*" is so widely used as a wildcard for any string that it would be confusing to use it in another way.

Alright. What single character would you prefer?

Edward

[Thomas Passin]

The current PR says this:

      '!' sets the directory to g.app.loadDir.
        '~' sets the directory to os.path.expanduser('~')
        '*' sets the directory to os.environ['LEO_BASE_DIRECTORY'] or
            os.path.expanduser('~') if the environment variable does not exist.

1. I don't see any need to use the expression "{{~}}". Just using "~" works now - no need to change it, IMHO.  It would add visual clutter for no benefit.

2. I'd like to see mnemonic symbols.  For example: 

    {{loadDir}}

    {{LEOBASE}}

While we're at it, let's have all the symbolic Leo directories available -

g.app.homeDir #C:/Users/tom
g.app.homeLeoDir #C:/Users/tom/.leo
g.app.leoDir #c:/Tom/git/leo-editor/leo
g.app.loadDir #c:/Tom/git/leo-editor/leo/core
g.app.extensionsDir #c:/Tom/git/leo-editor/leo/extensions
g.app.testDir #c:/Tom/git/leo-editor/leo/test

Then I could write, for example:

@file {{leoDir}}/plugins/new_plugin.py

@file ~/.leo/themes/tbp_new_theme.leo  # or {{homeLeoDir}}/themes/tbp_new_theme.leo

This would be very easy to read, write, and understand, and would be easy to implement.  Visual clutter would be at a minimum.

3. I'd like to add an alias for loadDir - leoCoreDir. It's more mnemonic.  I'm never sure I remember what loadDir means.

[Thomas Passin]

On Saturday, April 8, 2023 at 2:50:01 PM UTC-4 Thomas Passin wrote:

Then I could write, for example:

@file {{leoDir}}/plugins/new_plugin.py

@file ~/.leo/themes/tbp_new_theme.leo  # or {{homeLeoDir}}/themes/tbp_new_theme.leo

Well, I wouldn't be having a Leo outline in an @file node, I suppose [blushes].  Here's a more realistic example:

@clean ~/.leo/vr3/rst-dark.css 

[Edward K. Ream]

On Sat, Apr 8, 2023 at 1:50 PM Thomas Passin <tbp1...@gmail.com> wrote:

1. I don't see any need to use the expression "{{~}}". Just using "~" works now.

Thomas, you have just opened my eyes. Python should do all the work!

- os.path.expandvars substitutes environment variables!

- os.path.expanduser handles per-platform resolution of `~`.

See the links above for details.

As before, any path can be relative to the outline. Just "finalize" the resulting path using:

os.path.normpath(os.path.join(os.path.dirname(c.fileName()), path))




Summary

c.expand_path_expression should let os.path do all the work. No more security woes.

There is no need for path expressions! Bye-bye {{ and }}. No more Leo-only conventions.

c.expand_path_expression will collapse.

Edward

[Thomas Passin]

Excellent!  And if Leo were to export, say, LEODIR, which would be the leo directory, then there would not be a need for {leoDir} since one could write something like 

@file $LEODIR/themes/vr3/rst-dark.css

I like it.

[Edward K. Ream]

On Saturday, April 8, 2023 at 5:52:20 PM UTC-5 tbp1...@gmail.com wrote:

Excellent!  And if Leo were to export, say, LEODIR, which would be the leo directory, then there would not be a need for {leoDir} since one could write something like 

@file $LEODIR/themes/vr3/rst-dark.css

I like it.

So do I. The new code will be ready later this evening.

Edward

[jkn]

did the work that got done here ever get documented? I am looking at some

nodes with old-style path expressions and I am a bit confused what the current state of the art is.

Path expressions don't seem to be discussed in the current documentation.

FWIW, in relation to the posting above this, I would suggest that ${ENVNAME} works in a similar way to $ENVNAME - like in bash scripts, for instance.

Happy New Year!

Jon N

[jkn]

heh - see from a closer reading of the code and https://github.com/leo-editor/leo-editor/pull/3264 that

${ENVNAME} should already be supported, via os.path.expandvars().

I'm still having a little trouble getting the new form(s) to work as I expect,

i will experiment a bit more and report back.

[jkn]

heh again ... that caused a little bit of head scratching...

I wanted an environment variable $NEXTCLOUD, because the name of my nextcloud directory is different

on different machines. But this didn't seem to be working, even though $HOME seemed to work.

But I had this line:

    export NEXTCLOUD=~/nextCloud

in ~/.bashrc ... which is not applicable if I start leo from the desktop

I was confused because if I typed 'env' in a terminal, I *would* see the value.

So I moved the line to ~/.profile, and this behaves as I expect.

I remember when I first learned about the original path specifiers, with {{g.app.homedir}}

and {{sep}}, I thought "hmm ... there must be a good reason why Edward hasn't used the python

os.path functions to sort this out". I must have been carrying that thought with me for a while...

The documentation could do with improving around here though ... in fact I don't think

there is currently anything on path descriptors at all. The availability of $(ENVVAR} at

least would be useful.

Thanks for listening ... J^n

[Edward K. Ream]

On Tue, Jan 9, 2024 at 1:14 PM jkn <jkn...@nicorp.f9.co.uk> wrote:

did the work that got done here ever get documented?

Luke, use the find command!

Searching LeoDocs.leo for "path expression" yields these items from the 6.1 release notes:

- #1338: g.getUrlFromNode must not expand path expressions.

- #1341: Expansion of path expressions should be strictly limited.

My only memory of these issues is that path expressions must not execute arbitrary code!

It looks like the fixes happened before Leo started using PRs, so it's much harder to pinpoint the actual changes.

Edward

[Thomas Passin]

Edward made some changes during this thread, IIRC.

[jkn]

On Wednesday, January 10, 2024 at 1:00:58 PM UTC tbp1...@gmail.com wrote:

Edward made some changes during this thread, IIRC.

Yes ... I was trying to track the end results of this.

The description of {{sep}} etc. used to be on the leoeditor.com documentation,

but no longer seems to be there.

A short 'migration guide' paragraph would be useful, it was not clear to me

that my nodes with this style of directives need to be changed.

[Thomas Passin]

I just did some testing on Windows, and with an @file node Leo acts as if the os.path functions expanduser() and expandvars() are applied, as well as the  abspath() that would be expected (and probably realpath() but I didn't check that).  This means that you can set an environmental variable, say VAR, and use it in the path expression as $VAR.  I didn't find any variables that seem to have been added by Leo.

I didn't find where in LeoPyRef.leo this all happens.

[jkn]

Yes, I ended up with

    @path ${NEXTCLOUD}/dir1/dir2

        @file myfile.txt

(I prefer using ${NEXTCLOUD} to $NEXTCLOUD, but that is a very minor point)

one thing I note is that leo says eg:

    wrote file myfile.txt

where I would prefer it wrote the full path.

I now seem to have quite a few files scattered around called "{{g.app.homedir}}{{sep}}"...

    J^n