New library-search feature for scsh -- comments requested
Olin Shivers
appreciate feedback -- now is the time to get it right, before it's
released out into the world.
There's a long-standing problem with scripting in scsh that I'd like to fix
for scripts that need to load modules. In scsh, you can cause a script to load
in a file of code for a module by using the the
-lm <module-file>
switch on the command line (on the #! trigger line). But this requires you to
specify *too much* information, actually: you have to give the *complete* file
name, which is simply handed to LOAD. This turns out, in practice, to be a
real pain, because the system libraries may live in different directories
on different systems.
What you'd like to do, really, is say something like "look around for the
module file geometric-utils.scm, and load it in." That is, you'd like a
search path for the loader to go find libraries in some standard set of
library directories... where the definition of that "standard set" might
vary from installation to installation, or user to user.
Now, a really solid solution to this issue is going to need a real
redesign of the module loading system, which has to come from the S48
folks. But I am going to propose an interim fix for scsh users.
I am planning to add some new switches to the command-line parser.
-ll <module-file-name>
Load library module into config package.
This is just like the -lm switch, except that it searches the
library-directory path list for the file to load.
Specifically, it means: search through the LIBRARY-DIRECTORIES list of
directories looking for a module file of the given name, and load it
in.
The LIBRARY-DIRECTORIES list defaults to an installation-specific
value, which is typically
("/usr/local/lib/scsh/modules/")
If the environment variable $SCSH_LIB_DIRS is set, it is used
to determine the library search path. The value of this environment
variable treated as a sequence of s-expressions, which are "read"
from the string.
- A string is treated as a directory.
- #F is replaced with the default list
of directories.
A SCSH_LIB_DIRS assignment of this form
SCSH_LIB_DIRS='"." "/usr/contrib/lib/scsh/" #f "/home/shivers/lib/scsh"'
would produce this list of strings for the LIBRARY-DIRECTORIES list:
("." "/usr/contrib/lib/scsh/"
"/usr/local/lib/scsh/modules/"
"/home/shivers/lib/scsh")
Here is a sample bit of bash code that will add a subdir from
your home and project directories to the default list:
SCSH_LIB_DIRS="\"$HOME/lib/scsh\" \"/usr/project/lib/scsh\" #f"
Notice we had to backquote each string's double-quotes for bash.
Let's add the scsh process' current working directory to that list:
SCSH_LIB_DIRS="\".\" $SCSH_LIB_DIRS"
[One might consider allowing symbols as well as strings. The problem
is that this requires knowing how the reader deals with case when
reading symbols. Scsh's reader is case-preserving; R5RS's reader is
case-folding. I try to minimise scsh' *dependence* on case-preserving
symbol reading; the process notation is currently the only place where
it ever really matters.
The advantage is that setting the environment variable in bash
.profile or other init files becomes easier. The above example
becomes
SCSH_LIB_DIRS="$HOME/lib/scsh /usr/project/lib/scsh #f"
The backquoted-double-quotes go away, which is nicer.]
When searching for a directory containing a given library module,
nonexistent or read-protected directories are silently ignored;
it is not an error to have them in the LIBRARY-DIRECTORIES list.
It *is* a startup error if reading the $SCSH_LIB_DIRS env var causes
a read error, or produces a value that isn't a string or #f.
E.g., these are values of $SCSH_LIB_DIRS that will blow up scsh:
SCSH_LIB_DIRS="3.14 foo (3)" [Bogus values]
SCSH_LIB_DIRS="\"/usr/lib/scsh" [read error -- no close-quote]
SCSH_LIB_DIRS="." [read error -- illegal sexp]
Directory search can be recursive. A directory name that ends
with a slash is recursively searched. So this list
("/usr/local/lib/scsh/modules/" "/home/shivers/lib/scsh")
means
1. First search /usr/local/lib/scsh/modules and all its
subdirectories;
2. Then search /home/shivers/lib/scsh *non*-recursively.
Recursive search is depth-first, with directories ordered by the
Unix directory ordering, that is, the order in which ls(1) prints
out directories: ASCII alphabetical/lexicographical (which places
capitalised filenames before lower-case filenames), but dot files
sort before non-dot files.
+lp <lib-dir>
lp+ <lib-dir>
Add directory <lib-dir> to the beginning or end of the
LIBRARY-DIRECTORIES path list, respectively.
<lib-dir> is a *single* directory. It is not split at colons or
otherwise processed.
+lpe
lpe+
As above, except that ~ home-directory syntax and environment
variables are expanded out.
-lp-clear
-lp-default
Set the LIBRARY-DIRECTORIES path list to the empty list and the system
default, respectively.
These two switches are useful if you would like to protect
your script from influence by the $SCSH_LIB_PATH variable.
In these cases, the SCSH_LIB_PATH environment variable is never
even parsed, so a bogus value will not affect the script's
execution at all.
These switches and the $SCSH_LIB_PATH environment variable allow you to dump
useful modules into a central repository in, say /usr/local or your home
directory; your scripts can easily access them from these places.
As a design note, one might consider encoding the $SCSH_LIB_PATH variable in a
more tradtional Unix way, as a colon-delimited path list. There are some
problems with this approach, that would require us to "patch" the standard
Unix technique:
- Directories containing colons in them are not allowed -- which
is a problem, as /usr/foo:bar is a perfectly legal Unix file name.
We would need to make backslash a special quoting character to
handle this.
- Unix path lists usually use colon to *separate* file names,
not to *terminate* filenames. This is ambiguous in the empty-string
case. Is the empty string the empty list, or the singleton list
of the empty string -- i.e. does it parse as path list () or ("")?
We can patch this by going with a non-conventional colon-terminator
grammar for the path list encoding.
- We need a way to encode the system default path list.
We could adopt the TeX convention of using the empty string
to signify this. But this means we can't use empty string
as the root directory. We could either accept this limitation,
or change our slash-terminator-means-recursion convention to
a double-slash-terminator-means-recursion convention. Now we
can use "/" for the root directory, and empty string becomes
available.
With these three patches -- which give us a system rather different from
the standard (broken) Unix convention, we have something workable. This
strikes me as being neither fish nor fowl. If you are going to break with
convention, you might as well do something that makes it clear you have
departed from the standard method.
But I'm interested to hear people's opinions.
I am going to hack this into the development sources. As I said, I'd like to
hear from people what they think of the particulars of this design -- comments
or suggestions or criticisms would be greatly appreciated.
Thanks!
-Olin
Fernando D. Mato Mira
> I'm adding a new feature to scsh to support scripting, and I'd really
> appreciate feedback -- now is the time to get it right, before it's
> released out into the world.
>
> What you'd like to do, really, is say something like "look around for the
> module file geometric-utils.scm, and load it in." That is, you'd like a
> search path for the loader to go find libraries in some standard set of
> library directories... where the definition of that "standard set" might
> vary from installation to installation, or user to user.
>
> Now, a really solid solution to this issue is going to need a real
> redesign of the module loading system, which has to come from the S48
> folks. But I am going to propose an interim fix for scsh users.
Curiously enough, I'm fixing some code which is not searching for oreign object files
properly. I imagine you've already looked at the search-lists in the CL EXTENSIONS
package? The CMUCL LOAD facility is properly integrated with it, so it's all
explained
in its Users' Guide:
http://www.mindspring.com/~rtoy/software/cmu-user/node46.html#SECTION003134000000000000000
--
((( DANGER )) LISP BIGOT (( DANGER )) LISP BIGOT (( DANGER )))
Fernando D. Mato Mira
Real-Time SW Eng & Networking
Advanced Systems Engineering Division
CSEM
Jaquet-Droz 1 email: matomira AT acm DOT org
CH-2007 Neuchatel tel: +41 (32) 720-5157
Switzerland FAX: +41 (32) 720-5720
www.csem.ch www.vrai.com ligwww.epfl.ch/matomira.html
Mark Friedman
For Unix compatibility and minimal typing reasons I prefer a colon
delimited style path list. Here are my replies to Olin's comments
about that approach.
Olin Shivers <shi...@lambda.ai.mit.edu> writes:
> - Directories containing colons in them are not allowed -- which
> is a problem, as /usr/foo:bar is a perfectly legal Unix file name.
> We would need to make backslash a special quoting character to
> handle this.
Of course you have an equivalent problem with directory names
comtaining spaces in the case where you allow space delimited symbols
(e.g. SCSH_LIB_DIRS="$HOME/lib/scsh /usr/project/lib/scsh #f"),
although not with your initial space delimited strings approach (which
I really disliked). I don't think that makeing the backslash a quoting
character to be so heinous but you could instead use an old ed/sed/vi
trick and have the very first character of the path list be the
delimiter. You can even have a special character which begins the path
list to indicate that the next character is the delimiter (see my
comment below about #f to see why).
> - Unix path lists usually use colon to *separate* file names,
> not to *terminate* filenames. This is ambiguous in the empty-string
> case. Is the empty string the empty list, or the singleton list
> of the empty string -- i.e. does it parse as path list () or
> ("")?
I'm totally missing the point here. Why does it matter the empty
string denotes here? Could you give some concrete example?
> - We need a way to encode the system default path list.
> We could adopt the TeX convention of using the empty string
> to signify this. But this means we can't use empty string
> as the root directory. We could either accept this limitation,
> or change our slash-terminator-means-recursion convention to
> a double-slash-terminator-means-recursion convention. Now we
> can use "/" for the root directory, and empty string becomes
> available.
Is this is why you were worried about the empty string case? In any
case, it seems to me that you can still use #f here. If you accept
the restriction that pathnames must be absolute (i.e. start with a
slash) or start with dot then you have a large set of characters that
you can use for special path names like #f.
-Mark
Olin Shivers
Thanks for the review! I'm still sticking with my s-exp encoding, for the
following reasons:
From: Mark Friedman <bing...@yahoo.com>
Subject: Re: New library-search feature for scsh -- comments requested
Newsgroups: comp.lang.scheme.scsh
Date: 27 Oct 1999 13:54:31 -0700
For Unix compatibility and minimal typing reasons I prefer a colon
delimited style path list.
There's nothing with which to be compatible! No installed code base of
any kind uses the $SCSH_LIB_DIRS env var. We can make it be any format
we like. There is the issue of breaking with standard *conventions*. But
the standard conventions, in this case, are broken, so I'm less inclined
to weight this.
> - Directories containing colons in them are not allowed -- which
> is a problem, as /usr/foo:bar is a perfectly legal Unix file name.
> We would need to make backslash a special quoting character to
> handle this.
Of course you have an equivalent problem with directory names
comtaining spaces in the case where you allow space delimited symbols
(e.g. SCSH_LIB_DIRS="$HOME/lib/scsh /usr/project/lib/scsh #f"),
although not with your initial space delimited strings approach (which
I really disliked). I don't think that makeing the backslash a quoting
character to be so heinous but you could instead use an old ed/sed/vi
trick and have the very first character of the path list be the
delimiter. You can even have a special character which begins the path
list to indicate that the next character is the delimiter (see my
comment below about #f to see why).
Any of these approaches breaks with the existing Unix convention. What
do you think csh will do if you put a backslash-colon in the $PATH
variable? It will *still* split at the colon.
This kind of sorta-like-the-existing-convention-but-actually-different
convention is actually more dangerous than a clean break -- it's misleading.
> - Unix path lists usually use colon to *separate* file names,
> not to *terminate* filenames. This is ambiguous in the empty-string
> case. Is the empty string the empty list, or the singleton list
> of the empty string -- i.e. does it parse as path list () or
> ("")?
I'm totally missing the point here.
An example: how would you, in bash, stick a directory onto the end of $PATH?
Answer: you'd say
PATH="$PATH:/home/shivers/bin"
Oops -- what happens if the path is empty? Then you end up with
:/home/shivers/bin
(which denotes a *two* element path, which is a security hole, because now
you've just put the cwd into your search path; yikes.) when you really wanted
/home/shivers/bin
These ambiguities only bite you in boundary cases. But boundary cases
happen. Designing a system that will almost always work is not my idea of a
good thing, *especially* when it's so easy to do it right! By the way, here's
how you add a directory onto the end of PATH in sh correctly (this is adapted
from one of my dot files):
PATH=${PATH:+$PATH:}$DIR #path grammar sux big one
Isn't that straightforward? Blech. Infix grammars are losers.
> - We need a way to encode the system default path list.
> We could adopt the TeX convention of using the empty string
> to signify this. But this means we can't use empty string
> as the root directory. We could either accept this limitation,
> or change our slash-terminator-means-recursion convention to
> a double-slash-terminator-means-recursion convention. Now we
> can use "/" for the root directory, and empty string becomes
> available.
Is this is why you were worried about the empty string case? In any
case, it seems to me that you can still use #f here. If you accept
the restriction that pathnames must be absolute (i.e. start with a
slash) or start with dot then you have a large set of characters that
you can use for special path names like #f.
"foo" is a valid file name. I think it would be a little confusing to
tell people: "foo won't work, but ./foo will."
S-expression notation is a fairly general encoding mechanism. So instead of
coming up with some custom, patched encoding for a sequence-of-strings-
with-an-escape-for-the-default-sequence, I am just going to ride on top of
that encoding. Less to learn, and I've already got my parser.
As for minimal typing reasons: if I had to type these things in on every
command, I would probably find it onerous -- especially having to
backslash-escape the double-quotes when I was using env vars (e.g.,
SCSH_LIB_DIRS="\"$HOME/lib/scsh\" #f"
) would get old. But you *don't* do this stuff that much. It's mostly
in .profile or .login or .cshrc. Not that big a deal.
And if I threw in symbols-as-file-names, you wouldn't even have to
use the quotes for 99+% of the cases -- you'd be writing
SCSH_LIB_DIRS="$HOME/lib/scsh #f"
which is easy to write, and easy to read -- as long as your home directory
isn't a path name containing white space, like
"/Home Directories/Olin Shivers"
-Olin
David Rush
> - We could adopt the TeX convention of using the empty string
> to signify this. But this means we can't use empty string
> as the root directory. We could either accept this limitation,
> or change our slash-terminator-means-recursion convention to
> a double-slash-terminator-means-recursion convention. Now we
> can use "/" for the root directory, and empty string becomes
> available.
> With these three patches -- which give us a system rather different from
> the standard (broken) Unix convention
I'm not a scsh user, but I would like to say that the Unix convention
plus your patches is very much like kpathsearch in the TeX world
(which you cite). On the whole I think kpathsearch is brilliant,
although it would be very difficult (maintenance-wise) to have a
single kpathsearch db for an entire unix system. I particularly like
the way that it specifies recursive searches constrained by the
directory structure at both the head and the tail of the final path.
What I'm getting at, is why don't you use the library? It's a great
convention, it's already written, and IIRC S48 has an FFI, so you
might be able to just hack it in (handwave, handwave)...
Happy Hackin'
david rush
David Rush
> And if I threw in symbols-as-file-names, you wouldn't even have to
> use the quotes for 99+% of the cases -- you'd be writing
>
> SCSH_LIB_DIRS="$HOME/lib/scsh #f"
>
> which is easy to write, and easy to read -- as long as your home directory
> isn't a path name containing white space, like
> "/Home Directories/Olin Shivers"
Which is in fact happening more nad more with the encroachment of GUIs
into the proper dominion of pure-text. You might want to think about
it. OR not. Gripping hand, most *nix people already know how to use
escapes...
david rush
Christopher Browne
>Olin Shivers <shi...@lambda.ai.mit.edu> writes:
>> And if I threw in symbols-as-file-names, you wouldn't even have to
>> use the quotes for 99+% of the cases -- you'd be writing
>>
>> SCSH_LIB_DIRS="$HOME/lib/scsh #f"
>>
>> which is easy to write, and easy to read -- as long as your home
>> directory isn't a path name containing white space, like
>> "/Home Directories/Olin Shivers"
>
>into the proper dominion of pure-text. You might want to think about
>it. OR not. Gripping hand, most *nix people already know how to use
>escapes...
The problem with the way UNIX does things now is that it's continually
reparsing these things all the time.
The problem that comes up is that *any* possible choice of separators
will encroach on the characters that you can actually use as part of a
filename.
The fact that sometimes you get:
PATH=/oracle/HD1/bin:/usr/openwin:/usr/dt/bin:/opt/SUNWspro/bin:
/opt/SUNmfwm/bin:/usr/sap/HD1/SYS/exe/run:/bin:/usr/bin:/usr/sbin:\
/usr/bin/X11:/usr/ucb:/usr/ccs/bin:/usr/local/bin:\
/usr/local/bin/mvs:/etc:/usr/openwin/bin:/usr/bin:
and other times you get:
path=(/oracle/HD1/bin /usr/openwin /usr/dt/bin /opt/SUNWspro/bin
/opt/SUNmfwm/bin /usr/sap/HD1/SYS/exe/run /bin /usr/bin /usr/sbin
/usr/bin/X11 /usr/ucb /usr/ccs/bin /usr/local/bin /usr/local/bin/mvs
/etc /usr/openwin/bin /usr/bin)
all has the effect that:
a) Directories that you might want in your path can't contain colons,
and
a) Directories that you might want in your path can't contain spaces.
The "Lisp way" where you'd have a string for each one has the merit
that you can have *anything* inside the string.
This may be the advantage that the "Windows Way" brings; its
"advances" make it necessary to do something clearer than having text
that keeps getting repeatedly parsed...
--
"Let me get this straight:
A company that dominates the desktop, and can afford to hire an army of
the world's best programmers, markets what is arguably the world's LEAST
reliable operating system?
What's wrong with this picture?" -- <fr...@cc.UManitoba.CA>
cbbr...@ntlug.org- <http://www.hex.net/~cbbrowne/lsf.html>
Mark Friedman
> Thanks for the review! I'm still sticking with my s-exp encoding, for the
> following reasons:
Well, it's your ball.
> For Unix compatibility and minimal typing reasons I prefer a colon
> delimited style path list.
>
> There's nothing with which to be compatible! No installed code base of
> any kind uses the $SCSH_LIB_DIRS env var. ...
I meant conceptual compatibility with existing Unix conventions.
>
> > - Directories containing colons in them are not allowed -- which
> > is a problem, as /usr/foo:bar is a perfectly legal Unix
> > file name. We would need to make backslash a special
> > quoting character to handle this.
>
> Of course you have an equivalent problem with directory names
> comtaining spaces ... I don't think that makeing the backslash a quoting
> character to be so heinous but you could instead use an old ed/sed/vi
> trick and have the very first character of the path list be the
> delimiter. ...
>
> Any of these approaches breaks with the existing Unix convention. What
> do you think csh will do if you put a backslash-colon in the $PATH
> variable? It will *still* split at the colon.
But we're talking about a way to overcome something which will break
other Unix program anyway. Who cares if our fix breaks existing
programs in the case where it expresses something that is
unexpressable in those programs. We're not going to use it for those
programs.
>
> This kind of sorta-like-the-existing-convention-but-actually-different
> convention is actually more dangerous than a clean break -- it's misleading.
To me, There's an important distinction to be made between something
that's incompatibly different and something that is different becasue
it adds functionalit, i.e. is an extension.
> [confusion about empty paths]
>
> An example: how would you, in bash, stick a directory onto the end of $PATH?
> Answer: you'd say
> PATH="$PATH:/home/shivers/bin"
> Oops -- what happens if the path is empty? Then you end up with
> :/home/shivers/bin
> (which denotes a *two* element path, which is a security hole, because now
> you've just put the cwd into your search path; yikes.)
Who says that it denotes a two element path? This is a pathological
case that we should interpret as just /home/shivers/bin.
>
> > - We need a way to encode the system default path list.
>
> ... it seems to me that you can still use #f here. If you accept
> the restriction that pathnames must be absolute (i.e. start with a
> slash) or start with dot then you have a large set of characters that
> you can use for special path names like #f.
>
> "foo" is a valid file name. I think it would be a little confusing to
> tell people: "foo won't work, but ./foo will."
"foo" is a valid file name but we don't have to interpret it as a
valid path nale in a path list. My point is that we should could just
accept absolute path names and dot prefixed path names.
> S-expression notation is a fairly general encoding mechanism.
No argument here.
> So instead of coming up with some custom, patched encoding for a
> sequence-of-strings- with-an-escape-for-the-default-sequence, I am
> just going to ride on top of that encoding. Less to learn, and I've
> already got my parser.
That's fine. It's certainly a cleaner approach, but my point is that
you could come up with an encoding which is more compatible with the
Unix conventions and just enhances it.
> And if I threw in symbols-as-file-names, you wouldn't even have to
> use the quotes for 99+% of the cases -- you'd be writing
>
> SCSH_LIB_DIRS="$HOME/lib/scsh #f"
>
> which is easy to write, and easy to read -- as long as your home directory
> isn't a path name containing white space, like
> "/Home Directories/Olin Shivers"
>
> -Olin
So we still lose on spaces!? Sigh.
-Mark