I have two programs that I'm working with. One is a perl script that I
use to download data from the net and the other is a Lisp program that I
use to analyze the data. Instead of me manually calling the data
retrieval program ten thousand time a day, I would like to have the Lisp
program call it automatically. Are there any lisp commands that would
allow me to do this? I'm using clisp on a Red Hat 8 system.
Thanks
-Colin
Colin> Hello All I have two programs that I'm working with. One
Colin> is a perl script that I use to download data from the net
Colin> and the other is a Lisp program that I use to analyze the
Colin> data. Instead of me manually calling the data retrieval
Colin> program ten thousand time a day, I would like to have the
Colin> Lisp program call it automatically. Are there any lisp
Colin> commands that would allow me to do this? I'm using clisp
Colin> on a Red Hat 8 system.
Check out the execute command:
[12]> (execute "/bin/ls" "-l" "/bin")
total 2656
-rwxr-xr-x 1 root root 47464 Aug 3 15:15 afio
-rwxr-xr-x 1 root root 2684 Oct 7 10:46 arch
lrwxrwxrwx 1 root root 4 Oct 30 23:20 ash -> dash
-rwxr-xr-x 1 root root 580776 Nov 4 03:02 bash
-rwxr-xr-x 1 root root 13864 Oct 31 20:58 cat
-rwxr-xr-x 1 root root 16296 Oct 31 20:58 chgrp
-rwxr-xr-x 1 root root 16040 Oct 31 20:58 chmod
-rwxr-xr-x 1 root root 18184 Oct 31 20:58 chown
-rwxr-xr-x 1 root root 42664 Oct 31 20:58 cp
-rwxr-xr-x 1 root root 47656 Jun 22 15:09 cpio
[ snip ]
--
Cleaning your house while your kids are still growing is like shoveling
the walk before it stops snowing.
-- Phyllis Diller
[3]> (defvar *ls* (make-pipe-input-stream "ls -l")))
*LS*
[4]> (read-line *ls*)
"total 1130" ;
NIL
[5]> (read-line *ls*)
"-rw-r--r-- 1 kaz kaz 1873 Sep 24 12:47 .viminfo" ;
NIL
[6]> (read-line *ls*)
"-rwxr-xr-x 1 kaz kaz 215476 Oct 30 08:24 a.out" ;
NIL
Consider:
(run-program "<your perl script>")
run-program's syntax is dependent on your implementation.
Chris
Why can't you just say
(POSIX::system "perl program")
Doesn't lisp have standard POSIX bindings? How could that be
possible?
What does "lisp" mean here?
--
Erik Naggum, Oslo, Norway
Act from reason, and failure makes you rethink and study harder.
Act from faith, and failure makes you blame someone and push harder.
R5RS, of course! That's the version of Lisp whose unique identifier
has exactly four letters :-).
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://www3.sympatico.ca/cbbrowne/spreadsheets.html
Rules of the Evil Overlord #50. "My main computers will have their own
special operating system that will be completely incompatible with
standard IBM and Macintosh powerbooks."
<http://www.eviloverlord.com/>
(The following is not flame-bait; please don't take it that way.
Imagine that I have a very sincere and sad expression on my face.)
Any non-toy programming language produced in the last decade or two.
My impression is that the Common Lisp community wants a sort of Lisp
Machine Shell, and to ignore or hide the real system. I believe
that---in a general purpose programming langauge---access to the real
system is what makes the language useful. That's why e.g. perl has a
standard POSIX module and an easy way to access stuff written in C.
In contrast, I couldn't find any mention of POSIX in the Common Lisp
Hyperspec, and lisp implementors usually demote native functions to
something you access via a "foreign function interface". That's all
very toy-like, albeit a very big and resource intensive toy.
Does all that sound bitter? I'd really like to be able to use lisp
(any flavor) to do some real programming, but I've never seen an
implementation that suits my needs. Most don't even come close.
Somehow their goals are totally at odds with those of a typical unix
programmer. (Siskind's Stalin is the right idea, but it's too much of
a research prototype, and anyway that's Scheme, which I know you
hate.)
Then I will offer my sincere critique of what you have written. Imagine
that I have a skeptical and sometimes sarcastic expression on my face.
> My impression is that the Common Lisp community wants a sort of Lisp
> Machine Shell, and to ignore or hide the real system. I believe
Would so much effort have been made to make Common Lisp portable if all
that was wanted was a Lisp Machine Shell?
Common Lisp doesn't even dictate the size of a byte.
> that---in a general purpose programming langauge---access to the real
> system is what makes the language useful. That's why e.g. perl has a
> standard POSIX module and an easy way to access stuff written in C.
So, actually, what's important is POSIX and only POSIX? POSIX is so
important, and so unique, that it should've been canonicalized up there
with CONS?
> In contrast, I couldn't find any mention of POSIX in the Common Lisp
> Hyperspec, and lisp implementors usually demote native functions to
Have you bothered to consider looking elsewhere? Just because something
is not in the standard doesn't make it impossible.
> something you access via a "foreign function interface". That's all
> very toy-like, albeit a very big and resource intensive toy.
Very few languages let you write foreign language bindings from within
the language itself. What is so toy-like about it? Maybe you haven't
considered the true depth of the problem; in light of which, efforts such
as ALIEN (CMUCL) are quite incredible.
(1) Lisps usually have much different internal representations for data
than the local C compiler. This is to facilitate garbage collection
as well as dynamic type-checking.
(2) Any sort of copying garbage collector is going to make it difficult
for a (brain-dead) C program to hold onto pointers into Lisp space.
(3) The semantics of C APIs are usually shit. If you truly care about
writing proper C code, each syscall you make will probably have to be
accompanied by several lines of setup, error-checking, and possibly
recovery code. Turning these poor C interfaces into proper Lisp
interfaces takes more than simply mapping the function into a Lisp one.
(Not to mention the cases where the C API is just downright bad!)
(4) How many C programs have the reliability and robustness of even the
simplest Lisp program in the Lisp environment? Bugs in the various
C libraries--even in memory handling--are going to bite you twice as
hard because you aren't constantly stopping and restarting your
program (and letting the OS clean up the mess you made).
Whereas the typical half-assed Unix "scripting language" tends to buy
its C interface by being written in C; and thus sharing all of C's
mistakes (or else they are confronted with similar problems).
> Does all that sound bitter? I'd really like to be able to use lisp
> (any flavor) to do some real programming, but I've never seen an
> implementation that suits my needs. Most don't even come close.
You don't sound like you really tried. This is from the perspective of
someone who has used Common Lisp to do all sorts of what you call ``real
programming'' (though I imagine that it would be called ``grunt work''
by most programmers. Playing stupid games with Unix nuttiness is not
real programming).
> Somehow their goals are totally at odds with those of a typical unix
> programmer.
Yes; their goal is to have robust and portable code. This is entirely
at odds to the typical Unix programmer, who could never conceive of an
environment different from their own (or even, these days, of a byte
size that isn't an octet!).
> (Siskind's Stalin is the right idea, but it's too much of
> a research prototype, and anyway that's Scheme, which I know you
> hate.)
It's not that it's Scheme; it's that it's whole-program analysis. Using
a compiler such as Stalin indicates that you believe your program to be
bug-free and feature-complete; and therefore are willing to throw away
all the advantages of a normal Lisp environment with regard to error
handling and runtime code update.
--
; Matthew Danish <mda...@andrew.cmu.edu>
; OpenPGP public key: C24B6010 on keyring.debian.org
; Signed or encrypted mail welcome.
; "There is no dark side of the moon really; matter of fact, it's all dark."
Well, up to a point Lord Copper. It's not that often you'll see a perl
programmer calling out to the POSIX module directly; the interfaces
are usually so 'uncomfortable' from a Perl point of view. And I don't
think even its best friend would call XS 'easy' for anything but the
most trivial of hacks.
> In contrast, I couldn't find any mention of POSIX in the Common Lisp
> Hyperspec, and lisp implementors usually demote native functions to
> something you access via a "foreign function interface". That's all
> very toy-like, albeit a very big and resource intensive toy.
Coming back once more to the Perl POSIX module you seem to like, that
module is implemented via a 'foreign function interface', it's just
called 'XS' when you're writing in Perl. The Perl programmer cannot
rely on having all that functionality available, which is why you'll
find 'Pure Perl' implementations of all sorts of stuff that's
available directly in POSIX.
--
Piers
"It is a truth universally acknowledged that a language in
possession of a rich syntax must be in need of a rewrite."
-- Jane Austen?
Yes. That is what portability means.
| In contrast, I couldn't find any mention of POSIX in the Common Lisp
| Hyperspec, and lisp implementors usually demote native functions to
| something you access via a "foreign function interface". That's all very
| toy-like, albeit a very big and resource intensive toy.
Amazing.
Yes, that's exactly the problem: Implementors porting the lisp machine
style interface, instead of just lisp. Gabriel even writes about it
in his book: Lucid was forced to do that, even though he knew better.
> So, actually, what's important is POSIX and only POSIX?
If you have to ask....
> Have you bothered to consider looking elsewhere? Just because something
> is not in the standard doesn't make it impossible.
It's stupid to rely on implementation specific hacks. I thought you
were hot on portability?
> > something you access via a "foreign function interface". That's all
> > very toy-like, albeit a very big and resource intensive toy.
>
> Very few languages let you write foreign language bindings from within
> the language itself.
I don't want foreign language bindings. I want *native* language
bindings.
> Maybe you haven't
> considered the true depth of the problem;
I have, but I don't care at all. Either lisp can cut the mustard, or
it isn't useful. Not my problem.
> Yes; their goal is to have robust and portable code. This is entirely
> at odds to the typical Unix programmer, who could never conceive of an
> environment different from their own (or even, these days, of a byte
> size that isn't an octet!).
Ok, now we are down to the typical sort reply. Ad-homenim attacks on
the unbelievers. Why not just accept that unix programer's main goal
is robust and portable code, but that common lisp implementors haven't
mangaged to provide anything that they can actually use?
I mean, look, I'm the guy who wants to use lisp. I see the light!
Now I just need a compiler that works!
> It's not that it's Scheme; it's that it's whole-program analysis. Using
> a compiler such as Stalin indicates that you believe your program to be
> bug-free and feature-complete; and therefore are willing to throw away
> all the advantages of a normal Lisp environment with regard to error
> handling and runtime code update.
Yes, exactly. I want to throw away the DISadvantages of a normal Lisp
environment, in favor of the vastly superior unix environment in which
my computer actually operates.
It's a native function interface, and the core perl functions are
implemented using the same machinery. Take a look at pp_sys.c and see
for yourself.
Have you looked at the Ada POSIX bindings?
| Ok, now we are down to the typical sort reply. Ad-homenim attacks on
| the unbelievers. Why not just accept that unix programer's main goal
| is robust and portable code, but that common lisp implementors haven't
| mangaged to provide anything that they can actually use?
You have /got/ to be kidding! (Or are incredibly ignorant.)
| I mean, look, I'm the guy who wants to use lisp. I see the light! Now I
| just need a compiler that works!
Exactly what is "just lisp"?
>
> > So, actually, what's important is POSIX and only POSIX?
>
> If you have to ask....
No, I am sarcastically stating your position. You define portability by
conformance to POSIX. Lisp programmers do not.
>
> > Have you bothered to consider looking elsewhere? Just because something
> > is not in the standard doesn't make it impossible.
>
> It's stupid to rely on implementation specific hacks. I thought you
> were hot on portability?
If you can understand the irony in the statement "portable access to the
/etc/passwd file" then you can understand what is wrong with your
response.
> > > something you access via a "foreign function interface". That's all
> > > very toy-like, albeit a very big and resource intensive toy.
> >
> > Very few languages let you write foreign language bindings from within
> > the language itself.
>
> I don't want foreign language bindings. I want *native* language
> bindings.
Explain the distinction. You do realize that the native language for
most Common Lisp environments is... Common Lisp.
>
> > Maybe you haven't
> > considered the true depth of the problem;
>
> I have, but I don't care at all. Either lisp can cut the mustard, or
> it isn't useful. Not my problem.
Because you are so ignorant, and continue to insist on being so, you
won't realize that FFI is necessary when internal representations do not
match. And C's internal data representations are not suitable for use
by Lisp environments (at least, ones worth using).
And what is wrong with FFI's? They work, and are pretty easy to use for
most tasks. As I was saying before, you seem like you don't even try.
> > Yes; their goal is to have robust and portable code. This is entirely
> > at odds to the typical Unix programmer, who could never conceive of an
> > environment different from their own (or even, these days, of a byte
> > size that isn't an octet!).
>
> Ok, now we are down to the typical sort reply. Ad-homenim attacks on
> the unbelievers. Why not just accept that unix programer's main goal
> is robust and portable code, but that common lisp implementors haven't
> mangaged to provide anything that they can actually use?
``You are a fool.'' That is ad-hominem.
What I say about the typical Unix programmer, on the other hand, is
quite true. I had a discussion on #C on just this topic just a few
weeks ago, and the typical response was ``but of course a byte is 8
bits. how could it be any other way?'' Now let us consider the tons of
C code written which depends on this. It is probably similar in scale
to the problem of depending on 32-bit ints and 8-bit chars.
In the C world, the result of dividing 1 by 2 is 0. Fun.
if (!strcmp (s1, s2)) is considered an acceptable idiom for testing
whether two strings DO match.
Looking at the main.c file of ProFTPD, I see:
- 4 usages of time() which don't check for errors
- a usage of select() that doesn't check for errors
- 3 usages of close() that don't check for errors
- a usage of dup() that doesn't check for errors
Each of these could potentially be a source of mysterious bugs.
Need I go on? I only made it halfway through the file. And even though
the ProFTPD maintainers have made an effort to error-check many things,
they missed these.
Such is the vaunted robustness of Unix programs.
Your own attitude actually illustrates my point just as well. In your
world, portability means POSIX-compliance. Woe betide the system that
dares to not be POSIX.
>
> I mean, look, I'm the guy who wants to use lisp. I see the light!
> Now I just need a compiler that works!
>
> > It's not that it's Scheme; it's that it's whole-program analysis. Using
> > a compiler such as Stalin indicates that you believe your program to be
> > bug-free and feature-complete; and therefore are willing to throw away
> > all the advantages of a normal Lisp environment with regard to error
> > handling and runtime code update.
>
> Yes, exactly. I want to throw away the DISadvantages of a normal Lisp
> environment, in favor of the vastly superior unix environment in which
> my computer actually operates.
Why do you want to use Lisp, then? If you feel that the Unix
environment is "so superior", then obviously you have no need to get
involved with alternatives. What use is the portability of Lisp
programs, after all? Everyone uses Unix these days! So why support
anything else? Everyone knows that a program crash is an acceptable way
to deal with errors. Why bother having a condition system? Everyone
knows that 1/2 == 0 is close enough, why bother having rational
arithmetic? Aren't floating-point numbers good enough? Aren't we
patient enough to wait for the compile-link cycle to complete, for minor
updates? Can't we tolerate some downtime every so often, so we can add
new functionality to running programs?
Obviously, Lispers are too smug and self-satisfied to appreciate the
Unix goodness they've got. We should thank our lucky stars that we have
Unix and C to save us! How could we possibly improve on Unix? It is
perfect; the end result of 30 years of stagna^H^H^H^H^H^Hevolution!
Is my sarcasm too subtle?
Great! So what's the portable way to do what the original poster asked for?
"Using Lisp to Call another program in linux?"
That's hardly asking too much. Lisp systems usually provide a
non-standard, non-portable way to do it.
I suggested (POSIX::system "some program") as the obvious solution,
but one that would require the lisp community to recognise that there
is a standard, portable, API, that's almost always available,
certainly available if you're running on any form of unix or windows.
But that's just me being absurd, right, because it's contrary to the
Unix Hater's philosophy, or something.
It seems that you have some problems realizing that the request /cannot/
be portable to begin with.
If you think POSIX is portable, then only POSIX-validated systems count.
This is like saying Windows is portable because only Windows platforms
count.
| I suggested (POSIX::system "some program") as the obvious solution,
To a Common Lisp programmer, two colons means "this is wrong".
| but one that would require the lisp community to recognise that there is
| a standard, portable, API, that's almost always available, certainly
| available if you're running on any form of unix or windows.
But Common Lisp does not thusly limit itself.
It's just that I'm willing to settle for 90% portable.
But you are right... I don't understand why all the unix lisp
implementors cannot agree on what to name the lisp interface to the
standard function for running another program on unix. That's just
perverse.
> If you think POSIX is portable, then only POSIX-validated systems count.
> This is like saying Windows is portable because only Windows platforms
> count.
Windows includes POSIX; MS says so.
While you are happy to nit-pick this into oblivion, perl and python
have posix packages and get the job done with no fuss at all.
> | but one that would require the lisp community to recognise that there is
> | a standard, portable, API, that's almost always available, certainly
> | available if you're running on any form of unix or windows.
>
> But Common Lisp does not thusly limit itself.
Having a posix package, whereever possible, would not limit Common
Lisp. It would make it much more useful, though.
My objection is to the name "foreign". Some people want lisp to be a
universal shell that hides you from the real system. They call the
real system's library functions "foreign", because they want to demote
and discourage them. In contrast Java implementors call the same
functions "native", because they *are*, and getting access to them is
necessary and desirable.
So, a really good FFI is fine. Just call it a NFI, and encourage it's
use. Then when someone asks "Using Lisp to Call another program in
linux?", you just say, "Use the NFI to call the standard system()
routine, just like in perl."
> Your own attitude actually illustrates my point just as well. In your
> world, portability means POSIX-compliance. Woe betide the system that
> dares to not be POSIX.
My other computer runs Plan 9.
> Why do you want to use Lisp, then? If you feel that the Unix
> environment is "so superior", then obviously you have no need to get
> involved with alternatives.
Lisp as a language has a lot of nice features. I want those features
without the "lisp is a shell and holds the whole world" baggage.
Stalin, Gwydion, and similar compilers make a good stab at what I
want, but they fall short in practical ways. After reading lots of
Erik Naggum's postings I was inspired to try a full featured system
like common lisp, on the theory that a bigger and more serious
community would translate to better engineering. But alas, not.
In that case, Common Lisp is not known to comply.
| I don't understand why all the unix lisp implementors cannot agree on
| what to name the lisp interface to the standard function for running
| another program on unix.
Neither can I.
| While you are happy to nit-pick this into oblivion, perl and python
| have posix packages and get the job done with no fuss at all.
What truly amazes me is that you do not understand that Perl and Python
are /only/ implemented on POSIX systems and have no specification at all.
| > | but one that would require the lisp community to recognise that there is
| > | a standard, portable, API, that's almost always available, certainly
| > | available if you're running on any form of unix or windows.
| >
| > But Common Lisp does not thusly limit itself.
|
| Having a posix package, whereever possible, would not limit Common Lisp.
| It would make it much more useful, though.
Look, is this really hard for you? Common Lisp is not limited to running
on "any form of Unix or Windows", /meaning/, it runs on /more than/ Unix
and Windows and it cannot limit standard functionality that which works
on Unix and Windows because of Posix. Why do you think we have pathnames
instead of simply using strings? I assume you favor removing them, like
some of the people working for a well-known Common Lisp vendor do, but
that does not change the fact that Common Lisp is supposed to be portable
to /more/ systems than those that can conveniently work with strings.
One thing I cannot understand about Unix is why they were so incredibly
stupid as to use slash-delimited "strings" in the system call interface,
which makes it impossible even to quote the slash if it is desired in a
filename. This is so fucking unbelievably retarded that the morons who
thought this up should have been shot. The solution is /really/ simple.
Instead of a string with slash-delimited components that the kernel has
to hack up into substrings, convert all non-escaped slashes to NUL bytes
in the caller and thus use the illegal NUL for both directory and string
terminator. An initial NUL byte would mean the pathname is absolute.
The first empty string to follow in the list of strings would terminate
the pathname. (That is, two NUL bytes instead of one.) This would allow
the entire character set to be used in filenames and nobody would have to
be mortally afraid of any other character encoding that could clobber the
magic #x2F byte value. This braindamaged aspect of the Unix design shows
what kind of simple-minded approach these guys took to their types.
Wrong.
| In contrast Java implementors call the same functions "native", because
| they *are*, and getting access to them is necessary and desirable.
But interestingly, what is "native" is a subset of what is "foreign".
| Lisp as a language has a lot of nice features. I want those features
| without the "lisp is a shell and holds the whole world" baggage.
You may be surprised by this, but many of us in the Common Lisp community
have been there. I have, for instance, bemoaned the lack of ability to
use the entire machine word for integer arithmetic. When I implement
various things that need to talk to C or other foreign languages (grin), I
am constantly reminded of this very painful difference. I want to write
code for algorithms that have been optimized for (expt 2 n) architectures,
and I am usually quite unwilling to settle for (1- n), but the performance
of most Common Lisp code that works with integers that can be larger than
fixnums yet within the machine word range can range from good to abysmal
depending on the bignum performance and the randomness of boxing them.
I have also found the moronic system call interface in C to be a pain in
entire intestine because of the incredible braindamage in the use of
small integers for everything. Designing a sensible interface to Unix is
not an easy task. The fact that a preprocessor had to be invented to make
the code look half decent with those "macros" that turn into magic numbers
is quite the extraordinary defeat in language design in my view. If one
were to design a Common Lisp look-alike for this interface style, a large
number of symbols would have to be made visible to the programmer, not
just the compiler, as in C.
| After reading lots of Erik Naggum's postings I was inspired to try a full
| featured system like common lisp, on the theory that a bigger and more
| serious community would translate to better engineering. But alas, not.
Well, I take that as a compliment.
Actually perl runs on non-POSIX systems, like Plan 9 and MS-DOS, but
that doesn't matter. Also, python does have a spec and multiple
implementations, but that doesn't matter either. The point is, they
do the best they can under the circumstances.
> Look, is this really hard for you? Common Lisp is not limited to running
> on "any form of Unix or Windows", /meaning/, it runs on /more than/ Unix
> and Windows and it cannot limit standard functionality that which works
> on Unix and Windows because of Posix.
Right, so why not have a package, which is standard, but only
available on a subset of systems you support (the POSIX ones). Right
now your Common Lisp code has lots of conditionals to handle
implementation differences, so that's obviously not a problem. The
improvement I'm proposing is simply that you will have standard names
for, and access to, things that exist on 90% of the computers (99% of
the users!) where Common Lisp does run, so you have an easier time and
fewer implementation specific conditionals.
Perl and Python and other languages do that, but hey, you're the
expert. If it's not worth it, then don't bother.
As far as I can see, each of the Perl and Python vendors have chosen a
different strategy for their operating-system interface. That there is
only one vendor of each language makes it appear that all the vendors
agree, but it is equally trivially true that all vendors disagree. Or, in
Common Lisp terms, (= x) and (/= x) both yield true. So in this regard,
what is better for Perl and Python is that there is one implementation of
each (albeit ported), not that they all agree on anything. I think having
only one implementation of Common Lisp would be a dramatic setback, even
with the clear benefits that derive from it.
| Right, so why not have a package, which is standard, but only available
| on a subset of systems you support (the POSIX ones).
That would be good. An abstraction of the operating system services and
objects into Common Lisp would be fantastically welcome, precisely
because it is so goddamn hard to do.
| Right now your Common Lisp code has lots of conditionals to handle
| implementation differences, so that's obviously not a problem.
The problem is that there is no /one/ way to make the abstraction that
people can agree on it. Let us face the fact squarely: Common Lisp is
very similar to Unix in many ways, but for that reason is different in
ways that are hard to reconcile. Where Common Lisp functions accept
discrete values as input and return discrete values to the top-level,
Unix programs read a stream of input and produce a stream of output.
Where Common Lisp functions accept arguments of diverse types that are
read by the standard reader before the function is called, Unix programs
receive strings that are merely tokens of the input and have to interpret
them itself. Where Common Lisp functions receive keyword arguments with
values, Unix programs accept options as special strings. Where Common
Lisp functions may receive nested lists of values, Unix programs receive
a single list as arguments or parse the input stream. Where Common Lisp
offers special variables with globally visible changes, Unix programs
receive environment variables that are only inherited in the process tree.
Where Common Lisp functions resolve relative pathnames with respect to a
special variable, Unix programs resolve them with respect to a process's
current working directory. Where Common Lisp functions run in the same
memory image and can share data effortlessly, Unix programs must make
special provisions for sharing memory. Where Common Lisp functions can
effortlessly use network streams and remote procedure calls, Unix programs
have to make special provisions for both. Where Common Lisp functions
can call on the system parser to process input from files, Unix programs
have to roll their own parsers for configuration files and other control
languages. Where Common Lisp functions all live in the image and are
reachable by packages and symbols, Unix programs live on disk and are
reachable by path and filename. The conceptual similarity should be
quite obvious all through this list because the same functionality is
needed, but the implementation strategy is so different that it may seem
that these are different things that can live side-by-side. I think they
cannot. A certain "mapping" of the concepts between implementations need
to take place that is far from trivial, and which probably (and evidently)
differs from occasion to occasion.
| The improvement I'm proposing is simply that you will have standard names
| for, and access to, things that exist on 90% of the computers (99% of the
| users!) where Common Lisp does run, so you have an easier time and fewer
| implementation specific conditionals.
But this is the same problem that faces the foreign function interfaces.
The same things exist on both ends, but the connection between them has
to differ because of other inherent differences.
| Perl and Python and other languages do that, but hey, you're the expert.
| If it's not worth it, then don't bother.
I think the case is rather different than you expect. It is not the job
of making a standard interface that is so daunting, it is maintaining it
across the product of all Common Lisp implementations and all POSIX
systems. This is rather more work than implementing it only across the
various operating systems. Making one POSIX package for Perl means
making a dozen for Common Lisp, and coordinating them is a managerial
task that is really hard to accomplish satisfactorily.
What I actually admire in Perl is its ability to provide a very successful
abstracttion of the horrible mess that is collectively called Unix.
> Matthew Danish <mda...@andrew.cmu.edu> writes:
>> Have you bothered to consider looking elsewhere? Just because something
>> is not in the standard doesn't make it impossible.
>
> It's stupid to rely on implementation specific hacks. I thought you
> were hot on portability?
There is no huge yawning gulf between "part of the language standard"
and "implementation specific hack", as well you know. Last time I
checked, Posix was not part of the C language specification.
If you want agreed names for access to Posix facilities on
implementations that run on Posix platforms, say so. Many people
might even agree with you (though I doubt they'd consider it the major
priority that you obviously do). But I don't see why they should be
part of the language standard any more than they are in C, and I don't
think you can come up with any credible explanation either.
-dan
--
http://ww.telent.net/cliki/ - Link farm for free CL-on-Unix resources
> use to analyze the data. Instead of me manually calling the data
> retrieval program ten thousand time a day, I would like to have the Lisp
> program call it automatically. Are there any lisp commands that would
> allow me to do this? I'm using clisp on a Red Hat 8 system.
Typing (apropos "program") at the clisp listener produced the
following output
[1]> (apropos "program")
PROGRAM-ERROR class
SYSTEM::PROGRAM-ID function
SYSTEM::PROGRAM-NAME function
RUN-PROGRAM function
SYSTEM::SIMPLE-PROGRAM-ERROR class
SYSTEM::SIMPLE-SOURCE-PROGRAM-ERROR class
SYSTEM::SOURCE-PROGRAM-ERROR class
RUN-PROGRAM looks promising, so:
[2]> (describe 'run-program)
RUN-PROGRAM is the symbol RUN-PROGRAM, lies in #<PACKAGE EXT>, is accessible
in the packages COMMON-LISP-USER, EXT, FFI, SCREEN, SYSTEM, names a function.
[...]
#<COMPILED-CLOSURE RUN-PROGRAM> is a compiled function.
Argument list: (ARG0 &KEY ARGUMENTS INPUT OUTPUT IF-OUTPUT-EXISTS WAIT).
For more information, evaluate (DISASSEMBLE #'RUN-PROGRAM).
So, at a guess, (run-program "your perl program") might very well do
the trick.
[4]> (run-program "/usr/games/fortune")
Be free and open and breezy! Enjoy! Things won't get any better so
get used to it.
0
The ARGUMENTS argument suggests that it doesn't do shell parsing, so
if you need to do that, you might consider either passing said list
[7]> (run-program "test" :arguments '("-d" "/"))
0
[8]> (run-program "test" :arguments '("-f" "/"))
1
or, depending on your requirement, passing the whole command line to a
shell
[9]> (run-program "sh" :arguments '("-c" "test -f /"))
1
[10]> (run-program "sh" :arguments '("-c" "test -d /"))
0
More detail is doubtless available from the clisp documentation
Hope this helps. For the benefit of Scott Schwartz, that took four
minutes to find out, including the time taken to type this reply.
> Right, so why not have a package, which is standard, but only
> available on a subset of systems you support (the POSIX ones). Right
Because that's not what "standard" means.
> now your Common Lisp code has lots of conditionals to handle
> implementation differences, so that's obviously not a problem. The
> improvement I'm proposing is simply that you will have standard names
> for, and access to, things that exist on 90% of the computers (99% of
> the users!) where Common Lisp does run, so you have an easier time and
> fewer implementation specific conditionals.
It would be a worthy project but perhaps not a terribly worthwhile
project. As compared to, say, a decent standard for extensible
streams, or a meta-object protocol, or multithreading[*], saving the user
four minutes per supported implementation to look up a function name
is not really compelling.
How worthwhile it is depends on what type of problems you are trying
to solve. For the problems I want to solve (which involve writing
various types of servers for POSIX based systems), having "standard"
POSIX bindings would be a help (i.e. I wouldn't have to keep writing
my own). On the other hand none of the items in your list would be of
any help to me. However, I don't dismiss them as not being terribly
worthwhile because of this.
I don't know about Python, but I'm reasonably confident that Perl runs
on non POSIX systems. And, as Scott seems to fail to realise, using
the POSIX module is usually the counsel of someone who doesn't realise
that there's a Better Way To Do It available either in the core
libraries or on CPAN. The POSIX interfaces are painfully C-like, and
the module itself is bloody enormous.
It's also the case that, in general, Perl appears to be moving away
from having system dependent functions as part of the core language,
instead we're seeing a steady exodus of such functionality into
support modules. It's already happened with the DBM interface, which
got moved into the Tie::* system some time ago, it's in the process of
happening with the Socket related functions, and from what Larry has
said in his design docs for Perl, that trend will only continue.
> Does all that sound bitter? I'd really like to be able to use lisp (any
> flavor) to do some real programming, but I've never seen an implementation
> that suits my needs. Most don't even come close.
Which implementations, which can't use the `port' package from clocc, do you
have in mind? The last time I looked clocc was working with ACL, clisp,
cmucl and gcl. I'd be surprised to find that it was difficult to add support
for other Common Lisp implementations.
,----
| davep@hagbard:~$ clocc
| [1]> (require :port-shell #p"clocc:src;port;shell")
| ;; Loading file /usr/local/src/clocc/src/port/shell.fas ...
| ;; Loading of file /usr/local/src/clocc/src/port/shell.fas is finished.
| t
| [2]> (port:run-prog "ls")
| DbaMGR.zip Nautilus archive develop etc lib nng_list.txt temp
| Mail News bin docs html music spool work
| 0
| [3]>
`----
> Somehow their goals are totally at odds with those of a typical unix
> programmer. (Siskind's Stalin is the right idea, but it's too much of a
> research prototype, and anyway that's Scheme, which I know you hate.)
The above solution has worked for me more than once. What sort of "typical
programmer" does that make me?
--
Dave Pearson: | lbdb.el - LBDB interface.
http://www.davep.org/ | sawfish.el - Sawfish mode.
Emacs: | uptimes.el - Record emacs uptimes.
http://www.davep.org/emacs/ | quickurl.el - Recall lists of URLs.
The predominant platform that runs on most of the planet's computers
has no POSIX entry points, and most of the programmers targetting that
platform don't give a damn about it.
POSIX is just another language that needs to be targetted at the
underlying host platform. Some of those targettings are quite
suboptimal. (Does Cygwin have copy-on-write fork() yet, for instance?)
A good POSIX implementation requires a POSIX-friendly kernel. This is
not even true of some Unix-like operating systems; for example, the
threading model in the Linux kernel causes problems for an
implementation of POSIX threads.
Much of POSIX is useful only to people writing in C. For example
printf() is POSIX (by way of inheritance from ANSI C). You have FORMAT
in Lisp, so you don't need that stupidity. You have garbage
collection, so malloc() is of no use. Many interfaces are idiotic and
so require thick wrapping, like any place where you have to specify a
fixed-size buffer for a string (e.g. readlink). Writing a binding for
a function like readlink() requires you to write a whole bunch of code
to get intelligent behavior that just returns a string. E.g. I use
this implementation, and target that instead of targetting readlink
directly:
char *impl_readlink(const char *path)
{
size_t size = 256;
char *str = malloc(size);
char *temp;
if (str == 0)
goto bail;
for (;;) {
int result = readlink(path, str, size);
if (result == -1)
goto bail;
if (result < size) {
str[result] = 0;
break;
}
if (size * 2 < size)
goto bail;
size *= 2;
if ((temp = realloc(str, size)) == 0)
goto bail;
str = temp;
}
return str;
bail:
free(str);
return 0;
}
Quite likely, no symbolic link will need 256 bytes of space, but you
never know. So this resizing logic is required to do it right every
time.
All decent Lisp implementations have a way to glue to C. But you have
already expressed your view elsethread that foreign function
interfaces ``demote'' the outside functions and are
resource-intensive.
I think that the foreign function languages offered by Lisp
implementations are more convenient than what implementations of other
languages offer. They leverage Lisp macros to create powerful and
useful mini-languages in which you can declare any C interface, and
even annotate it with semantics, like direction of data transfer of
pointer parameters, and memory management responsibility. When
targeting the above impl_readlink in CLISP, I can tell it that the
returned data came from malloc, so it automatically takes care of
freeing it. No programming is required other than annotating the
interface.
There can be various overheads in FFI calls, depending on
implementation. So what; use the interface intelligently, then. For
instance, if you are wrapping a graphics library written in C, don't
call put_pixel(x,y) a million times from Lisp to create an image.
Expose something less primitive that operates on entire raster lines
or image regions, and call that.
There is a system limit on the maximal length of symbolic links. You are
supposed to query the system for this value and allocate that much space.
Given the inimitable programmer-friendliness of C and Posix, the idea of
supplying a function that simply returned an allocated string would be
impossible.
That is irrational. Do you also object to ``negative number'' on the
grounds that it discourages subtraction of a larger quantity from a
smaller one? A FFI does what it does, regardless of how you spell it.
> Some people want lisp to be a
> universal shell that hides you from the real system. They call the
> real system's library functions "foreign", because they want to demote
> and discourage them.
That's crock. It takes someone a lot of work to develop a good FFI;
why would they discourage its use?
Maybe you live in a place where ``foreign'' is a dirty word or
something?
As in, ``those goddamned foreign functions are taking away jobs that
should be done in the Lisp image''. ;)
> So, a really good FFI is fine. Just call it a NFI, and encourage it's
> use. Then when someone asks "Using Lisp to Call another program in
Note that the term ``native'' has connotations of supremacy of the C
compilers's calling conventions. It's very inappropriate in the
context of a Lisp system that already generates native code. There is
a connotation behind the term ``native'' that an inefficient
interpreted system is gaining access to the processor (as in
``native'' code versus interpreted code). The term foreign captures
the idea that there are different calling conventions and differences
in data representation. The two realms are mutually foreign.
> linux?", you just say, "Use the NFI to call the standard system()
> routine, just like in perl."
This would be poor advice on two counts. Firstly, the Lisp you are
using may provide a way to run external programs as an extension, so
you'd be expending unnecessary effort to set up the FFI. Secondly, the
system() function calls the command interpreter, as in /bin/sh -c
"your command here". There can be, firstly, security issues there.
Secondly, you need to shell-escape the command argument list that you
want to run, so that the interpreter won't break tokens on your
whitespace, or interpret meta-characters. A call like (run-program
"vi" "<file>"), if implemented over system(), has to translate the
string to, for instance, "vi '<file>'" to protect the < > characters
from serving as redirection operators. For running external programs,
you really want a combination of fork(), exec*() and wait*(). That
extension for running programs may already provide a straightforward
interface to these three.
Wrong. To get useful POSIX, there is the Interix product which
Microsoft owns now. They probably bought it to bury it, because it
commoditizes their OS.
Then there is Red Hat's cygwin (www.cygwin.com).
> Having a posix package, whereever possible, would not limit Common
> Lisp. It would make it much more useful, though.
Even C doesn't have a POSIX package; the ANSI C standard does not
mention POSIX anywhere. ISO C++ doesn't mention POSIX anywhere.
Implementations of these languages have easy access to platform
functions, because those functions are defined using C headers.
Everyone else is a second-class citizen who must target the C. Lisp is
no worse off than any other non-C, non-C++ language. Lisp is better
off because of the ability to write sub-languages to do the job, so
you don't have to leave the language.
I predict that over the next few years, there will emerge a common FFI
syntax for all Lisps, and it will be de-facto standard (standard
meaning working with many implementations, not in the way you use the
word to denote the features of one-implementation-wonder languages
like perl). There is the UFFI project, the obvious candidate. It will
be easier to use C functions from Lisp than from C++, because there
will be header file parsing that will not barf on C++ keywords and
incompatible namespace rules.
By the way, what POSIX are you talking about anyway? POSIX is a moving
target, being bloated all the time. Do you mean 1990, or 200x POSIX or
something in between?
I agree that not all platforms support the POSIX semantics. However,
that isn't any reason not to provide an interface for the subset that
they do provide.
> Much of POSIX is useful only to people writing in C.
Perhaps so, but there is also much that can be used from any language.
The last two program I wrote used the following POSIX/Unix functions:
accept, bind, close, connect, dup2, execlp, fcntl, getsockopt,
gettimeofday, getuid, inet_aton, ioctl, kill, listen, open, pipe,
poll, read, recv, send, seteuid, setsockopt, sigaction, socket, wait,
and write. I used these all using the "foreign function interface" of
the particular implementation I chose. I accept having to do this as
the cost of doing business, but it would reduce the cost if they were
already available in some (semi-)standard library.
There is a non-C, non-C++ language with multiple implementations
that does have a standard Posix interface: SML. The details are at
http://cm.bell-labs.com/cm/cs/what/smlnj/doc/basis/pages/posix-chapter.html
Granted there aren't that many SML implementations and the
spec. doesn't cover the whole of POSIX but IMHO it is a step in the
right direction.
> Wrong. To get useful POSIX, there is the Interix product which
> Microsoft owns now. They probably bought it to bury it, because it
> commoditizes their OS.
The old Interix product is now part of Microsoft's "Windows Services
for UNIX" product.
> ... the ANSI C standard does not
> mention POSIX anywhere.
of course it does. even the old one:
ansi x3.159-1989, rationale at pp. 87:
"the comittee anticipates that IEEE 1003 will wish to standardize
the kill function in the POSIX specification." :)
> ... POSIX is a moving
> target, being bloated all the time. Do you mean 1990, or 200x POSIX or
> something in between?
professionals who have any interest in working with posix refer to
the recent open group standard IEEE Std 1003.1-2001 (ISBN 0-7381-3010-9).
anything else is history. if you search around, you may still be able to
find a pdf copy of the open group final draft. if not, order the CD set.
oz
---
given enough bugs, all eyeballs are shallow.
I've encountered Unix boxes where the calls to find
these maxima all returned 2**31, which was quite a bit
beyond what malloc() was prepared to believe. I, too,
ended up with a few grow-the-buffer-and-retry loops.
> My objection is to the name "foreign". Some people want lisp to be a
> universal shell that hides you from the real system. They call the
> real system's library functions "foreign", because they want to demote
> and discourage them. In contrast Java implementors call the same
> functions "native", because they *are*, and getting access to them is
> necessary and desirable.
How about `aboriginal', `primeval', or `autochthonous'?
Splendid! I'll go with autochthonous, since I'm feeling Jungian
today.
Chris
The analogy that I think inspired the term is "foreign language". If you
normally speak English, French is a foreign language, and vice versa.
No superiority is implied, they're just different.
So a FFI is a way for a program written in Lisp to call a function written
in C, as it's a foreign language from its perspective.
--
Barry Margolin, bar...@genuity.net
Genuity, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
> ... lisp implementors usually demote native functions to
> something you access via a "foreign function interface". That's all
> very toy-like, albeit a very big and resource intensive toy.
>
> Does all that sound bitter? I'd really like to be able to use lisp
> (any flavor) to do some real programming, but I've never seen an
> implementation that suits my needs. Most don't even come close.
> Somehow their goals are totally at odds with those of a typical unix
> programmer. (Siskind's Stalin is the right idea, but it's too much of
> a research prototype, and anyway that's Scheme, which I know you
> hate.)
For those who don't like to deal with "foreign function interfaces",
we just announced in this newsgroup our brand new Operating System
Interface module. I think it goes a long way toward doing what you
want, even though it's not specifically Posix oriented (it does
gather as many common Unix system functionalities as practical, and
even tries to include MS in the fold, though not all of the functionality
is appropriate in Windows systems). We're also about to announce the
newest version of our opensource FTP server, which uses this interface
and which now contains no FFI calls.
The original announcement can be found here:
http://makeashorterlink.com/?L39612D62
and it refers to our Tech Corner article which is here:
http://www.franz.com/support/tech_corner/index.lhtml#osi_patch
--
Duane Rettig du...@franz.com Franz Inc. http://www.franz.com/
555 12th St., Suite 1450 http://www.555citycenter.com/
Oakland, Ca. 94607 Phone: (510) 452-2000; Fax: (510) 452-0182
> The analogy that I think inspired the term is "foreign language".
> If you normally speak English, French is a foreign language, and
> vice versa. No superiority is implied, they're just different.
>
> So a FFI is a way for a program written in Lisp to call a function
> written in C, as it's a foreign language from its perspective.
Well, life is passing us by. My daughter takes French, and she got
into the French Honor Society. There was an awards ceremony. The
awards ceremony was not about foreign languages. It was about `world
languages'.
Personally I think we should keep calling them foreign function calls,
but if we need to change the name, call them `primitive' function
calls --- and yes, I am considering denotation here. :-)
--
Fred Gilham gil...@csl.sri.com || "If I thought there was anything at
all in your arguments, I should have to be not only a theist, but an
Episcopalian to boot," he said, after one interchange, reckoning that
since Episcopalianism was, in his book, that than which nothing could
be worse, this was an effective reductio ad absurdum. - J. R. Lucas
> My objection is to the name "foreign". Some people want lisp to be a
> universal shell that hides you from the real system. They call the
> real system's library functions "foreign", because they want to demote
> and discourage them.
So when I, say, describe French as a `foreign language' for English
speakers I am demoting and discouraging it? Right, of course!
--tim
> Right, so why not have a package, which is standard, but only
> available on a subset of systems you support (the POSIX ones). Right
> now your Common Lisp code has lots of conditionals to handle
> implementation differences, so that's obviously not a problem. The
> improvement I'm proposing is simply that you will have standard names
> for, and access to, things that exist on 90% of the computers (99% of
> the users!) where Common Lisp does run, so you have an easier time and
> fewer implementation specific conditionals.
Might I suggest that rather than sitting around complaining about how
you want something and causing everyone to dislike you, a better
approach might be to try and do something towards making it exist? I
think Posix bindings would be useful since I pretty much only use
Posix systems (well, I use windows as well, which is Posix in about
the sense that any MS system conforms to any standard, namely not at
all). However as there's a lot of work to do to make them suitable
for CL: many things need to have wrappers which provide some kind of
sensible error behaviour (such as signalling an error, not silently
returning some `can't happen' value), many other things need to be
changed to, say, handle pathnames as well as strings, and so on. It's
a good lot of work, but the result would be useful - even without an
implementation, a document on `suggested Posix bindings for CL' would
be useful.
--tim
> My objection is to the name "foreign". Some people want lisp to be a
> universal shell that hides you from the real system. They call the
> real system's library functions "foreign", because they want to demote
> and discourage them. In contrast Java implementors call the same
> functions "native", because they *are*, and getting access to them is
> necessary and desirable.
Funny you should mention that. The real system's library functions may be
called "foreign", but the Lisp syntax treats them as first class citizens.
Paolo
--
EncyCMUCLopedia * Extensive collection of CMU Common Lisp documentation
http://www.paoloamoroso.it/ency/README
What makes you think I haven't? My name is on some of the code in
scm, for example. (There used to be more, but it got subducted over
the years. That's scheme not common lisp, though, so maybe it doesn't
count.)
In any event, the fact that a lot of people here seem to think that it
wouldn't be of value dissuades me from getting involved.
Then, too, there were a lot of messages around here lately wondering
why python was more popular than lisp. Don't shoot the messenger for
answering your question.
This is a very odd sentiment. Are you not doing things mainly for your
own enjoyment?
| Then, too, there were a lot of messages around here lately wondering why
| python was more popular than lisp.
I must have missed these messages. But then I tend to ignore appeals to
popularity completely.
> What makes you think I haven't? My name is on some of the code in
> scm, for example. (There used to be more, but it got subducted over
> the years. That's scheme not common lisp, though, so maybe it doesn't
> count.)
Well, as far as I know you haven't done stuff on Posix bindings for
CL, which is what I was referring to. I didn't mean to claim you
hadn't done stuff on other things (which would be arrogant of me, to
put it mildly, and insulting), and I'm sorry if I gave that
impression.
> In any event, the fact that a lot of people here seem to think that it
> wouldn't be of value dissuades me from getting involved.
I think bindings to posix systems would be of value. Great value in
fact. I think it's a lot of work though, but I'd like to see it done.
> Then, too, there were a lot of messages around here lately wondering
> why python was more popular than lisp. Don't shoot the messenger for
> answering your question.
Not from me, though!
--tim