As far as I know, the only two standard interfaces with the Lisp compiler are the functions COMPILE and COMPILE-FILE. The first compiles a single function, the other compiles a file (dealing properly with top-level forms etc. etc.).
Suppose I have code in text form, coming from an arbitrary stream (e.g. the network, or a compressed archive, or...): is (1)
? Probably not... (rhetorical question) but, exactly what kind of disaster can I expect if I use (2) instead of (1)? I'd like to avoid dealing with temporary files if I can...
Alessio Stalla <alessiosta...@gmail.com> writes: > Suppose I have code in text form, coming from an arbitrary stream > (e.g. the network, or a compressed archive, or...): is (1)
No, it's not. The first form executes the /run time compiler/, the second one the /file compiler/. They differ in how they deal with EVAL-WHEN, and how they coalesce constants.
> ? Probably not... (rhetorical question) but, exactly what kind of > disaster can I expect if I use (2) instead of (1)? I'd like to avoid > dealing with temporary files if I can...
Some implementations offer a COMPILE-FROM-STREAM. Look at how SWANK-COMPILE-STRING is implemented in the different SWANK backends. (The swank-foo.lisp files in the source checkout of Slime.)
Alessio Stalla <alessiosta...@gmail.com> writes: > Suppose I have code in text form, coming from an arbitrary stream > (e.g. the network, or a compressed archive, or...): is (1)
No, it's not. The first form executes the /run time compiler/, the second one the /file compiler/. They differ in how they deal with EVAL-WHEN, and how they coalesce constants.
> ? Probably not... (rhetorical question) but, exactly what kind of > disaster can I expect if I use (2) instead of (1)? I'd like to avoid > dealing with temporary files if I can...
Some implementations offer a COMPILE-FROM-STREAM. Look at how SWANK-COMPILE-STRING is implemented in the different SWANK backends. (The swank-foo.lisp files in the source checkout of Slime.)
On Feb 2, 8:18 pm, "Tobias C. Rittweiler" <t...@freebits.de.invalid> wrote:
> Alessio Stalla <alessiosta...@gmail.com> writes: > Some implementations offer a COMPILE-FROM-STREAM. Look at how > SWANK-COMPILE-STRING is implemented in the different SWANK > backends. (The swank-foo.lisp files in the source checkout of Slime.)
Thanks. I looked up the definition for ABCL, which is the implementation in which I'm interested, and... basically it calls (compile nil (read ...))
So I guess I'm out of luck. Temporary files will be, for now...
Wouldn't this be fit as one of those mythical additions to the ANSI standard? In a world where distributed computing is becoming increasingly important, isn't having the compiler tied to files only (in a language like Lisp, which has dynamic code compilation as one of its strong points) too much of a limitation? Especially given that it's probably trivial to adapt the compiler to work on streams other than files (at least, it seems to me it's trivial, but I know very little of how Lisp compilers work...).
> On Feb 2, 8:18 pm, "Tobias C. Rittweiler" <t...@freebits.de.invalid> > wrote:
> > Alessio Stalla <alessiosta...@gmail.com> writes: > > Some implementations offer a COMPILE-FROM-STREAM. Look at how > > SWANK-COMPILE-STRING is implemented in the different SWANK > > backends. (The swank-foo.lisp files in the source checkout of Slime.)
> Thanks. I looked up the definition for ABCL, which is the > implementation in which I'm interested, and... basically it calls > (compile nil (read ...))
> So I guess I'm out of luck. Temporary files will be, for now...
> Wouldn't this be fit as one of those mythical additions to the ANSI > standard? In a world where distributed computing is becoming > increasingly important, isn't having the compiler tied to files only > (in a language like Lisp, which has dynamic code compilation as one of > its strong points) too much of a limitation?
Why? Doesn't your OS provide you with some kind of 'tmp' directory?
> Especially given that > it's probably trivial to adapt the compiler to work on streams other > than files (at least, it seems to me it's trivial, but I know very > little of how Lisp compilers work...).
> Alessio
The idea of a file compiler allows also a multi-pass compiler, where one or passes of a Lisp compiler might generate some output and one or more passes of an external compiler (C, Assembler) might generate further output. Also note that the file compiler COMPILEs the file, but does not necessarily LOAD the resulting code. So, if a file compiler compiles (defun foo ...), after compilation there is possibly no trace of the function FOO not even then symbol in the runtime, where the compiler runs. So, a stream compiler is a different model. For a stream compiler that uses a C compiler to generate the native code, it still means writing and reading of files (the C code), even though you think it just compiles a stream.
One could specify a stream compiler in a standard, but it is some work to write down what it should do and how. It could work similar to some REPLs that read code expressions, compile those and execute those. Note that not all REPLs (even in compiled Lisps) compile the expressions in the REPL.
Alessio Stalla wrote: > On Feb 2, 8:18 pm, "Tobias C. Rittweiler" <t...@freebits.de.invalid> > wrote: >> Alessio Stalla <alessiosta...@gmail.com> writes: >> Some implementations offer a COMPILE-FROM-STREAM. Look at how >> SWANK-COMPILE-STRING is implemented in the different SWANK >> backends. (The swank-foo.lisp files in the source checkout of Slime.)
> Thanks. I looked up the definition for ABCL, which is the > implementation in which I'm interested, and... basically it calls > (compile nil (read ...))
> So I guess I'm out of luck. Temporary files will be, for now...
> Wouldn't this be fit as one of those mythical additions to the ANSI > standard? In a world where distributed computing is becoming > increasingly important, isn't having the compiler tied to files only > (in a language like Lisp, which has dynamic code compilation as one of > its strong points) too much of a limitation? Especially given that > it's probably trivial to adapt the compiler to work on streams other > than files (at least, it seems to me it's trivial, but I know very > little of how Lisp compilers work...).
It's trivial to do, not necessarily trivial to decide how to do it. When you're compiling a file, you have access to the entire file while you're doing it, and can make decisions based on (effectively) global knowledge about what you're compiling. If you're compiling just one s-expression, you know everything about that, and you know what you don't know about the rest of the universe. If you're compiling from a stream, what do you know? Are you compiling only the contents of streams that have been flushed and closed? everything that comes in, piece by piece? everything that comes in, but remembering (for some length of time) the ongoing compilation environment for literals etc? Depending on what kind of stream you're compiling from and why, the desired behavior will probably be obvious, but that doesn't necessarily make for good standardization.
> The idea of a file compiler allows also a multi-pass compiler, where > one or passes of a Lisp compiler might generate some output > and one or more passes of an external compiler (C, Assembler) > might generate further output. Also note that the file compiler > COMPILEs the file, but does not necessarily LOAD the resulting code. > So, if a file compiler compiles (defun foo ...), after > compilation there is possibly no trace of the function FOO > not even then symbol in the runtime, where the compiler > runs. So, a stream compiler is a different model. For a stream > compiler that uses a C compiler to generate the native > code, it still means writing and reading of files (the C code), > even though you think it just compiles a stream.
> One could specify a stream compiler in a standard, but > it is some work to write down what it should do and how. > It could work similar to some REPLs that read code expressions, > compile those and execute those. Note that not all REPLs > (even in compiled Lisps) compile the expressions in the > REPL.
Yes, I see. It makes much sense. Thanks for the detailed explanation.
I still think that having a portable compiler-from-stream, even if somehow limited, would be a good thing, but I understand it would be difficult to specify what it should do and how, and in some implementations it would be probably difficult or even impossible to implement it, too. That is, without resorting to temporary files. So, *sigh*, let's do this temp file thingie... did I mention I hate temp files? ;)
> On Feb 2, 9:08 pm, "jos...@corporate-world.lisp.de" <jos...@corporate-
> world.lisp.de> wrote: > > Why? Doesn't your OS provide you with some kind of 'tmp' directory?
> Yes, but having to explicitly deal with temp files (on multiple > OSes...) vs having it Just Work Out Of The Box... it's not the same > thing ;)
> > The idea of a file compiler allows also a multi-pass compiler, where > > one or passes of a Lisp compiler might generate some output > > and one or more passes of an external compiler (C, Assembler) > > might generate further output. Also note that the file compiler > > COMPILEs the file, but does not necessarily LOAD the resulting code. > > So, if a file compiler compiles (defun foo ...), after > > compilation there is possibly no trace of the function FOO > > not even then symbol in the runtime, where the compiler > > runs. So, a stream compiler is a different model. For a stream > > compiler that uses a C compiler to generate the native > > code, it still means writing and reading of files (the C code), > > even though you think it just compiles a stream.
> > One could specify a stream compiler in a standard, but > > it is some work to write down what it should do and how. > > It could work similar to some REPLs that read code expressions, > > compile those and execute those. Note that not all REPLs > > (even in compiled Lisps) compile the expressions in the > > REPL.
> Yes, I see. It makes much sense. Thanks for the detailed explanation.
> I still think that having a portable compiler-from-stream, even if > somehow limited, would be a good thing, but I understand it would be > difficult to specify what it should do and how, and in some > implementations it would be probably difficult or even impossible to > implement it, too. That is, without resorting to temporary files. > So, *sigh*, let's do this temp file thingie... did I mention I hate > temp files? ;)
> Thanks, > Alessio
For example: when you use COMPILE in GCL to generate native code, it also needs to call the external C compiler. Which means: tmp files.
Alessio Stalla <alessiosta...@gmail.com> writes: > Thanks. I looked up the definition for ABCL, which is the > implementation in which I'm interested, and... basically it calls > (compile nil (read ...))
> So I guess I'm out of luck. Temporary files will be, for now...
> Wouldn't this be fit as one of those mythical additions to the ANSI > standard? In a world where distributed computing is becoming > increasingly important, isn't having the compiler tied to files only > (in a language like Lisp, which has dynamic code compilation as one of > its strong points) too much of a limitation? Especially given that > it's probably trivial to adapt the compiler to work on streams other > than files (at least, it seems to me it's trivial, but I know very > little of how Lisp compilers work...).
Well, I suppose it would be somewhat interesting. The real question is what the return value of such a COMPILE-STREAM function would be. Perhaps it needs to be some type of STREAM that is compatible with the LOAD function.
The fundamental difference between COMPILE and COMPILE-FILE is that the first one produces a function, which is a first-class lisp object that can later be FUNCALLed, APPLYed, etc. This can be done directly in the lisp system.
The latter has the side effect of producing a compiled file and returns the file name of that file. To actually get the definitions into your lisp, you still have to load the compiled file. The difference is that if you were to do:
(compile 'factorial (lambda (x) (if (<= x 1) 1 (* x (factorial (1- x))))))
then you could immediately invoke
(factorial 3)
and get an answer. If instead, you were to call
(compile-file factorial.lisp)
where the file factorial.lisp contained
(defun factorial (x) (if (<= x 1) 1 (* x (factorial (1- x)))))
then you could NOT call
(factorial 3)
without an error unless you first LOADed the compiled version of the file.
So to make COMPILE-FILE work with a stream, one first has to figure out what it should return, since there isn't any place for it to put the compiled version of the stream contents. I suppose that one option might be to have it perform side effects on the lisp environment. In that case it would be more like a COMPILE-AND-LOAD-STREAM operation. I suppose another option would be for it to just produce a stream with the contents of the compiled file. But then one would also have to add a LOAD-STREAM function, since there isn't any current CL function that will load the contents of a stream into the lisp system.
So, this all could be doable, but there are still some issues that would need to be thrashed out first.
-- Thomas A. Russ, USC/Information Sciences Institute
Alessio Stalla <alessiosta...@gmail.com> wrote: > On Feb 2, 8:18 pm, "Tobias C. Rittweiler" <t...@freebits.de.invalid> > wrote: > > Alessio Stalla <alessiosta...@gmail.com> writes: > > Some implementations offer a COMPILE-FROM-STREAM. Look at how > > SWANK-COMPILE-STRING is implemented in the different SWANK > > backends. (The swank-foo.lisp files in the source checkout of Slime.)
> Thanks. I looked up the definition for ABCL, which is the > implementation in which I'm interested, and... basically it calls > (compile nil (read ...))
> So I guess I'm out of luck. Temporary files will be, for now...
I wonder about following: find any existing text file F. Define its first character as macro character and associate with it function that reads your stream (as a single form wrapped in progn) and than changes file position to the end of F. Now compile F -- I am not sure if this will work as expected, but in your place I would investigate such way.
On Feb 3, 4:20 am, Waldek Hebisch <hebi...@math.uni.wroc.pl> wrote:
> Alessio Stalla <alessiosta...@gmail.com> wrote: > > On Feb 2, 8:18 pm, "Tobias C. Rittweiler" <t...@freebits.de.invalid> > I wonder about following: find any existing text file F. Define > its first character as macro character and associate with it > function that reads your stream (as a single form wrapped in progn) > and than changes file position to the end of F. Now compile F -- I am > not sure if this will work as expected, but in your place I would > investigate such way.
!!! This solution is so diabolically ingenious that I've got to try it, even if it feels to me "dirtier" than the temp-file based approach ;) I'll report back with my findings...
> So to make COMPILE-FILE work with a stream, one first has to figure out > what it should return, since there isn't any place for it to put the > compiled version of the stream contents. I suppose that one option > might be to have it perform side effects on the lisp environment. In > that case it would be more like a COMPILE-AND-LOAD-STREAM operation.
In that case you could trivially implement it as:
(loop for form = (read stream nil stream) until (eq form stream) do (compile nil form)) ; only defuns
or perhaps:
(loop for form = (read stream nil stream) until (eq form stream) do (funcall (compile nil (lambda () ,form)))) ; any form
> I > suppose another option would be for it to just produce a stream with the > contents of the compiled file. But then one would also have to add a > LOAD-STREAM function, since there isn't any current CL function that > will load the contents of a stream into the lisp system.
> So, this all could be doable, but there are still some issues that would > need to be thrashed out first.
> -- > Thomas A. Russ, USC/Information Sciences Institute
> t...@sevak.isi.edu (Thomas A. Russ) writes: >> So to make COMPILE-FILE work with a stream, one first has to figure out >> what it should return, since there isn't any place for it to put the >> compiled version of the stream contents. I suppose that one option >> might be to have it perform side effects on the lisp environment. In >> that case it would be more like a COMPILE-AND-LOAD-STREAM operation.
> In that case you could trivially implement it as:
> (loop for form = (read stream nil stream) until (eq form stream) > do (compile nil form)) ; only defuns
> or perhaps:
> (loop for form = (read stream nil stream) until (eq form stream) > do (funcall (compile nil (lambda () ,form)))) ; any form
No, a COMPILE-FROM-STREAM operation should act as file compiler. (Directed to any implementator who wants to export such functionality.)
Although I do agree that the above is probably what the OP was looking for.
> > So to make COMPILE-FILE work with a stream, one first has to figure out > > what it should return, since there isn't any place for it to put the > > compiled version of the stream contents. I suppose that one option > > might be to have it perform side effects on the lisp environment. In > > that case it would be more like a COMPILE-AND-LOAD-STREAM operation.
> In that case you could trivially implement it as:
> (loop for form = (read stream nil stream) until (eq form stream) > do (compile nil form)) ; only defuns
> or perhaps:
> (loop for form = (read stream nil stream) until (eq form stream) > do (funcall (compile nil (lambda () ,form)))) ; any form
(in-whining-mode)
If we had standardized environments (which we don't, as implementors deviate or do not implement from the CLTL2 description), we could probably do something along the lines of
(with-compilation-environment (loop for form = (read stream nil stream) until (eq form stream) do (funcall (compile nil (lambda () ,form))))
> > I > > suppose another option would be for it to just produce a stream with the > > contents of the compiled file. But then one would also have to add a > > LOAD-STREAM function, since there isn't any current CL function that > > will load the contents of a stream into the lisp system.
> > So, this all could be doable, but there are still some issues that would > > need to be thrashed out first.
> > -- > > Thomas A. Russ, USC/Information Sciences Institute
Alessio Stalla wrote: > Wouldn't this be fit as one of those mythical additions to the ANSI > standard? In a world where distributed computing is becoming > increasingly important, isn't having the compiler tied to files only
It isn't, as you have observed.
What exactly should a "mythical" compile-stream function return if you feed it a bunch of "lisp text"?
Alessio Stalla wrote: > Wouldn't this be fit as one of those mythical additions to the ANSI > standard? In a world where distributed computing is becoming > increasingly important, isn't having the compiler tied to files only
It isn't tied to files only, as you have observed.
What exactly should a "mythical" compile-stream function return if you feed it a bunch of "lisp text"?
If you want to evaluate that, shove it through read+eval. In some implementations, eval always compiles things before evaluation.
That's my understanding of it. It may well be that this isn't feasible for reasons I don't know. I also now understand (thanks to Rainer) that at least on some implementations a compile-from-stream would be necessarily implemented as my pseudo-code above.
> If you want to evaluate that, shove it through read+eval. In some > implementations, eval always compiles things before evaluation.
I didn't fully express my requirements. I'm using an implementation whose eval does not compile code, I can't use another implementation, and I need my code compiled (and it comes from a stream).
...and I hate temp files :D
Alessio
PS btw, in implementations where eval compiles before evaluation, it probably compiles using the "run-time compiler" and not the "file compiler", which as I have learned differ in some aspects.
Marco Antoniotti <marc...@gmail.com> wrote: +--------------- | If we had standardized environments (which we don't, as implementors | deviate or do not implement from the CLTL2 description), we could | probably do something along the lines of | | (with-compilation-environment | (loop for form = (read stream nil stream) until (eq form stream) | do (funcall (compile nil `(lambda () ,form)))) +---------------
Depending on the implementation, using WITH-COMPILATION-UNIT [instead of the mythical WITH-COMPILATION-ENVIRONMENT] *might* buy you something in the above. Maybe.
-Rob
----- Rob Warnock <r...@rpw3.org> 627 26th Avenue <URL:http://rpw3.org/> San Mateo, CA 94403 (650)572-2607
On Feb 4, 6:10 am, r...@rpw3.org (Rob Warnock) wrote:
> Marco Antoniotti <marc...@gmail.com> wrote: > +--------------- > | If we had standardized environments (which we don't, as implementors > | deviate or do not implement from the CLTL2 description), we could > | probably do something along the lines of > | > | (with-compilation-environment > | (loop for form = (read stream nil stream) until (eq form stream) > | do (funcall (compile nil `(lambda () ,form)))) > +---------------
> Depending on the implementation, using WITH-COMPILATION-UNIT > [instead of the mythical WITH-COMPILATION-ENVIRONMENT] *might* > buy you something in the above. Maybe.
Alessio Stalla <alessiosta...@gmail.com> writes: > On Feb 2, 9:08 pm, "jos...@corporate-world.lisp.de" <jos...@corporate- > world.lisp.de> wrote:
> > Why? Doesn't your OS provide you with some kind of 'tmp' directory?
> Yes, but having to explicitly deal with temp files (on multiple > OSes...) vs having it Just Work Out Of The Box... it's not the same > thing ;)
Well, unfortunately, temp directories aren't addressed by the standard. But part of the reason is that the standard tries to make very minimal assumptions about the file system of operating systems where it runs, which is presumably why there isn't a nice DIRECTORY-P function. =:o
Anyway, the approach I would take would be to use logical pathnames to handle the multiple OS issues. Then you could either directly customize them, or at least put in conditional setup of the logical pathname translations. That will at least insulate the rest of your code form such concerns.
You could just write to "TEMP:output.lisp" and then (load (compile-file "TEMP:output.lisp")) to get it into your lisp system.
The setup of the TEMP logical host is somewhat implementation dependent. On the plus side, all of the six lisp implementations* I just tried on my machine were kind enough to have :UNIX as one of the features, thus making it easy to define the logical host for unix machines:
Thanks for your input, but since I find myself in a special case (I'm working specifically on ABCL so I don't need a standard solution), I used the underlying Java file API, which has a nice createTempFile() method that works on all the platforms supported by the JVM. Certainly uglier than using a pure-Lisp solution, but it spares me all the details of figuring out where the temp dir is in various OS versions...
Alessio
On Feb 5, 1:06 am, t...@sevak.isi.edu (Thomas A. Russ) wrote:
> Alessio Stalla <alessiosta...@gmail.com> writes: > > On Feb 2, 9:08 pm, "jos...@corporate-world.lisp.de" <jos...@corporate- > > world.lisp.de> wrote:
> > > Why? Doesn't your OS provide you with some kind of 'tmp' directory?
> > Yes, but having to explicitly deal with temp files (on multiple > > OSes...) vs having it Just Work Out Of The Box... it's not the same > > thing ;)
> Well, unfortunately, temp directories aren't addressed by the standard. > But part of the reason is that the standard tries to make very minimal > assumptions about the file system of operating systems where it runs, > which is presumably why there isn't a nice DIRECTORY-P function. =:o
> Anyway, the approach I would take would be to use logical pathnames to > handle the multiple OS issues. Then you could either directly customize > them, or at least put in conditional setup of the logical pathname > translations. That will at least insulate the rest of your code form > such concerns.
> You could just write to "TEMP:output.lisp" and then > (load (compile-file "TEMP:output.lisp")) > to get it into your lisp system.
> The setup of the TEMP logical host is somewhat implementation dependent. > On the plus side, all of the six lisp implementations* I just tried on > my machine were kind enough to have :UNIX as one of the features, thus > making it easy to define the logical host for unix machines:
|
