Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

COMPILE-FILE without a file...

6 views
Skip to first unread message

Alessio Stalla

unread,
Feb 2, 2009, 2:00:15 PM2/2/09
to
...or, compiling arbitrary text streams.

As far as I know, the only two standard interfaces with the Lisp
compiler are the functions COMPILE and COMPILE-FILE. The first
compiles a single function, the other compiles a file (dealing
properly with top-level forms etc. etc.).

Suppose I have code in text form, coming from an arbitrary stream
(e.g. the network, or a compressed archive, or...): is (1)

(compile nil `(lambda () ,@(read stream)))

equivalent to (2)

(progn
(dump-to-file stream "file.lisp")
(compile-file "file.lisp"))

? Probably not... (rhetorical question) but, exactly what kind of
disaster can I expect if I use (2) instead of (1)? I'd like to avoid
dealing with temporary files if I can...

Thanks,
Alessio Stalla

Alessio Stalla

unread,
Feb 2, 2009, 2:04:30 PM2/2/09
to
On Feb 2, 8:00 pm, Alessio Stalla <alessiosta...@gmail.com> wrote:
> exactly what kind of disaster can I expect if I use (2) instead of (1)?

Sorry, I meant use (1) instead of (2)...

Tobias C. Rittweiler

unread,
Feb 2, 2009, 2:18:16 PM2/2/09
to
Alessio Stalla <alessi...@gmail.com> writes:

> Suppose I have code in text form, coming from an arbitrary stream
> (e.g. the network, or a compressed archive, or...): is (1)
>
> (compile nil `(lambda () ,@(read stream)))
>
> equivalent to (2)
>
> (progn
> (dump-to-file stream "file.lisp")
> (compile-file "file.lisp"))

No, it's not. The first form executes the /run time compiler/, the
second one the /file compiler/. They differ in how they deal with
EVAL-WHEN, and how they coalesce constants.


> ? Probably not... (rhetorical question) but, exactly what kind of
> disaster can I expect if I use (2) instead of (1)? I'd like to avoid
> dealing with temporary files if I can...

Some implementations offer a COMPILE-FROM-STREAM. Look at how
SWANK-COMPILE-STRING is implemented in the different SWANK
backends. (The swank-foo.lisp files in the source checkout of Slime.)

-T.

Tobias C. Rittweiler

unread,
Feb 2, 2009, 2:18:26 PM2/2/09
to
Alessio Stalla <alessi...@gmail.com> writes:

> Suppose I have code in text form, coming from an arbitrary stream
> (e.g. the network, or a compressed archive, or...): is (1)
>
> (compile nil `(lambda () ,@(read stream)))
>
> equivalent to (2)
>
> (progn
> (dump-to-file stream "file.lisp")
> (compile-file "file.lisp"))

No, it's not. The first form executes the /run time compiler/, the


second one the /file compiler/. They differ in how they deal with
EVAL-WHEN, and how they coalesce constants.

> ? Probably not... (rhetorical question) but, exactly what kind of
> disaster can I expect if I use (2) instead of (1)? I'd like to avoid
> dealing with temporary files if I can...

Some implementations offer a COMPILE-FROM-STREAM. Look at how

Alessio Stalla

unread,
Feb 2, 2009, 2:34:13 PM2/2/09
to
On Feb 2, 8:18 pm, "Tobias C. Rittweiler" <t...@freebits.de.invalid>
wrote:

> Alessio Stalla <alessiosta...@gmail.com> writes:
> Some implementations offer a COMPILE-FROM-STREAM. Look at how
> SWANK-COMPILE-STRING is implemented in the different SWANK
> backends. (The swank-foo.lisp files in the source checkout of Slime.)

Thanks. I looked up the definition for ABCL, which is the
implementation in which I'm interested, and... basically it calls
(compile nil (read ...))

So I guess I'm out of luck. Temporary files will be, for now...

Wouldn't this be fit as one of those mythical additions to the ANSI
standard? In a world where distributed computing is becoming
increasingly important, isn't having the compiler tied to files only
(in a language like Lisp, which has dynamic code compilation as one of
its strong points) too much of a limitation? Especially given that
it's probably trivial to adapt the compiler to work on streams other
than files (at least, it seems to me it's trivial, but I know very
little of how Lisp compilers work...).

Alessio

jos...@corporate-world.lisp.de

unread,
Feb 2, 2009, 3:08:28 PM2/2/09
to
On 2 Feb., 20:34, Alessio Stalla <alessiosta...@gmail.com> wrote:
> On Feb 2, 8:18 pm, "Tobias C. Rittweiler" <t...@freebits.de.invalid>
> wrote:
>
> > Alessio Stalla <alessiosta...@gmail.com> writes:
> > Some implementations offer a COMPILE-FROM-STREAM. Look at how
> > SWANK-COMPILE-STRING is implemented in the different SWANK
> > backends. (The swank-foo.lisp files in the source checkout of Slime.)
>
> Thanks. I looked up the definition for ABCL, which is the
> implementation in which I'm interested, and... basically it calls
> (compile nil (read ...))
>
> So I guess I'm out of luck. Temporary files will be, for now...
>
> Wouldn't this be fit as one of those mythical additions to the ANSI
> standard? In a world where distributed computing is becoming
> increasingly important, isn't having the compiler tied to files only
> (in a language like Lisp, which has dynamic code compilation as one of
> its strong points) too much of a limitation?

Why? Doesn't your OS provide you with some kind of 'tmp' directory?

> Especially given that
> it's probably trivial to adapt the compiler to work on streams other
> than files (at least, it seems to me it's trivial, but I know very
> little of how Lisp compilers work...).
>
> Alessio

The idea of a file compiler allows also a multi-pass compiler, where
one or passes of a Lisp compiler might generate some output
and one or more passes of an external compiler (C, Assembler)
might generate further output. Also note that the file compiler
COMPILEs the file, but does not necessarily LOAD the resulting code.
So, if a file compiler compiles (defun foo ...), after
compilation there is possibly no trace of the function FOO
not even then symbol in the runtime, where the compiler
runs. So, a stream compiler is a different model. For a stream
compiler that uses a C compiler to generate the native
code, it still means writing and reading of files (the C code),
even though you think it just compiles a stream.

One could specify a stream compiler in a standard, but
it is some work to write down what it should do and how.
It could work similar to some REPLs that read code expressions,
compile those and execute those. Note that not all REPLs
(even in compiled Lisps) compile the expressions in the
REPL.

Paul Wallich

unread,
Feb 2, 2009, 3:15:39 PM2/2/09
to

It's trivial to do, not necessarily trivial to decide how to do it. When
you're compiling a file, you have access to the entire file while you're
doing it, and can make decisions based on (effectively) global knowledge
about what you're compiling. If you're compiling just one s-expression,
you know everything about that, and you know what you don't know about
the rest of the universe. If you're compiling from a stream, what do you
know? Are you compiling only the contents of streams that have been
flushed and closed? everything that comes in, piece by piece? everything
that comes in, but remembering (for some length of time) the ongoing
compilation environment for literals etc? Depending on what kind of
stream you're compiling from and why, the desired behavior will probably
be obvious, but that doesn't necessarily make for good standardization.

paul

Alessio Stalla

unread,
Feb 2, 2009, 3:24:20 PM2/2/09
to
On Feb 2, 9:08 pm, "jos...@corporate-world.lisp.de" <jos...@corporate-
world.lisp.de> wrote:

> Why? Doesn't your OS provide you with some kind of 'tmp' directory?

Yes, but having to explicitly deal with temp files (on multiple
OSes...) vs having it Just Work Out Of The Box... it's not the same
thing ;)

> The idea of a file compiler allows also a multi-pass compiler, where
> one or passes of a Lisp compiler might generate some output
> and one or more passes of an external compiler (C, Assembler)
> might generate further output. Also note that the file compiler
> COMPILEs the file, but does not necessarily LOAD the resulting code.
> So, if a file compiler compiles (defun foo ...), after
> compilation there is possibly no trace of the function FOO
> not even then symbol in the runtime, where the compiler
> runs. So, a stream compiler is a different model. For a stream
> compiler that uses a C compiler to generate the native
> code, it still means writing and reading of files (the C code),
> even though you think it just compiles a stream.
>
> One could specify a stream compiler in a standard, but
> it is some work to write down what it should do and how.
> It could work similar to some REPLs that read code expressions,
> compile those and execute those. Note that not all REPLs
> (even in compiled Lisps) compile the expressions in the
> REPL.

Yes, I see. It makes much sense. Thanks for the detailed explanation.

I still think that having a portable compiler-from-stream, even if
somehow limited, would be a good thing, but I understand it would be
difficult to specify what it should do and how, and in some
implementations it would be probably difficult or even impossible to
implement it, too. That is, without resorting to temporary files.
So, *sigh*, let's do this temp file thingie... did I mention I hate
temp files? ;)

Thanks,
Alessio

jos...@corporate-world.lisp.de

unread,
Feb 2, 2009, 3:47:06 PM2/2/09
to

For example: when you use COMPILE in GCL to generate native code,
it also needs to call the external C compiler. Which means:
tmp files.

Thomas A. Russ

unread,
Feb 2, 2009, 5:25:45 PM2/2/09
to
Alessio Stalla <alessi...@gmail.com> writes:
> Thanks. I looked up the definition for ABCL, which is the
> implementation in which I'm interested, and... basically it calls
> (compile nil (read ...))
>
> So I guess I'm out of luck. Temporary files will be, for now...
>
> Wouldn't this be fit as one of those mythical additions to the ANSI
> standard? In a world where distributed computing is becoming
> increasingly important, isn't having the compiler tied to files only
> (in a language like Lisp, which has dynamic code compilation as one of
> its strong points) too much of a limitation? Especially given that
> it's probably trivial to adapt the compiler to work on streams other
> than files (at least, it seems to me it's trivial, but I know very
> little of how Lisp compilers work...).

Well, I suppose it would be somewhat interesting. The real question is
what the return value of such a COMPILE-STREAM function would be.
Perhaps it needs to be some type of STREAM that is compatible with the
LOAD function.

The fundamental difference between COMPILE and COMPILE-FILE is that the
first one produces a function, which is a first-class lisp object that
can later be FUNCALLed, APPLYed, etc. This can be done directly in the
lisp system.

The latter has the side effect of producing a compiled file and returns
the file name of that file. To actually get the definitions into your
lisp, you still have to load the compiled file. The difference is that
if you were to do:

(compile 'factorial
(lambda (x) (if (<= x 1) 1 (* x (factorial (1- x))))))

then you could immediately invoke

(factorial 3)

and get an answer. If instead, you were to call

(compile-file factorial.lisp)

where the file factorial.lisp contained

(defun factorial (x)
(if (<= x 1) 1 (* x (factorial (1- x)))))

then you could NOT call

(factorial 3)

without an error unless you first LOADed the compiled version of the
file.

So to make COMPILE-FILE work with a stream, one first has to figure out
what it should return, since there isn't any place for it to put the
compiled version of the stream contents. I suppose that one option
might be to have it perform side effects on the lisp environment. In
that case it would be more like a COMPILE-AND-LOAD-STREAM operation. I
suppose another option would be for it to just produce a stream with the
contents of the compiled file. But then one would also have to add a
LOAD-STREAM function, since there isn't any current CL function that
will load the contents of a stream into the lisp system.

So, this all could be doable, but there are still some issues that would
need to be thrashed out first.

--
Thomas A. Russ, USC/Information Sciences Institute

Waldek Hebisch

unread,
Feb 2, 2009, 10:20:36 PM2/2/09
to
Alessio Stalla <alessi...@gmail.com> wrote:
> On Feb 2, 8:18 pm, "Tobias C. Rittweiler" <t...@freebits.de.invalid>
> wrote:
> > Alessio Stalla <alessiosta...@gmail.com> writes:
> > Some implementations offer a COMPILE-FROM-STREAM. Look at how
> > SWANK-COMPILE-STRING is implemented in the different SWANK
> > backends. (The swank-foo.lisp files in the source checkout of Slime.)
>
> Thanks. I looked up the definition for ABCL, which is the
> implementation in which I'm interested, and... basically it calls
> (compile nil (read ...))
>
> So I guess I'm out of luck. Temporary files will be, for now...
>

I wonder about following: find any existing text file F. Define
its first character as macro character and associate with it
function that reads your stream (as a single form wrapped in progn)
and than changes file position to the end of F. Now compile F -- I am
not sure if this will work as expected, but in your place I would
investigate such way.

--
Waldek Hebisch
heb...@math.uni.wroc.pl

Alessio Stalla

unread,
Feb 3, 2009, 3:35:49 AM2/3/09
to
On Feb 3, 4:20 am, Waldek Hebisch <hebi...@math.uni.wroc.pl> wrote:

> Alessio Stalla <alessiosta...@gmail.com> wrote:
> > On Feb 2, 8:18 pm, "Tobias C. Rittweiler" <t...@freebits.de.invalid>
> I wonder about following: find any existing text file F.  Define
> its first character as macro character and associate with it
> function that reads your stream (as a single form wrapped in progn)
> and than changes file position to the end of F.  Now compile F -- I am
> not sure if this will work as expected, but in your place I would
> investigate such way.

!!! This solution is so diabolically ingenious that I've got to try
it, even if it feels to me "dirtier" than the temp-file based
approach ;)
I'll report back with my findings...

Thanks for the inspiration
Alessio

Pascal J. Bourguignon

unread,
Feb 3, 2009, 8:37:15 AM2/3/09
to
t...@sevak.isi.edu (Thomas A. Russ) writes:
> So to make COMPILE-FILE work with a stream, one first has to figure out
> what it should return, since there isn't any place for it to put the
> compiled version of the stream contents. I suppose that one option
> might be to have it perform side effects on the lisp environment. In
> that case it would be more like a COMPILE-AND-LOAD-STREAM operation.

In that case you could trivially implement it as:

(loop for form = (read stream nil stream) until (eq form stream)
do (compile nil form)) ; only defuns

or perhaps:

(loop for form = (read stream nil stream) until (eq form stream)
do (funcall (compile nil (lambda () ,form)))) ; any form


> I
> suppose another option would be for it to just produce a stream with the
> contents of the compiled file. But then one would also have to add a
> LOAD-STREAM function, since there isn't any current CL function that
> will load the contents of a stream into the lisp system.
>
> So, this all could be doable, but there are still some issues that would
> need to be thrashed out first.
>
> --
> Thomas A. Russ, USC/Information Sciences Institute

--
__Pascal Bourguignon__

Tobias C. Rittweiler

unread,
Feb 3, 2009, 8:46:40 AM2/3/09
to
p...@informatimago.com (Pascal J. Bourguignon) writes:

> t...@sevak.isi.edu (Thomas A. Russ) writes:
>> So to make COMPILE-FILE work with a stream, one first has to figure out
>> what it should return, since there isn't any place for it to put the
>> compiled version of the stream contents. I suppose that one option
>> might be to have it perform side effects on the lisp environment. In
>> that case it would be more like a COMPILE-AND-LOAD-STREAM operation.
>
> In that case you could trivially implement it as:
>
> (loop for form = (read stream nil stream) until (eq form stream)
> do (compile nil form)) ; only defuns
>
> or perhaps:
>
> (loop for form = (read stream nil stream) until (eq form stream)
> do (funcall (compile nil (lambda () ,form)))) ; any form

No, a COMPILE-FROM-STREAM operation should act as file
compiler. (Directed to any implementator who wants to export such
functionality.)

Although I do agree that the above is probably what the OP was looking
for.

-T.

Marco Antoniotti

unread,
Feb 3, 2009, 9:20:15 AM2/3/09
to
On Feb 3, 2:37 pm, p...@informatimago.com (Pascal J. Bourguignon)
wrote:

> t...@sevak.isi.edu (Thomas A. Russ) writes:
>
> > So to make COMPILE-FILE work with a stream, one first has to figure out
> > what it should return, since there isn't any place for it to put the
> > compiled version of the stream contents.  I suppose that one option
> > might be to have it perform side effects on the lisp environment.  In
> > that case it would be more like a COMPILE-AND-LOAD-STREAM operation.  
>
> In that case you could trivially implement it as:
>
> (loop for form = (read stream nil stream) until (eq form stream)
>       do (compile nil form)) ; only defuns
>
> or perhaps:
>
> (loop for form = (read stream nil stream) until (eq form stream)
>       do (funcall (compile nil (lambda () ,form)))) ; any form


(in-whining-mode)

If we had standardized environments (which we don't, as implementors
deviate or do not implement from the CLTL2 description), we could
probably do something along the lines of

(with-compilation-environment


(loop for form = (read stream nil stream) until (eq form stream)
do (funcall (compile nil (lambda () ,form))))

... and now I run to hide my hand....

Cheers
--
Marco
www.european-lisp-symposium.org

Matthias Buelow

unread,
Feb 3, 2009, 11:13:08 AM2/3/09
to
Alessio Stalla wrote:

> Wouldn't this be fit as one of those mythical additions to the ANSI
> standard? In a world where distributed computing is becoming
> increasingly important, isn't having the compiler tied to files only

It isn't, as you have observed.

What exactly should a "mythical" compile-stream function return if you
feed it a bunch of "lisp text"?

Matthias Buelow

unread,
Feb 3, 2009, 11:15:23 AM2/3/09
to
Alessio Stalla wrote:

> Wouldn't this be fit as one of those mythical additions to the ANSI
> standard? In a world where distributed computing is becoming
> increasingly important, isn't having the compiler tied to files only

It isn't tied to files only, as you have observed.

What exactly should a "mythical" compile-stream function return if you
feed it a bunch of "lisp text"?

If you want to evaluate that, shove it through read+eval. In some
implementations, eval always compiles things before evaluation.

Alessio Stalla

unread,
Feb 3, 2009, 2:43:43 PM2/3/09
to
On Feb 3, 5:15 pm, Matthias Buelow <m...@incubus.de> wrote:
> It isn't tied to files only, as you have observed.

Well, the "file compiler" can only compile files. Of course the files'
contents can come from anywhere.

> What exactly should a "mythical" compile-stream function return if you
> feed it a bunch of "lisp text"?

Uhm... a compiled function? Which when invoked does the same things as
if you had done

(load (compile-file (make-temp-file-reading-from stream)))

That's my understanding of it. It may well be that this isn't feasible
for reasons I don't know.
I also now understand (thanks to Rainer) that at least on some
implementations a compile-from-stream would be necessarily implemented
as my pseudo-code above.

> If you want to evaluate that, shove it through read+eval. In some
> implementations, eval always compiles things before evaluation.

I didn't fully express my requirements. I'm using an implementation
whose eval does not compile code, I can't use another implementation,
and I need my code compiled (and it comes from a stream).

...and I hate temp files :D

Alessio

PS btw, in implementations where eval compiles before evaluation, it
probably compiles using the "run-time compiler" and not the "file
compiler", which as I have learned differ in some aspects.

Tobias C. Rittweiler

unread,
Feb 3, 2009, 4:39:14 PM2/3/09
to
Matthias Buelow <m...@incubus.de> writes:

> What exactly should a "mythical" compile-stream function return if you
> feed it a bunch of "lisp text"?

A fasl stream, which can be passed to LOAD to make the contained
toplevel forms take effect.

-T.

Rob Warnock

unread,
Feb 4, 2009, 12:10:43 AM2/4/09
to
Marco Antoniotti <mar...@gmail.com> wrote:
+---------------

| If we had standardized environments (which we don't, as implementors
| deviate or do not implement from the CLTL2 description), we could
| probably do something along the lines of
|
| (with-compilation-environment
| (loop for form = (read stream nil stream) until (eq form stream)
| do (funcall (compile nil `(lambda () ,form))))
+---------------

Depending on the implementation, using WITH-COMPILATION-UNIT
[instead of the mythical WITH-COMPILATION-ENVIRONMENT] *might*
buy you something in the above. Maybe.


-Rob

-----
Rob Warnock <rp...@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607

Marco Antoniotti

unread,
Feb 4, 2009, 7:25:08 AM2/4/09
to
On Feb 4, 6:10 am, r...@rpw3.org (Rob Warnock) wrote:

> Marco Antoniotti  <marc...@gmail.com> wrote:
> +---------------
> | If we had standardized environments (which we don't, as implementors
> | deviate or do not implement from the CLTL2 description), we could
> | probably do something along the lines of
> |
> | (with-compilation-environment
> |    (loop for form = (read stream nil stream) until (eq form stream)
> |          do (funcall (compile nil `(lambda () ,form))))
> +---------------
>
> Depending on the implementation, using WITH-COMPILATION-UNIT
> [instead of the mythical WITH-COMPILATION-ENVIRONMENT] *might*
> buy you something in the above. Maybe.

Yes. But I was deliberatly whining! :)

Cheers
--
Marco

>
> -Rob
>
> -----
> Rob Warnock                     <r...@rpw3.org>

Thomas A. Russ

unread,
Feb 4, 2009, 7:06:00 PM2/4/09
to
Alessio Stalla <alessi...@gmail.com> writes:

> On Feb 2, 9:08  pm, "jos...@corporate-world.lisp.de" <jos...@corporate-
> world.lisp.de> wrote:
>
> > Why? Doesn't your OS provide you with some kind of 'tmp' directory?
>
> Yes, but having to explicitly deal with temp files (on multiple
> OSes...) vs having it Just Work Out Of The Box... it's not the same
> thing ;)

Well, unfortunately, temp directories aren't addressed by the standard.
But part of the reason is that the standard tries to make very minimal
assumptions about the file system of operating systems where it runs,
which is presumably why there isn't a nice DIRECTORY-P function. =:o

Anyway, the approach I would take would be to use logical pathnames to
handle the multiple OS issues. Then you could either directly customize
them, or at least put in conditional setup of the logical pathname
translations. That will at least insulate the rest of your code form
such concerns.

You could just write to "TEMP:output.lisp" and then
(load (compile-file "TEMP:output.lisp"))
to get it into your lisp system.

The setup of the TEMP logical host is somewhat implementation dependent.
On the plus side, all of the six lisp implementations* I just tried on
my machine were kind enough to have :UNIX as one of the features, thus
making it easy to define the logical host for unix machines:

(setf (logical-pathname-translations "TEMP")
#+:UNIX '(("TEMP:*.*" "/tmp/*.*")))

Other OSes are left as an exercise for the reader.


* Allegro, CCL, CLISP, CMUCL, Lispworks, SBCL

Alessio Stalla

unread,
Feb 5, 2009, 8:46:22 AM2/5/09
to
Thanks for your input, but since I find myself in a special case (I'm
working specifically on ABCL so I don't need a standard solution), I
used the underlying Java file API, which has a nice createTempFile()
method that works on all the platforms supported by the JVM. Certainly
uglier than using a pure-Lisp solution, but it spares me all the
details of figuring out where the temp dir is in various OS
versions...

Alessio

On Feb 5, 1:06 am, t...@sevak.isi.edu (Thomas A. Russ) wrote:

0 new messages