How do I serialize a function?
Kim Minh Kaplan
far:
* (with-input-from-string
(stream "f<?kmcl (dotimes (i 2) ?>o<?kmcl ) ?>bar")
(parse-template-stream stream))
#<FUNCTION {11D53445}>
NIL
NIL
* (funcall *)
foobar
"bar"
Now, to save some computation I would like to save this function for
reuse. I tried using plain WRITE or PRIN1 alas this does not write
something that can be READ back:
* (with-output-to-string (s) (prin1 (parse-template-string "foo")
s))
"#<FUNCTION {122AD1C5}>"
Obviously every Common Lisp system can do this internally (it is
needed for COMPILE-FILE and LOAD). But is there a portable way to
access this functionnality from user land? If not what are the non
portable ways? I am mostly interested in SBCL and CLisp but other
informations are welcome.
Kim Minh.
--
http://www.kim-minh.com/
Pascal Bourguignon
If you've written parse-template-stream, then you should know!
parse-template-stream should be building a s-expr, perhaps something
like:
(progn
(princ "f")
(dotimes (i 2)
(princ "o"))
(princ "bar"))
for the above input.
To obtain a function, you should have put this s-expr inside a lambda
form and called compile:
(compile nil `(lambda () ,s-expr))
Well, look at it more closely! this s-expr and this `(lambda () ,sexpr)
are plain symbolic lists. You can write them and read them back with
no problem.
Assuming parse-template-stream is of the form:
(defun parse-template-stream (stream)
(compile nil (generate-function-from-kmcl stream)))
LISP> (defun parse-template-stream (stream)
(compile nil (generate-function-from-kmcl stream)))
PARSE-TEMPLATE-STREAM
LISP> (with-input-from-string
(stream "f<?kmcl (dotimes (i 2) ?>o<?kmcl ) ?>bar")
(parse-template-stream stream))
#<COMPILED-FUNCTION NIL> ;
NIL ;
NIL
LISP> (funcall *)
foobar
"bar"
LISP> (with-input-from-string
(stream "f<?kmcl (dotimes (i 2) ?>o<?kmcl ) ?>bar")
(generate-function-from-kmcl stream))
(LAMBDA NIL (PROGN (PRINC "f") (DOTIMES (I 2) (PRINC "o")) (PRINC "bar")))
LISP> (with-open-file (out "saved-function.lisp"
:direction :output
:if-does-not-exist :create
:if-exists :supersede)
(with-input-from-string
(stream "f<?kmcl (dotimes (i 2) ?>o<?kmcl ) ?>bar")
(print (generate-function-from-kmcl stream) out)))
(LAMBDA NIL (PROGN (PRINC "f") (DOTIMES (I 2) (PRINC "o")) (PRINC "bar")))
LISP> (cat "saved-function.lisp")
(LAMBDA NIL (PROGN (PRINC "f") (DOTIMES (I 2) (PRINC "o")) (PRINC "bar")))
LISP> (compile nil (with-open-file (in "saved-function.lisp") (read in)))
#<COMPILED-FUNCTION NIL> ;
NIL ;
NIL
LISP> (funcall *)
foobar
"bar"
LISP>
--
__Pascal Bourguignon__ http://www.informatimago.com/
CAUTION: The mass of this product contains the energy equivalent of
85 million tons of TNT per net ounce of weight.
Kim Minh Kaplan
> If you've written parse-template-stream, then you should know!
>
> parse-template-stream should be building a s-expr, perhaps something
> like:
>
> (progn
> (princ "f")
> (dotimes (i 2)
> (princ "o"))
> (princ "bar"))
>
> for the above input.
>
> To obtain a function, you should have put this s-expr inside a lambda
> form and called compile:
>
> (compile nil `(lambda () ,s-expr))
Yes, that's how it works.
> Well, look at it more closely! this s-expr and this `(lambda () ,sexpr)
> are plain symbolic lists. You can write them and read them back with
> no problem.
But I would like to write and read the *compiled* form.
> LISP> (compile nil (with-open-file (in "saved-function.lisp") (read in)))
This step would omit the compilation stage... Something like:
(with-open-file (in "saved-function.fasl" :element-type
'unsigned-byte) (read in))
For the moment I'm exploring saving it as a defun, then using
compile-file to build the fasl & load to read it. But it smells like a
hugly kludge.
Kim Minh
--
http://www.kim-minh.com/
Pascal Bourguignon
> Pascal Bourguignon wrote:
>
>> If you've written parse-template-stream, then you should know!
>>
>> parse-template-stream should be building a s-expr, perhaps something
>> like:
>>
>> (progn
>> (princ "f")
>> (dotimes (i 2)
>> (princ "o"))
>> (princ "bar"))
>>
>> for the above input.
>>
>> To obtain a function, you should have put this s-expr inside a lambda
>> form and called compile:
>>
>> (compile nil `(lambda () ,s-expr))
>
> Yes, that's how it works.
>
>> Well, look at it more closely! this s-expr and this `(lambda () ,sexpr)
>> are plain symbolic lists. You can write them and read them back with
>> no problem.
>
> But I would like to write and read the *compiled* form.
The only standard way to write a compiled form, is to call
COMPILE-FILE, and the only standard way to read a compiled form, is to
call LOAD.
LISP> (let ((some-name 'saved-function))
(with-open-file (out "saved-function.lisp"
:direction :output
:if-does-not-exist :create
:if-exists :supersede)
(with-input-from-string
(stream "f<?kmcl (dotimes (i 2) ?>o<?kmcl ) ?>bar")
(print `(defun ,some-name
,@(cdr (generate-function-from-kmcl stream))) out))))
(DEFUN SAVED-FUNCTION NIL
(PROGN (PRINC "f") (DOTIMES (I 2) (PRINC "o")) (PRINC "bar")))
LISP> (cat "saved-function.lisp")
(DEFUN SAVED-FUNCTION NIL
(PROGN (PRINC "f") (DOTIMES (I 2) (PRINC "o")) (PRINC "bar")))
LISP> (compile-file "saved-function.lisp")
;; Compiling file /local/users/pjb/saved-function.lisp ...
;; Wrote file /local/users/pjb/saved-function.fas
0 errors, 0 warnings
#P"/local/users/pjb/saved-function.fas" ;
NIL ;
NIL
LISP> (load"saved-function")
;; Loading file /local/users/pjb/saved-function.fas ...
;; Loaded file /local/users/pjb/saved-function.fas
T
LISP> (saved-function)
foobar
"bar"
LISP>
>> LISP> (compile nil (with-open-file (in "saved-function.lisp") (read in)))
>
> This step would omit the compilation stage... Something like:
>
> (with-open-file (in "saved-function.fasl" :element-type
> 'unsigned-byte) (read in))
>
> For the moment I'm exploring saving it as a defun, then using
> compile-file to build the fasl & load to read it. But it smells like a
> hugly kludge.
No, this is the way you must do it (by the standard).
Note that in any case, .fasl files are implementation and version
dependant. Saving the source form has the advantage that you can read
back the functions in different implementations and in different
versions of the same implementation.
--
__Pascal Bourguignon__ http://www.informatimago.com/
Grace personified,
I leap into the window.
I meant to do that.
Pascal Costanza
> Pascal Bourguignon wrote:
>
>> If you've written parse-template-stream, then you should know!
>>
>> parse-template-stream should be building a s-expr, perhaps something
>> like:
>>
>> (progn
>> (princ "f")
>> (dotimes (i 2)
>> (princ "o"))
>> (princ "bar"))
>>
>> for the above input.
>>
>> To obtain a function, you should have put this s-expr inside a lambda
>> form and called compile:
>>
>> (compile nil `(lambda () ,s-expr))
>
> Yes, that's how it works.
>
>> Well, look at it more closely! this s-expr and this `(lambda () ,sexpr)
>> are plain symbolic lists. You can write them and read them back with
>> no problem.
>
> But I would like to write and read the *compiled* form.
That's not possible, not even for the compiler. What the compiler does
is to store function _definitions_ in a file, not functions.
Functions are runtime objects that potentially close over runtime
bindings. It's not possible to serialize such objects without losing the
references to such bindings.
Consider:
(defvar *fun*
(let ((x 42))
(list (lambda () x)
(lambda (y) (setq x y)))))
Now consider the following program fragment:
(progn
(store (first *fun*) some-file)
(funcall (second *fun*) 4711)
(let ((fun (retrieve some-file)))
(funcall fun)))
The idea is that 'store serializes a function into some file, and
'retrieve deserializes a function from some file. So in the above
fragment, (second *fun*) and the local 'fun should be the same.
The question boils down to how you can implement 'store and 'retrieve
such that the above fragment (correctly) returns 4711 (i.e., whatever
value the local x in *fun* has).
Pascal
--
3rd European Lisp Workshop
July 3 - Nantes, France - co-located with ECOOP 2006
http://lisp-ecoop06.bknr.net/
Kim Minh Kaplan
Pascal Costanza wrote:
> That's not possible, not even for the compiler. What the compiler does
> is to store function _definitions_ in a file, not functions.
>
> Functions are runtime objects that potentially close over runtime
> bindings. It's not possible to serialize such objects without losing the
> references to such bindings.
Ok, I had never looked at it this way. Then I guess i want to store
the definition of the function in a file. Anyway, I finally did it by
first writing a .lisp file and then calling compile-file on it. It has
the (minor) drawback that the function cannot be anonymous.
> Consider:
>
> (defvar *fun*
> (let ((x 42))
> (list (lambda () x)
> (lambda (y) (setq x y)))))
>
> Now consider the following program fragment:
>
> (progn
> (store (first *fun*) some-file)
> (funcall (second *fun*) 4711)
> (let ((fun (retrieve some-file)))
> (funcall fun)))
>
> The idea is that 'store serializes a function into some file, and
> 'retrieve deserializes a function from some file. So in the above
> fragment, (second *fun*) and the local 'fun should be the same.
Thanks for the clear example.
--
Kim Minh.
http://www.kim-minh.com/
Pascal Bourguignon
> Pascal Costanza wrote:
>
>> That's not possible, not even for the compiler. What the compiler does
>> is to store function _definitions_ in a file, not functions.
>>
>> Functions are runtime objects that potentially close over runtime
>> bindings. It's not possible to serialize such objects without losing the
>> references to such bindings.
>
> Ok, I had never looked at it this way. Then I guess i want to store
> the definition of the function in a file. Anyway, I finally did it by
> first writing a .lisp file and then calling compile-file on it. It has
> the (minor) drawback that the function cannot be anonymous.
You can still have anonymous functions:
(defvar *anon*)
(defun save-anonymous-function (fname args body)
(let ((fname (make-pathname :type "LISP" :case :common :defaults fname)))
(with-open-file (src fname :direction :output
:if-does-not-exist :create :if-exists :supersede)
(print `(defparameter *anon* (lambda ,args ,body)) src))
(compile-file fname)))
(defun load-anonymous-function (fname)
(let ((*load-verbose* nil)
(*anon* nil)) ; to avoid modifying the global one.
(load fname)
*anon*))
(let ((f1 (save-anonymous-function "f1" '() '(progn (print :f1) 1)))
(f2 (save-anonymous-function "f2" '() '(progn (print :f2) 2))))
(values
(funcall (load-anonymous-function f1))
(funcall (load-anonymous-function f2))
(funcall (load-anonymous-function f2))
(funcall (load-anonymous-function f1))))
;; Compiling file /local/users/pjb/f1.lisp ...
;; Wrote file /local/users/pjb/f1.fas
0 errors, 0 warnings
;; Compiling file /local/users/pjb/f2.lisp ...
;; Wrote file /local/users/pjb/f2.fas
0 errors, 0 warnings
:F1
:F2
:F2
:F1
1 ;
2 ;
2 ;
1
David Lichteblau
>> But I would like to write and read the *compiled* form.
> That's not possible, not even for the compiler. What the compiler does
> is to store function _definitions_ in a file, not functions.
Not portably, yes.
For SBCL, I have written sb-heapdump to be able to save all kinds
objects to files, including functions.
(I believe Allegro also offers an API to write function objects into
fasls or fasl-like files, but I have not tried that.)
> Functions are runtime objects that potentially close over runtime
> bindings. It's not possible to serialize such objects without losing the
> references to such bindings.
True, in this case you need to save both closures into one file
together.
Transcript of your example with sb-heapdump:
CL-USER(4): (defvar *fun*
(let ((x 42))
(list (lambda () x)
(lambda (y) (setq x y)))))
*FUN*
CL-USER(5): (sb-heapdump:dump-object *fun* "/tmp/fun.heap")
8192 bytes written
"/tmp/fun.heap"
CL-USER(6): (defvar *bar* (sb-heapdump:load-dumpfile "/tmp/fun.heap"))
; loading /tmp/fun.heap[0] mmap 0.0s fixup 0.0s done
*BAR*
CL-USER(7): (funcall (second *fun*) 4711)
4711
CL-USER(8): (funcall (car *fun*))
4711
CL-USER(9): (funcall (car *bar*))
42
CL-USER(10): (funcall (second *bar*) 123)
123
CL-USER(11): (funcall (first *bar*))
123
CL-USER(12): *bar*
(#<CLOSURE (LAMBDA #) {8000001D}> #<CLOSURE (LAMBDA #) {800002CD}>)
CL-USER(13): *fun*
(#<CLOSURE (LAMBDA #) {5231697D}> #<CLOSURE (LAMBDA #) {5231698D}>)
Kalle Olavi Niemitalo
> If not what are the non portable ways? I am mostly interested
> in SBCL and CLisp but other informations are welcome.
For CLISP, see CUSTOM:*PRINT-CLOSURE*.
Timofei Shatrov
tried to confuse everyone with this message:
Hmm... why didn't I hear of that before? Works with both interpreted and
compiled closures. Great stuff!
--
|Don't believe this - you're not worthless ,gr---------.ru
|It's us against millions and we can't take them all... | ue il |
|But we can take them on! | @ma |
| (A Wilhelm Scream - The Rip) |______________|
Kim Minh Kaplan
Pascal Bourguignon wrote:
> You can still have anonymous functions:
>
> (defvar *anon*)
>
> (defun save-anonymous-function (fname args body)
> (let ((fname (make-pathname :type "LISP" :case :common :defaults fname)))
> (with-open-file (src fname :direction :output
> :if-does-not-exist :create :if-exists :supersede)
> (print `(defparameter *anon* (lambda ,args ,body)) src))
> (compile-file fname)))
>
> (defun load-anonymous-function (fname)
> (let ((*load-verbose* nil)
> (*anon* nil)) ; to avoid modifying the global one.
> (load fname)
> *anon*))
This works great, thank you. I tried a similar way but using defun
instead of defvar. This would clobber the original value... Your
trick with defparameter and then locally biding it before calling load
is clever. I'll use that (and look at the other CLisp and SBCL
dependant solutions if they have some advantages).
Kim Minh.