Large data structures work with ACL but nor CLISP, CMUCL?
Bruce Lambert
CLISP and CMUCL to crash when I give arguments greater than about 15000. On
ACL, using the excl:tenuring switch, it works fine up to 25000 or more. Here
is the code:
(defun make-sim-matrix (num-docs)
"Makes sim matrix as vector of vectors."
(declare (optimize (speed 3) (safety 1) (compilation-speed 0) (space 0))
(type (integer 0 65535) num-docs))
(excl:tenuring
(let ((sim-matrix
(the simple-array
(make-array num-docs
:initial-element
(the simple-array (make-array 0 :element-type
'(unsigned-byte 16)))
:element-type 'simple-array))))
(dotimes (i num-docs)
(declare (type (integer 0 65535) i))
(setf (aref sim-matrix i)
(make-array i
:initial-element 0
:element-type '(unsigned-byte 16)))) (values
sim-matrix))))
What's going on? Why won't this work under CLISP or CMUCL? I am running
Solaris 8 on Tatung Ultrasparc hardware with 512 MB RAM, 900MB swap.
Thanks.
-bruce
lamb...@uic.edu
Raymond Wiker
> I sometimes need to allocate very large data strucutres. The code causes
> CLISP and CMUCL to crash when I give arguments greater than about 15000. On
> ACL, using the excl:tenuring switch, it works fine up to 25000 or more. Here
> is the code:
25000^2 is 625 million elements. If you allocate 4 bytes per
element, you'll need 2.5 GB to hold it all. It may be that CMUCL needs
4 bytes per element in this case, while ACL may be able to allocate
only 2 bytes per element.
It may be an idea to check what your current process limits
are, particularly the maximum data segment size.
--
Raymond Wiker Mail: Raymon...@fast.no
Senior Software Engineer Web: http://www.fast.no/
Fast Search & Transfer ASA Phone: +47 23 01 11 60
P.O. Box 1677 Vika Fax: +47 35 54 87 99
NO-0120 Oslo, NORWAY Mob: +47 48 01 11 60
Try FAST Search: http://alltheweb.com/
Raymond Toy
Bruce> I sometimes need to allocate very large data strucutres. The code causes
Bruce> CLISP and CMUCL to crash when I give arguments greater than about 15000. On
Bruce> ACL, using the excl:tenuring switch, it works fine up to 25000 or more. Here
Bruce> is the code:
Bruce> (defun make-sim-matrix (num-docs)
Bruce> "Makes sim matrix as vector of vectors."
Bruce> (declare (optimize (speed 3) (safety 1) (compilation-speed 0) (space 0))
Bruce> (type (integer 0 65535) num-docs))
Bruce> (excl:tenuring
Bruce> (let ((sim-matrix
Bruce> (the simple-array
Bruce> (make-array num-docs
Bruce> :initial-element
Bruce> (the simple-array (make-array 0 :element-type
Bruce> '(unsigned-byte 16)))
Bruce> :element-type 'simple-array))))
Bruce> (dotimes (i num-docs)
Bruce> (declare (type (integer 0 65535) i))
Bruce> (setf (aref sim-matrix i)
Bruce> (make-array i
Bruce> :initial-element 0
Bruce> :element-type '(unsigned-byte 16)))) (values
Bruce> sim-matrix))))
Bruce> What's going on? Why won't this work under CLISP or CMUCL? I am running
The dynamic heap size for CMUCL is limited. With recent versions of
CMUCL (after 18c), you can use -dynamic-space-size <n> to set the size
(the default is 256 MB, I think). However, you are still limited to a
maximum of 1 GB.
Ray
Joe Marshall
pointers, you might want to allocate it outside of the
lisp heap. There are different ways of doing this
depending on the Lisp implementation you are using.
Allocating outside the heap will also keep the GC
from trying to relocate this object.
"Bruce Lambert" <lamb...@attbi.com> wrote in message
news:GWbl8.60169$Yv2.25245@rwcrnsc54...
Paolo Amoroso
wrote:
> I sometimes need to allocate very large data strucutres. The code causes
> CLISP and CMUCL to crash when I give arguments greater than about 15000. On
Try using (ROOM T) with CLISP and CMU CL to check how much space your data
structures take.
Paolo
--
EncyCMUCLopedia * Extensive collection of CMU Common Lisp documentation
http://www.paoloamoroso.it/ency/README
[http://cvs2.cons.org:8000/cmucl/doc/EncyCMUCLopedia/]
Bruce Lambert
news:Lqnl8.31214$44.85...@typhoon.ne.ipsvc.net...
> When allocating something this huge that doesn't contain
> pointers, you might want to allocate it outside of the
> lisp heap. There are different ways of doing this
> depending on the Lisp implementation you are using.
> Allocating outside the heap will also keep the GC
> from trying to relocate this object.
I think this is what the excl:tenuring macro does in ACL. Does anyone know
the equivalent trick in CMUCL or CLISP?
-bruce
Bruce Lambert
"Raymond Toy" <t...@rtp.ericsson.se> wrote in message
news:4n7ko95...@rtp.ericsson.se...
> The dynamic heap size for CMUCL is limited. With recent versions of
> CMUCL (after 18c), you can use -dynamic-space-size <n> to set the size
> (the default is 256 MB, I think). However, you are still limited to a
> maximum of 1 GB.
Ray,
I tried 'cmucl -dynamic-space-size 1000' and the program crapped out even
quier than before. I also tried with various smaller values
of -dynamic-space-size, with no apparent effect. Suggestions? How can I tell
how much space is allocated to the heap once CMUCL starts up. (room) does
not seem to be informative enough, or I'm not looking in the right place.
Thanks.
-bruce
Bruce Lambert
> CLISP and CMUCL to crash
I must apologize to the good people who develop CLISP! It may have crashed
an earlier version of CLISP, but a more recent version succeeds in
allocating a very large array:
3. Break [4]> (time (progn (make-sim-matrix 25000) t))
Real time: 176.75488 sec.
Run time: 26.77 sec.
Space: 625355224 Bytes
GC: 11, GC time: 14.55 sec.
Still can't get it to work on CMUCL however, and would appreciate any help
in doing so.
-bruce
Rahul Jain
> Still can't get it to work on CMUCL however, and would appreciate any help
> in doing so.
Works for me (TM).
> lisp -dynamic-space-size 800
CMU Common Lisp release x86-linux 3.0.8 18c+ 31 December 2001 build 3030, running on photino
For support see http://www.cons.org/cmucl/support.html Send bug reports to the debian BTS.
or to pvan...@debian.org
type (help) for help, (quit) to exit, and (demo) to see the demos
Loaded subsystems:
Python 1.0, target Intel x86
CLOS based on PCL version: September 16 92 PCL (f)
* (setq *bytes-consed-between-gcs* 100000000
*gc-verbose* nil)
NIL
* (defun make-sim-matrix (num-docs)
"Makes sim matrix as vector of vectors."
(declare (optimize (speed 3) (safety 1) (compilation-speed 0) (space 0))
(type (integer 0 65535) num-docs))
(let ((sim-matrix
(the simple-array
(make-array num-docs
:initial-element
(the simple-array
(make-array 0 :element-type '(unsigned-byte 16)))
:element-type 'simple-array))))
(dotimes (i num-docs)
(declare (type (integer 0 65535) i))
(setf (aref sim-matrix i)
(make-array i
:initial-element 0
:element-type '(unsigned-byte 16))))
(values sim-matrix)))
MAKE-SIM-MATRIX
* (compile 'make-sim-matrix)
Compiling LAMBDA (NUM-DOCS):
Compiling Top-Level Form:
MAKE-SIM-MATRIX
NIL
NIL
* (time (progn (make-sim-matrix 25000) t))
Compiling LAMBDA NIL:
Compiling Top-Level Form:
Evaluation took:
8.98 seconds of real time
3.24 seconds of user run time
3.82 seconds of system run time
[Run times include 5.58 seconds GC run time]
420 page faults and
924856373 bytes consed.
T
*
--
-> -/ - Rahul Jain - \- <-
-> -\ http://linux.rice.edu/~rahul -=- mailto:rj...@techie.com /- <-
-> -/ "Structure is nothing if it is all you got. Skeletons spook \- <-
-> -\ people if [they] try to walk around on their own. I really /- <-
-> -/ wonder why XML does not." -- Erik Naggum, comp.lang.lisp \- <-
|--|--------|--------------|----|-------------|------|---------|-----|-|
(c)1996-2002, All rights reserved. Disclaimer available upon request.
Raymond Toy
Bruce> "Raymond Toy" <t...@rtp.ericsson.se> wrote in message
Bruce> news:4n7ko95...@rtp.ericsson.se...
>> The dynamic heap size for CMUCL is limited. With recent versions of
>> CMUCL (after 18c), you can use -dynamic-space-size <n> to set the size
>> (the default is 256 MB, I think). However, you are still limited to a
>> maximum of 1 GB.
Bruce> Ray,
Bruce> I tried 'cmucl -dynamic-space-size 1000' and the program crapped out even
Bruce> quier than before. I also tried with various smaller values
Bruce> of -dynamic-space-size, with no apparent effect. Suggestions? How can I tell
Hmm, a tricky bug. I don't know what to do. I tried running your
test on my machine, but it only has 256 MB, so when it started
swapping, I killed it.
Bruce> how much space is allocated to the heap once CMUCL starts up. (room) does
Bruce> not seem to be informative enough, or I'm not looking in the right place.
Try the following:
* (alien:def-alien-variable ("dynamic_space_size" dynamic-space-size) c-call::int)
* dynamic-space-size
Ray
Raymond Toy
Bruce> "Raymond Toy" <t...@rtp.ericsson.se> wrote in message
Bruce> news:4n7ko95...@rtp.ericsson.se...
>> The dynamic heap size for CMUCL is limited. With recent versions of
>> CMUCL (after 18c), you can use -dynamic-space-size <n> to set the size
>> (the default is 256 MB, I think). However, you are still limited to a
>> maximum of 1 GB.
Bruce> Ray,
Bruce> I tried 'cmucl -dynamic-space-size 1000' and the program crapped out even
Bruce> quier than before. I also tried with various smaller values
Bruce> of -dynamic-space-size, with no apparent effect. Suggestions? How can I tell
I tried this again. Yes, there seems to be a problem with a size of
1000. Try 500. I used the following modified code to test this:
(defun make-sim-matrix (num-docs)
"Makes sim matrix as vector of vectors."
(declare (optimize (speed 3) (safety 1) (compilation-speed 0) (space 0))
(type (integer 0 65535) num-docs))
(let ((sim-matrix
(the simple-array
(make-array num-docs
:initial-element
nil))))
(dotimes (i num-docs)
(declare (type (integer 0 65535) i))
(setf (aref sim-matrix i)
(make-array i
:element-type '(unsigned-byte 16))))
(values sim-matrix)))
sim-matrix is initialized to nil instead of a zero-element matrix,
since (upgraded-array-element-type 'simple-array) is T. I also
deleted the initializer for the 16-bit arrays since CMUCL always
initializes arrays to 0.
With this, I can (make-sim-matrix 10000) without a problem. I tried a
size of 25000 and let it run until top said it was using 416 MB of
memory before I killed the process. (I only have 256 MB on my
machine and it was swapping like mad.)
Ray
Paolo Amoroso
wrote:
> of -dynamic-space-size, with no apparent effect. Suggestions? How can I tell
> how much space is allocated to the heap once CMUCL starts up. (room) does
> not seem to be informative enough, or I'm not looking in the right place.
(ROOM T)
Bruce L. Lambert, Ph.D.
>
> Compiling LAMBDA NIL:
> Compiling Top-Level Form:
>
> Evaluation took:
> 8.98 seconds of real time
> 3.24 seconds of user run time
> 3.82 seconds of system run time
> [Run times include 5.58 seconds GC run time]
> 420 page faults and
> 924856373 bytes consed.
Rahul,
What hardware are you running on?
-bruce
Rahul Jain
> Rahul,
>
> What hardware are you running on?
K6-3 400, VIA MVP3 chipset, 640MB RAM, ~500MB swap.
Joe Marshall
> "Joe Marshall" <prunes...@attbi.com> wrote in message
> news:Lqnl8.31214$44.85...@typhoon.ne.ipsvc.net...
> > When allocating something this huge that doesn't contain
> > pointers, you might want to allocate it outside of the
> > lisp heap. There are different ways of doing this
> > depending on the Lisp implementation you are using.
> > Allocating outside the heap will also keep the GC
> > from trying to relocate this object.
>
> I think this is what the excl:tenuring macro does in ACL. Does anyone know
> the equivalent trick in CMUCL or CLISP?
No, the excl:tenuring macro in ACL sets the generation spread of the GC
to zero and then performs two flips. In effect, this causes the live
data to migrate to a more static space.
I was suggesting allocating outside the heap altogether.