Help needed in handling large LISP file
P. Srinivas
Hi,
I have this problem of loading a large file (160Mb) that contain
a huge S-expression into LISP system. I am running Allegro CL 4.3.1
on Ultra Sparc (Solaris). But when I tried to load this file, it tries
for 2 hrs before it gives up saying that it can not satisfy the
memory request.
I will explain the problem in detail.
I have a huge tree created by some other LISP program that is saved
into a file as a huge S-expression so that when it gets loaded into
another LISP system, the tree structured is re-created preserving
the links and some of the important attributes of each node.
But the problem is that when I tried to load, Allegor CL gives up
with a memory problem. I know this is the system problem not
ACL problem. I have 64Mb of RAM and a 1Gb of swap partition.
ACL spends half of the CPU time swaping and garbage collecting.
My questions are as follows:
1) Are there any ACL parameters/Solaris parameters that
I should change to handle large data?
2) Are there any better, more efficient ways of saving and re-creating
large set of objects. I know that the real solution is to go
for full-fledged OODMS. But that is not a solution for now.
I just need to handle with basic CL and file system.
3) Why does one CL was able to handle the large tree and able to save
into a file, but the same size tree can not be created(loaded)
into another LISP process. Both tehse LISP processes are
on the same machine and same LISP. (I am running each of these LISP
processes as exclusively so that the two are not running at the
same time). Are there any extra overheads inharent in constructing
objects from LISP forms from a file?
4) Dumping in a binary form is not a solution as the objects in the
source tree and the objects in the target tree are not exactly
instances of the same class. They differ in a minor ways. What I
was doing is changing the class-names and attributes names to
the target object system while saving into a file.
5) Does compilation of this hge file help? (Assming it can be compiled
without any problem, which, I suspect, might not be the case.
(have anybody tried to compile a LISP file of 160Mb size?).
Any help from LISP hackers is very much apreciated.
srini
--------------------
Srinivas Palthepu Email: sr...@cs.usask.ca
ARIES laboratory, Phones: (306) 966-8654 (lab)
Department of Computer Science, (306) 966-4759 (office)
University of Saskatchewan, (306) 373-6724 (home)
57 Campus Dr, Saskatoon, SK, S7N 5A9 Fax: (306) 966-4884
Laurent Arditi
Are you sure your tree actually was not a graph?
Even if you didn't build it as a graph, you may have unintentional
structure
sharing. This is why, you were able to construct the tree and save it
but
produced a so huge file.
The solution is:
1) to really handle your structure as a graph and not as a tree. So you
will save
a graph which will be much smaller than the tree.
Or
2) to save the tree as you do now. But when you load it, do not use a
single (read).
Instead, use a deep-recursive function to load it node by node, and
creating a new node
only when it does not already exist.
P. Srinivas wrote:
>
> Hi,
>
> I have this problem of loading a large file (160Mb) that contain
> a huge S-expression into LISP system. I am running Allegro CL 4.3.1
> on Ultra Sparc (Solaris). But when I tried to load this file, it tries
> for 2 hrs before it gives up saying that it can not satisfy the
> memory request.
>
> I will explain the problem in detail.
>
> I have a huge tree created by some other LISP program that is saved
> into a file as a huge S-expression so that when it gets loaded into
> another LISP system, the tree structured is re-created preserving
> the links and some of the important attributes of each node.
> But the problem is that when I tried to load, Allegor CL gives up
> with a memory problem. I know this is the system problem not
> ACL problem. I have 64Mb of RAM and a 1Gb of swap partition.
> ACL spends half of the CPU time swaping and garbage collecting.
> ...
--
============================================================ |\/\/\/|
Laurent Arditi Texas Instruments France | |
Email: lar...@tif.ti.com MS. 21, BP 5 ( O O )
Tel: +33 (0)4 9322 2856 06271 Villeneuve-Loubet Cedex o |
Erik Naggum
* Laurent Arditi -> P. Srinivas
| Are you sure your tree actually was not a graph? Even if you didn't
| build it as a graph, you may have unintentional structure sharing. This
| is why, you were able to construct the tree and save it but produced a so
| huge file.
|
| The solution is:
it doesn't appear that this is the problem, but if it is, I'd venture
that
(setq *print-circle* t)
should take care of these problems in the writing Lisp image. however,
it _might_ take a little longer to write the tree to disk now...
if Allegro CL complains about memory problems before it finishes loading,
it might be because it is trying too hard to keep objects in new-space.
the macro TENURING in the FRANZ (or EXCL) package will try to make most
allocations in old-space. if you know you will need a lot of old-space,
you can also pre-allocate that space with SYSTEM:RESIZE-AREAS, as in:
(system:resize-areas :old (* 160 1024 1024))
Allegro CL can also be built with a larger maximum heap estimate that
defaults to about 120M, but I don't know what this would do to help you.
there's also a LOAD-APPLICATION macro that might be useful. consult the
User Guide for explanations on how to use this and the other functions
mentioned. see also ROOM which should be helpful in reporting where
memory is used.
#:Erik
--
God grant me serenity to accept the code I cannot change,
courage to change the code I can, and wisdom to know the difference.
Stanley
On 19 Feb 1998 03:54:00 GMT, sr...@sparkle.usask.ca (P. Srinivas)
wrote:
>Hi,
>
>I have this problem of loading a large file (160Mb) that contain
>a huge S-expression into LISP system. I am running Allegro CL 4.3.1
>on Ultra Sparc (Solaris). But when I tried to load this file, it tries
>for 2 hrs before it gives up saying that it can not satisfy the
>memory request.
>
>I will explain the problem in detail.
>
>I have a huge tree created by some other LISP program that is saved
>into a file as a huge S-expression so that when it gets loaded into
>another LISP system, the tree structured is re-created preserving
>the links and some of the important attributes of each node.
>But the problem is that when I tried to load, Allegor CL gives up
>with a memory problem. I know this is the system problem not
>ACL problem. I have 64Mb of RAM and a 1Gb of swap partition.
>ACL spends half of the CPU time swaping and garbage collecting.
>
Some suggestions:
!) make sure to use *print-circle* t i fyou think there is significant
shared structure and the resulting performance is "reasonable".
(we found it that in some cases having print-circle enabled slowed the
writing down too much for the limited amount of structure sharing
that we had)
2) if you can alter the program that writes the data,
then fix the s-expression to be something that can be loaded
(via load) and does not need to be read (via read/eval).
3) In the s-expression you write out, define some macros that are
then evaluated during load so as to minimize the structure being
created during the load. Properly done, this can decrease the
size of the file as well.
4) Consider writing the data with your own recursive write routine
(rather than simply use print on on the s-expression).
Then, when writing it out, use #. so as to make the reader evaluate
selected sub-expressions during reading/loading
(rather than creating a big structure that must be interpreted)
This may make the writing process a bit slower, but it can be a huge
improvement in time (and space) required during loading.
5) If you have lots of repeated text strings and such, consider doing
a transformation during the write process to use a reader macro to
reduce the size of the data file. In one case, we cut the file size
by about 40% doing this.
Put this all together, we have often written 10 to 15mb+ lisp files
that are rather simple structure, but LOTS of data:
(setq *the-input-surface*
(make-surface
:patches
(list (list #.(make-point 1.2345 3.245 594.93)
.....))
The other case has smaller files (2-3mb) but lots of structure
(nested 20-30 levels), with some significant structure sharing.
(Some macros in the expression use a hashtable on arguments
during reading so as to achieve better structure sharing after
recovery than we had before)
I don't know how your data compares, but we found these tricks to be
essential in getting the load time to be reasonable.
Feel free to email me directly if you want - I don't read
comp.lang.lisp on a regular basis.
- Stanley Knutson