Hi,

I have this problem of loading a large file (160Mb) that contain
a huge S-expression into LISP system. I am running Allegro CL 4.3.1
on Ultra Sparc (Solaris). But when I tried to load this file, it tries
for 2 hrs before it gives up saying that it can not satisfy the
memory request.

I will explain the problem in detail.

I have a huge tree created by some other LISP program that is saved
into a file as a huge S-expression so that when it gets loaded into
another LISP system, the tree structured is re-created preserving
the links and some of the important attributes of each node.
But the problem is that when I tried to load, Allegor CL gives up
with a memory problem. I know this is the system problem not
ACL problem. I have 64Mb of RAM and a 1Gb of swap partition.
ACL spends half of the CPU time swaping and garbage collecting.

My questions are as follows:

1) Are there any ACL parameters/Solaris parameters that
   I should change to handle large data?

2) Are there any better, more efficient ways of saving and re-creating
   large set of objects. I know that the real solution is to go
   for full-fledged OODMS. But that is not a solution for now.
   I just need to handle with basic CL and file system.

3) Why does one CL was able to handle the large tree and able to save
   into a file, but the same size tree can not be created(loaded)
   into another LISP process. Both tehse LISP processes are
   on the same machine and same LISP. (I am running each of these LISP
   processes as exclusively so that the two are not running at the
   same time). Are there any extra overheads inharent in constructing
   objects from LISP forms from a file?

4) Dumping in a binary form is not a solution as the objects in the
   source tree and the objects in the target tree are not exactly
   instances of the same class. They differ in a minor ways. What I
   was doing is changing the class-names and attributes names to
   the target object system while saving into a file.

5) Does compilation of this hge file help? (Assming it can be compiled
   without any problem, which, I suspect, might not be the case.
   (have anybody tried to compile a LISP file of 160Mb size?).

Any help from LISP hackers is very much apreciated.

Are you sure your tree actually was not a graph?
Even if you didn't build it as a graph, you may have unintentional
structure
sharing. This is why, you were able to construct the tree and save it
but
produced a so huge file.

The solution is:
1) to really handle your structure as a graph and not as a tree. So you
will save
a graph which will be much smaller than the tree.
 Or
2) to save the tree as you do now. But when you load it, do not use a
single (read).
Instead, use a deep-recursive function to load it node by node, and
creating a new node
only when it does not already exist.

* Laurent Arditi -> P. Srinivas
| Are you sure your tree actually was not a graph?  Even if you didn't
| build it as a graph, you may have unintentional structure sharing. This
| is why, you were able to construct the tree and save it but produced a so
| huge file.
|
| The solution is:

  it doesn't appear that this is the problem, but if it is, I'd venture
  that

(setq *print-circle* t)

  should take care of these problems in the writing Lisp image.  however,
  it _might_ take a little longer to write the tree to disk now...

  if Allegro CL complains about memory problems before it finishes loading,
  it might be because it is trying too hard to keep objects in new-space.
  the macro TENURING in the FRANZ (or EXCL) package will try to make most
  allocations in old-space.  if you know you will need a lot of old-space,
  you can also pre-allocate that space with SYSTEM:RESIZE-AREAS, as in:

(system:resize-areas :old (* 160 1024 1024))

  Allegro CL can also be built with a larger maximum heap estimate that
  defaults to about 120M, but I don't know what this would do to help you.

  there's also a LOAD-APPLICATION macro that might be useful.  consult the
  User Guide for explanations on how to use this and the other functions
  mentioned.  see also ROOM which should be helpful in reporting where
  memory is used.

On 19 Feb 1998 03:54:00 GMT, sr...@sparkle.usask.ca (P. Srinivas)
wrote:

!) make sure to use *print-circle* t i fyou think there is significant
 shared structure and the resulting performance is "reasonable".
(we found it that in some cases having print-circle enabled slowed the
writing down too much for the limited amount of structure sharing
that we had)

2) if you can alter the program that writes the data,
then fix the s-expression to be something that can be loaded
(via load) and does not need to be read (via read/eval).

3) In the s-expression you write out, define some macros that are
then evaluated during load so as to minimize the structure being
created during the load.  Properly done, this can decrease the
size of the file as well.

4) Consider writing the data with your own recursive write routine
 (rather than simply use print on on the s-expression).
Then, when writing it out, use #. so as to make the reader evaluate
selected sub-expressions during reading/loading
(rather than creating a big structure  that must be interpreted)
This may make the writing process a bit slower, but it can be a huge
improvement in time (and space) required during loading.

5) If you have lots of repeated text strings and such, consider doing
 a transformation during the write process to use a reader macro to
 reduce the size of the data file.  In one case, we cut the file size
 by about 40% doing this.

Put this all together, we have often written 10 to 15mb+ lisp files
that are rather simple structure, but LOTS of data:

(setq *the-input-surface*
  (make-surface
    :patches
   (list (list #.(make-point 1.2345 3.245 594.93)
          .....))

The other case has smaller files (2-3mb) but lots of structure
(nested 20-30 levels), with some significant structure sharing.
(Some macros in the expression use a hashtable on arguments
 during reading so as to achieve better structure sharing after
 recovery than we had before)

I don't know how your data compares, but we found these tricks to be
 essential in getting the load time to be reasonable.

Feel free to email me directly if you want - I don't read
comp.lang.lisp on a regular basis.