Macaulay2 Resource Management for Loops

[Jared Painter]

Hi, a student and I have written some code to randomly generate monomial ideals satisfying certain properties and test the Tor Algebra structure on each random ideal.  We are using a for loop to do this and would like to let the loop run through millions of ideals.  The problem is that Macaulay just keeps eating up more and more memory until eventually the program is Killed.  The code is extremely long, but in a nutshell, I have written a function to build the generators for the random monomial ideal that it stores in a list and then we test the Algebra Structure of this ideal, each time we find a different structure that information is stored in a separate list and the list that keeps the generators for the random ideal is cleared with each run of my loop.  The list that stores all of the ideals and Tor Algebra structures found is fairly small (at least right now), so I am not really sure why the code would be using so much memory?  Is there something inherent in Macaulay looping structures that cause them to continue to store information unless you somehow manually clear them out?  I must confess I am not too familiar as to how Macaulay manages memory, but when I run my code on over 500000 iterations of the for loop, then it quickly accumulates more than 8GB of memory.

Thanks,

Jared

[David Eisenbud]

One thing that could make trouble is if you create a new ring each time -- I believe they don't go away by themselves. The rings should be created outside the loop.

Cordially,

David

--

David Eisenbud, Director, 

Mathematical Sciences Research Institute

and Professor of Mathematics, 

University of California, Berkeley

--
You received this message because you are subscribed to the Google Groups "Macaulay2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to macaulay2+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/macaulay2/3a2d5ec9-2e06-41cb-96ae-17b2b2e94e37n%40googlegroups.com.

[Fabio Tanturri]

Hi,

I had the same problem with some computations I am doing. I thought it would be nice to have a selective "clearAll" function, which allows one to forget almost everything (say, every local variable in a for loop, or everything except for some variables). I tried to use it, but if I remember correctly that function erases just everything. I might be wrong though.

Here is the workaround I am adopting. I have a file producing a list of cases I want to test. Then I have a for loop, but in every execution I call for an external M2 process (using the runExternalM2 package) for testing one case and get the answer as a string. I then collect the answer in an output file, which is updated after every cycle. This has the advantage that everything I need to test the input is just thrown away after every cycle execution (for instance, every new ring you define).
Of course this workaround requires that your output is simple enough to be saved as an external string. Sometimes some external processes I run fail (I could not understand why), but in such cases I call for the usual function inside the main M2 process. This limits the resources your program is requiring and lets you run many more cases before running out of memory.

I don't know to what extent this solution can be improved, but I am sure it can, as I am definitively not an expert myself.

Best,
Fabio

[Jared Painter]

I spent a lot of time yesterday looking for different ways to see how the memory is being used in Linux (something I am not so familiar with right now) and based on what I saw is seems that the memory that M2 is eating is just cached memory space.  The thing is I am not sure how to clear it out because it is being cached under root and even running commands to clear the cache logged in as root won't do it.  I am not sure how I would create each new ring outside the loop, since the loop is what is creating the different rings for me.  I will look at the runExternalM2 package and see if I can make that work, but it seems like there should be some way to simply pick memory blocks where these things are assigned and clear them out with each run of the loop, I just don't know how to do that in Linux with M2.

Thanks for the feedback.

Jared 

[Mahrud Sayrafi]

It might help if you're able to share the code you're trying to run. There are various small changes that can add up (e.g. how you use lists or hashtables).

One immediate suggestion is to break up your code into one or more functions. Macaulay2 has an automatic garbage collector which frees memory blocks once they are not pointed to by anything else, so if you allocate a bunch of large objects in a function and return a single integer in the end, since there is no reference to the large objects, they will be deallocated.

Mahrud

To view this discussion on the web visit https://groups.google.com/d/msgid/macaulay2/99484790-f82c-414b-a712-49dcf49dc6c2n%40googlegroups.com.

[Jared Painter]

I have adapted and simplified my code to use the runExternalM2 command, but I have a new problem and the old one still seems to exist in a different form.  The main thing now is that the code is taking 20 times longer to run using the runExternalM2 command than it does without it.  Also I am still eating up memory, just not as quickly as before.  I am just doing test runs right now and haven't really put any debugging in but the loop I am running for my runExternalM2 part is attached below:

compClassList={};

for ii from 1 to n do(

h:=runExternalM2("/home/jpainter/Documents/KoszulStructureFinder_Ver1_0_1.m2",BuildGens3,(a,b,c,r));
if length(compClassList)==0 then(
    compClassList=append(compClassList,h#value);)
else(
    jj=0;   
    while jj<length(compClassList) do(
    if h#value#1==compClassList#jj#1 then(
        jj=length(compClassList);)
    else (
        if jj==length(compClassList)-1 then(
        compClassList=append(compClassList,h#value);)
        else(jj=jj+1);););););  

I have a sort feature that runs after this but it hasn't started yet.  I have left this loop running all night, about 11 hour now and it has only gone through about 150000 reps of the main loop.  For more clarification, the BuildGens3 function is a function that I wrote to build a certain type of random monomial ideal and returns the generators of the ideal and its Tor Algebra structure in a list.  Every time a new Tor Algebra structure is found the while loop in the code below adds it to the compClassList, I am thinking that while loop is where I am still eating up memory I guess.  So the speed problem, comes into focus when I run this code without the runExternalM2 command with 100000 reps it takes less than an hour to complete.  Any suggestions on how to speed this up further?  I suppose I could do another runExternalM2 in the while loop, but haven't had time to think that one through yet.

Thanks.