Critical memory leak with J/Link

[J. McKenzie Alexander]

I've encountered a curious, and completely disabling, memory leak when
I use J/Link. The memory leak is such that a program can easily
consume over 1.5GB of virtual memory and will crash the system if left
to run for an extended period of time. The leak occurs under both
Windows XP and Mac OS X (Panther), with both machines running
Mathematica 5.0 (J/Link version 2.1.1). The XP box has Sun Java
1.4.1_03 (build 2) installed, and the OS X box has java 1.4.2_03 (build
1.4.2_03-117.1) installed.

Here's the simplest example I've found.

// MemoryLeaker.java
import com.wolfram.jlink.*;
public class MemoryLeaker {
Expr expr;
public MemoryLeaker(Expr e) {
expr = e;
}
public Expr getExpr() {
return expr;
}
}

Once MemoryLeaker is compiled, load it into Mathematica and use it as
follows (adjusting the AddToClassPath command as necessary, of course).

In[1]:=
Needs["JLink`"];
<<DiscreteMath`Combinatorica`

In[3]:=
AddToClassPath["/Users/jalex/Source/tmp/"];

In[4]:=
LoadJavaClass["MemoryLeaker"];

In[5]:=
g=GridGraph[50,50];

In[6]:=
memoryLeaker=JavaNew["MemoryLeaker",
NormalizeVertices[g]
];

In[7]:=
Do[
memoryLeaker@getExpr[],{50}
];

The above run leaves J/Link, on my OS X box, in the following state:

ID Name %CPU Real Memory Virtual Memory
1287 J/Link 0.00 265.50 MB 823.50 MB

Memory is not freed with calls to JavaGC[]. It is only freed with
QuitJava[] or quitting the kernel.

The real kicker is that if you replace line 6 with the following:

In[6]:=
memoryLeaker=JavaNew["MemoryLeaker", g ];

no leak occurs. However, if you replace line 6 with

In[6]:=
g2 = NormalizeVertices[g];
memoryLeaker = JavaNew["MemoryLeaker", g2];

you still get the memory leak.

Would someone please explain what I'm doing wrong in the above example
or, if I'm not doing anything wrong, a work-around?

Cheers,

Jason
--
Dr J. McKenzie Alexander
Department of Philosophy, Logic and Scientific Method
London School of Economics and Political Science
Houghton Street, London WC2A 2AE

[Werner Schuster]

> Memory is not freed with calls to JavaGC[]. It is only freed with
> QuitJava[] or quitting the kernel.

You might want to refer to the "Reference Counts and Memory Management"
section of the J/Link Userguide;

JavaGC[] won't be of much help, if there is still a valid reference
to the Object; in your case, the "g" expression still points to the
MemoryHolder, which in turn still references the large Graph Expression
that you passed in. This is, as far as I see it, correct behaviour
(Mathematica or J/Link doesn't know that you don't intend to use
the "g" expression/reference anymore, so it keeps it).

You can use either JavaBlock[] (wrap your code inside it) or one
of the Release* Expressions (ReleaseObject[],...) to get rid of the
reference (possibly followed by a JavaGC[] to make sure the GC frees the
memory).

I suppose that should solve your problem (unless I'm missing
something else).

murphee
--
Werner Schuster (murphee)
Student of SoftwareEngineering and KnowledgeManagement
Maintainer of the OGO-JOGI Project @ http://ogo-jogi.sourceforge.net/
Blog @ http://jroller.com/page/murphee

[J. McKenzie Alexander]

>> Memory is not freed with calls to JavaGC[]. It is only freed with
>> QuitJava[] or quitting the kernel.
>

> JavaGC[] won't be of much help, if there is still a valid reference
> to the Object; in your case, the "g" expression still points to the
> MemoryHolder, which in turn still references the large Graph Expression
> that you passed in. This is, as far as I see it, correct behaviour
> (Mathematica or J/Link doesn't know that you don't intend to use
> the "g" expression/reference anymore, so it keeps it).
>
> You can use either JavaBlock[] (wrap your code inside it) or one
> of the Release* Expressions (ReleaseObject[],...) to get rid of the
> reference (possibly followed by a JavaGC[] to make sure the GC frees
> the
> memory).
>
> I suppose that should solve your problem (unless I'm missing
> something else).

Thanks for your reply, but I'm afraid the problem is a bit more subtle
than it first appear. The problem doesn't concern how Mathematica keeps
references to returned Java objects unless explicitly cleared with
Scan[ReleaseJavaObject,PeekObjects[]] or something similar. The Java
object returned in the body of the Do[...] loop is the *same* Java
object each time. Mathematica realises this and does not make a copy
of the object or continue to increase the reference count each time.
If you look at the class definition of MemoryLeaker, you'll see that
it stores a reference to whatever single Expr object is passed to its
constructor when it is created, and repeated calls to
memoryLeaker@getExpr[] simply return a reference to that same object.

The problem is that, out of two *very* similar but slightly different
sequences of commands, one causes a huge leak but the other doesn't.
Compare the following sequence of commands:

In[1]:=
Needs["JLink`"];
<<DiscreteMath`Combinatorica`

In[3]:=
AddToClassPath["/Users/jalex/Source/tmp/"];

In[4]:=
LoadJavaClass["MemoryLeaker"];

In[5]:=
g=GridGraph[50,50];

In[6]:=
memoryLeaker=JavaNew["MemoryLeaker",g];

In[7]:=
Do[
memoryLeaker@getExpr[],{50}
];

In[8]:=
PeekObjects[]//Length

Out[8]=
1

State of J/Link process at end
=======================
Process Name %CPU Real memory Virtual memory
1490 J/Link 0.00 20.71 MB 313.33 MB

Ok - fine. In my experience this is normal memory usage. Compare this
with the following. The kernel was restarted between the above run and
this run, and the cell tags indicate that the exact same sequence of
command evaluation was followed. Note that the *only* difference is at
In[6], where instead of passing the GridGraph to the constructor, we
pass the output of the Combinatorica function NormalizeVertices.

In[1]:=
Needs["JLink`"];
<<DiscreteMath`Combinatorica`

In[3]:=
AddToClassPath["/Users/jalex/Source/tmp/"];

In[4]:=
LoadJavaClass["MemoryLeaker"];

In[5]:=
g=GridGraph[50,50];

In[6]:=
memoryLeaker=JavaNew["MemoryLeaker",
NormalizeVertices[g]
];

In[7]:=
Do[
memoryLeaker@getExpr[],{50}
];

In[8]:=
PeekObjects[]//Length

Out[8]=
1

State of J/Link process at end
=======================
Process Name %CPU Real memory Virtual memory
1492 J/Link 0.50 279.81 MB 823.50 MB

See the difference? That's a *huge* leak for no apparent reason. And
this isn't a bug in Apple's JVM because it's perfectly reproducible on
a Windows XP box running Mathematica (as noted in my original message).

One may suspect that something funny is going on with the fact that we
passed NormalizeVertices[g] to the constructor instead of a graph
object itself. Well, compare the second sequence of commands (the one
we just saw) with the following. Note that the only difference between
the above and the below is that In[6] assigns the output of
NormalizeVertices[g] to g2, and then we pass g2 to the MemoryLeaker
constructor. As before, the kernel was restarted between the above run
and this one. (Quitting the kernel destroys the J/Link connection and
the memory is freed.)

In[1]:=
Needs["JLink`"];
<<DiscreteMath`Combinatorica`

In[3]:=
AddToClassPath["/Users/jalex/Source/tmp/"];

In[4]:=
LoadJavaClass["MemoryLeaker"];

In[5]:=
g=GridGraph[50,50];

g2=NormalizeVertices[g];

In[7]:=
memoryLeaker=JavaNew["MemoryLeaker",g2];

In[8]:=
Do[
memoryLeaker@getExpr[],{50}
];

In[9]:=
PeekObjects[]//Length

Out[9]=
1

State of J/Link process at end
=======================
Process Name %CPU Real memory Virtual memory
1500 J/Link 0.40 299.09 MB 823.50 MB

Any solutions or suggestions would be appreciated. This phenomenon is
killing my J/Link app.

[Todd Gayley]

At 02:39 AM 4/15/2004, J. McKenzie Alexander wrote:
>I've encountered a curious, and completely disabling, memory leak when
>I use J/Link. The memory leak is such that a program can easily
>consume over 1.5GB of virtual memory and will crash the system if left
>to run for an extended period of time. The leak occurs under both
>Windows XP and Mac OS X (Panther), with both machines running
>Mathematica 5.0 (J/Link version 2.1.1). The XP box has Sun Java
>1.4.1_03 (build 2) installed, and the OS X box has java 1.4.2_03 (build
>1.4.2_03-117.1) installed.
>
>Here's the simplest example I've found.
>
> // MemoryLeaker.java
> import com.wolfram.jlink.*;
> public class MemoryLeaker {
> Expr expr;
> public MemoryLeaker(Expr e) {
> expr = e;
> }
> public Expr getExpr() {
> return expr;
> }
> }
>
>Once MemoryLeaker is compiled, load it into Mathematica and use it as
>follows (adjusting the AddToClassPath command as necessary, of course).
>

> In[1]:=
> Needs["JLink`"];
> <<DiscreteMath`Combinatorica`
>
> In[3]:=
> AddToClassPath["/Users/jalex/Source/tmp/"];
>
> In[4]:=
> LoadJavaClass["MemoryLeaker"];
>
> In[5]:=
> g=GridGraph[50,50];
>
> In[6]:=
> memoryLeaker=JavaNew["MemoryLeaker",
> NormalizeVertices[g]
> ];
>
> In[7]:=
> Do[
> memoryLeaker@getExpr[],{50}
> ];
>

>The above run leaves J/Link, on my OS X box, in the following state:
>
>ID Name %CPU Real Memory Virtual Memory
>1287 J/Link 0.00 265.50 MB 823.50 MB
>

>Memory is not freed with calls to JavaGC[]. It is only freed with
>QuitJava[] or quitting the kernel.
>

>The real kicker is that if you replace line 6 with the following:
>

> In[6]:=

> memoryLeaker=JavaNew["MemoryLeaker", g ];
>
>no leak occurs. However, if you replace line 6 with
>
> In[6]:=
> g2 = NormalizeVertices[g];
> memoryLeaker = JavaNew["MemoryLeaker", g2];
>
>you still get the memory leak.
>
>Would someone please explain what I'm doing wrong in the above example
>or, if I'm not doing anything wrong, a work-around?

Jason,

Thanks for this bug report. We are studying the leak right now, but we know
that it is down inside MathLink itself and has to do with calling
MLTransferExpression() to move certain unusual types of data between links.
The leak can be easily duplicated in a C-language MathLink program.

Until this is fixed, there is a partial workaround in your J/Link program
that might be sufficient for you. You need to change the way the Expr is
stored internally so that MLTransferExpression() is not used to send it to
Mathematica. This can be done by calling any Expr method that "unwinds" the
expression data off of its highly optimized cache on an internal loopback
link. Any Expr method that inspects the expression in any way will perform
this unwinding; here is an example that uses length():

// MemoryLeaker.java
import com.wolfram.jlink.*;
public class MemoryLeaker {
Expr expr;
public MemoryLeaker(Expr e) {
expr = e;

// ADDED LINE. This has the side effect of unwinding the
// expression data off the internal loopback link:
expr.length();

}
public Expr getExpr() {
return expr;
}
}

This will make the movement of the data from Java to Mathematica not leak
memory (it will also make it a bit slower, but this will probably not be a
concern). Unfortunately, the initial send of the expression from
Mathematica to Java still uses MLTransferExpression() internally and this
cannot be changed. If you are only sending the data to Java once, and
retrieving it many times, the fix provided above will be enough for you,
but if you are sending expressions from Mathematica to Java as many times
as the other way around, then you will still have a problem.

If the above workaround is not sufficient for you, and you cannot wait for
a fix to show up in MathLink, then contact me directly and we can discuss
more complicated techniques, such as avoiding the Expr class entirely.

Todd Gayley
Wolfram Research