PermGen issue
MikeM
Nutter on implementing dynamic languages on the JVM:
http://blog.headius.com/2008/09/first-taste-of-invokedynamic.html
Nutter mentions a possible OutOfMemory condition that can occur due to
filling the PermGen part of the heap with class information, due to
the fact that JRuby emits lots of classes. Seems like Clojure would
have the same issue, but I haven't experienced it. Since I have
started using Clojure in long-running webapps, with the intent of
making additions and changes on the fly, should I be concerned about
PermGen with Clojure?
Rich Hickey
potentially more classes than a Java app might, if it had the same
number of methods, the difference is not that great, as there is far
more function reuse in Clojure, and thus fewer functions than you
would have methods.
More to the permgen point, Clojure does not do any dynamic ephemeral
class creation. It creates classes for the functions you define.
Period.
In particular Clojure does not:
- Create classes to wrap calls to Java, dynamically or otherwise.
Clojure's access to Java is either via reflection, or, with type
inference, direct. No wrappers!
- Dynamically create classes as an optimization technique. Some
languages have dynamic OO strategies that were predicated on
interpretation. When trying to make them run fast on the JVM, you want
to compile to bytecode the results of that strategy, often based on
dynamic data, at runtime. And on the JVM, bytecode means classes.
Because the runtime profile or class/method hierarchy might change,
these optimizations sometimes need to be thrown away and recalculated,
thus potentially accumulating garbage in any class-related area the
JVM doesn't clean up. I expect the collectibility of classes will
improve in future VMs.
On a long-running Clojure web app you'd just need to take care not to
reload anything that didn't need reloading, but loading fixed fns
should not present a permgen problem. In any case, you know what's
going on, a fn definition creates a class, no other hidden class gen.
Note also that by fn definition I mean the occurrence in code, not the
execution:
(defn foo [x] (fn [] x))
creates two classes, no matter how many times foo is called. Each call
to foo creates an instance of the same class.
One nice thing about Clojure is it was written to run well on today's
(actually yesterday's) JVM. JVM enhancements will be nice, but we're
not waiting for nor dependent upon them.
Rich
Randall R Schulz
> On Sep 14, 6:26 am, MikeM <michael.messini...@invista.com> wrote:
> > If you haven't already seen it, an interesting post from Charles
> > Nutter on implementing dynamic languages on the
> > JVM:http://blog.headius.com/2008/09/first-taste-of-invokedynamic.ht
> >ml
> >
> > Nutter mentions a possible OutOfMemory condition that can occur due
I've had similar questions, though I haven't yet read Charles' article.
> Nope. While Clojure does use a class per function, and thus
> potentially more classes than a Java app might, if it had the same
> number of methods, the difference is not that great, as there is far
> more function reuse in Clojure, and thus fewer functions than you
> would have methods.
What is the magnitude of the space overhead of a class' internal
representation, beyond the bytecodes comprising its method(s)?
> More to the permgen point, Clojure does not do any dynamic ephemeral
> class creation. It creates classes for the functions you define.
> Period.
>
> ...
>
> Note also that by fn definition I mean the occurrence in code, not
> the execution:
>
> (defn foo [x] (fn [] x))
>
> creates two classes, no matter how many times foo is called. Each
> call to foo creates an instance of the same class.
Why is an instance creation required for an invocation? I thought you
used the JVM stack for calling and parameter passing both for Clojure
functions and (naturally) for calls to "ordinary" Java methods. If each
invocation of a Clojure function requires instantiation of the
function's class, aren't you effectively using those instances as call
stack frames? And does this not limit function call rate to
significantly less than the object allocation rate? And doesn't it
rapidly tear through the eden space?
If it isn't obvious, I know very little about the JVM or about how you
compile Clojure to JVM bytecodes, so perhaps I'm way off. Please excuse
my ignorance.
> One nice thing about Clojure is it was written to run well on today's
> (actually yesterday's) JVM. JVM enhancements will be nice, but we're
> not waiting for nor dependent upon them.
I've also been wondering whether there's anything coming in the JVM that
will relax the current TCO limitations in Clojure?
> Rich
Randall Schulz
Rich Hickey
On Sep 14, 10:09 am, Randall R Schulz <rsch...@sonic.net> wrote:
> On Sunday 14 September 2008 05:57, Rich Hickey wrote:
>
> > On Sep 14, 6:26 am, MikeM <michael.messini...@invista.com> wrote:
> > > If you haven't already seen it, an interesting post from Charles
> > > Nutter on implementing dynamic languages on the
> > > JVM:http://blog.headius.com/2008/09/first-taste-of-invokedynamic.ht
> > >ml
>
> > > Nutter mentions a possible OutOfMemory condition that can occur due
> > > to filling the PermGen part of the heap with class information, ...
>
> I've had similar questions, though I haven't yet read Charles' article.
>
> > Nope. While Clojure does use a class per function, and thus
> > potentially more classes than a Java app might, if it had the same
> > number of methods, the difference is not that great, as there is far
> > more function reuse in Clojure, and thus fewer functions than you
> > would have methods.
>
> What is the magnitude of the space overhead of a class' internal
> representation, beyond the bytecodes comprising its method(s)?
>
is likely to change from JVM to JVM.
> > More to the permgen point, Clojure does not do any dynamic ephemeral
> > class creation. It creates classes for the functions you define.
> > Period.
>
> > ...
>
> > Note also that by fn definition I mean the occurrence in code, not
> > the execution:
>
> > (defn foo [x] (fn [] x))
>
> > creates two classes, no matter how many times foo is called. Each
> > call to foo creates an instance of the same class.
>
> Why is an instance creation required for an invocation?
which an instance is created.
> I thought you
> used the JVM stack for calling and parameter passing both for Clojure
> functions and (naturally) for calls to "ordinary" Java methods.
> If each
> invocation of a Clojure function requires instantiation of the
> function's class
> > One nice thing about Clojure is it was written to run well on today's
> > (actually yesterday's) JVM. JVM enhancements will be nice, but we're
> > not waiting for nor dependent upon them.
>
> I've also been wondering whether there's anything coming in the JVM that
> will relax the current TCO limitations in Clojure?
it has high priority, though.
Rich
Randall R Schulz
> On Sep 14, 10:09 am, Randall R Schulz <rsch...@sonic.net> wrote:
> > On Sunday 14 September 2008 05:57, Rich Hickey wrote:
> > > ...
> > >
> > > Note also that by fn definition I mean the occurrence in code,
> > > not the execution:
> > >
> > > (defn foo [x] (fn [] x))
> > >
> > > creates two classes, no matter how many times foo is called. Each
> > > call to foo creates an instance of the same class.
> >
> > Why is an instance creation required for an invocation?
>
> It's not. This function, foo, returns a closure. It is the closure
> for which an instance is created.
OK. I misunderstood. So a closure is created only when the function is
treated like a value?
> > I thought you used the JVM stack for calling and parameter passing
> > both for Clojure functions and (naturally) for calls to "ordinary"
> > Java methods.
>
> I do.
>
> > If each invocation of a Clojure function requires instantiation of
> > the function's class
>
> It doesn't.
I looked at the source for clojure.lang.IFn and the invoke(...) methods
are not static. What instance of the function's implementing Java class
is used in cases where no closure is required?
> ...
>
> Rich
Randall Schulz
Rich Hickey
On Sep 14, 5:16 pm, Randall R Schulz <rsch...@sonic.net> wrote:
> On Sunday 14 September 2008 07:47, Rich Hickey wrote:
>
>
>
> > On Sep 14, 10:09 am, Randall R Schulz <rsch...@sonic.net> wrote:
> > > On Sunday 14 September 2008 05:57, Rich Hickey wrote:
> > > > ...
>
> > > > Note also that by fn definition I mean the occurrence in code,
> > > > not the execution:
>
> > > > (defn foo [x] (fn [] x))
>
> > > > creates two classes, no matter how many times foo is called. Each
> > > > call to foo creates an instance of the same class.
>
> > > Why is an instance creation required for an invocation?
>
> > It's not. This function, foo, returns a closure. It is the closure
> > for which an instance is created.
>
> OK. I misunderstood. So a closure is created only when the function is
> treated like a value?
>
sticks it in a var. Once you have an instance it's a perfectly fine
value.
> > > I thought you used the JVM stack for calling and parameter passing
> > > both for Clojure functions and (naturally) for calls to "ordinary"
> > > Java methods.
>
> > I do.
>
> > > If each invocation of a Clojure function requires instantiation of
> > > the function's class
>
> > It doesn't.
>
> I looked at the source for clojure.lang.IFn and the invoke(...) methods
> are not static. What instance of the function's implementing Java class
> is used in cases where no closure is required?
>
even when nothing is closed over, in order to find the implementations
of the interface methods (i.e. for the vtable in C++ parlance).
Rich