Does 'require' with the :reload option have a tendency to build up memory?

[Rayne]

I've got a curious little bit of a memory leak of sorts that I'm
trying to narrow down.

I have an application (betcha can guess what it is if you know me from
IRC :>) that, in order to reload plugins, requires each of them with
the :reload option whenever you ask them to be reloaded.

Each of these plugins calls a macro that defines a set of defmethods
for a multimethod in a namespace that never does get reloaded. They
also define a single function that isn't a method.

Whenever these 'plugins' are reloaded (there are about 20 of them),
the memory my application uses is raised by about 3MB, as monitored
with htop. It's very consistent and always raises 2-3MB each time. The
less plugins being reloaded, the less dramatic the memory jump.

Now, my question is: can require with :reload a lot of namespaces like
this cause this sort of thing to happen? I mostly just need to know
whether or not I'm going in the wrong direction. I've never had this
sort of problem before.

If so, are there any steps I can take to keep memory from building up
like this?

[Chouser]

I'd recommend using a memory profiling tool that lists the object
counts by class. I think jvisualvm that comes with Sun's JDK
will do it, and I know yourkit will. Check your counts before
a :reload and again after -- might give a good clue as to what's
going on.

--Chouser
http://joyofclojure.com/

[Rayne]

Indeed, that I did. I ran it through jvisualvm and it's definitely
growing objects. It's odd though, because I don't see any reason why
any of the namespaces I'm reloading would do that. It's not any one
namespace, but all of them together. That's why I was thinking that it
may have possibly been some implicit weirdness with reload.

On Sep 2, 8:29 pm, Chouser <chou...@gmail.com> wrote:

[Chouser]

On Fri, Sep 3, 2010 at 6:05 AM, Rayne <discip...@gmail.com> wrote:
> Indeed, that I did. I ran it through jvisualvm and it's definitely
> growing objects. It's odd though, because I don't see any reason why
> any of the namespaces I'm reloading would do that. It's not any one
> namespace, but all of them together. That's why I was thinking that it
> may have possibly been some implicit weirdness with reload.

Which classes show the most increase in instance count? Or
perhaps you could attach or post the full list of before and
after instance counts somewhere.

--Chouser
http://joyofclojure.com/

[Rayne]

Indeed. I'm assuming you're a little familiar with yourkit, right?
Anyway, here are some snapshots I took. The first snapshot is the
application before any reloads are done. The typical state. The second
one (it has a -1 at the end of the name) is a snapshot of it after
about 6 reloads. I'm very interested to see what you get out of this.
Like I said, this stuff is greek to me.

http://acidrayne.net/files/snapshots.tar.gz

-Rayne

On Sep 3, 2:04 pm, Chouser <chou...@gmail.com> wrote:

[Stuart Sierra]

Reloading code does use memory. But the old, "dead" code will be
garbage collected eventually.

You could try to clear out stale references by calling "remove-ns"
before "require".

-S

[Rayne]

I did try that. It doesn't appear to have any effect whatsoever.

[Rayne]

As it turns out, this wasn't a memory leak at all. I decided to see if
I could max sexpbot's memory out by reloading. I got it to rise around
20-30 megs and then it stabilized and eventually jumped down 10 megs
and didn't rise again (gave up 10 reloads later). I don't know how
this stuff works, but I suppose maybe the memory just remained
reserved or something. GCing does absolutely nothing. For a smaller-
scale test, you can just try requiring clojure.java.io or some other
namespace about 100-500 times and watch memory while doing so. It'll
grow pretty substantially.

Anyway, it's doesn't appear to be a real problem at all.

[Hubert Iwaniuk]

Have you tried class unloading options of JVM?
CMSClassUnloadingEnabled and TraceClassUnloading

HTH,
Hubert.

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
>