finding retained head

496 views
Skip to first unread message

Brian Craft

unread,
Sep 10, 2013, 5:36:29 PM9/10/13
to clo...@googlegroups.com
From jmap output it's pretty clear I'm retaining the head of a seq somehow, but I have no idea how, or where. Is there any way to find the reference, besides staring at code waiting for enlightenment?

Brian Craft

unread,
Sep 10, 2013, 6:24:29 PM9/10/13
to clo...@googlegroups.com
Trying jhat now, reference chain here:

--> cavm.h2$load_exp$fn__165@0x2aaab4b04660 (56 bytes) (field matrix:)
--> clojure.lang.LazySeq@0x2aaab4b05388 (48 bytes) (field s:)
--> clojure.lang.Cons@0x2aaab4b0fe08 (48 bytes) (field _more:)
--> clojure.lang.LazySeq@0x2aaab4b10330 (48 bytes) (field s:)
--> clojure.lang.Cons@0x2aaab6543ec0 (48 bytes) (field _more:)

This looks to me like the head is being retained as a parameter to a function. load_exp signature is like so:

defn load-exp [file timestamp filehash matrix]

But surely passing a seq to a function doesn't retain the head?

Here's the body of the function:

(defn load-exp [file timestamp filehash matrix]
  (kdb/transaction
    (let [cid (merge-cohort file)
          exp (merge-exp file (format-timestamp timestamp) filehash cid)
          sids (load-samples cid (:samples (meta matrix)))]
      (load-exp-samples exp sids)
      (load-exp-matrix exp matrix))))


load-exp-matrix calls (dorun (map)) on matrix. There's one other reference here, where we pull the metadata. Does any of this retain the head?

Brian Craft

unread,
Sep 10, 2013, 8:24:44 PM9/10/13
to clo...@googlegroups.com
It seems to be something about the jdbc/transaction macro. If I rewrite load_exp so it does nothing but walk the seq, it still consumes the heap:

(defn load-exp [file timestamp filehash matrix]
  (jdbc/transaction
    (dorun (map identity matrix))))


If I remove the jdbc/transaction call, it's fine.

(defn load-exp [file timestamp filehash matrix]
  (dorun (map identity matrix)))

Brian Craft

unread,
Sep 10, 2013, 8:44:16 PM9/10/13
to clo...@googlegroups.com
Copying the pattern of jdbc/transaction*, I tried this:

(defn blah [func] (func))
(defn load-exp [file timestamp filehash matrix]
  (blah
    (fn [] (dorun (map identity matrix)))))


which also consumes the heap.

Can anyone explain to me what's happening here? Something about creating a anonymous function?

Armando Blancas

unread,
Sep 10, 2013, 10:34:32 PM9/10/13
to clo...@googlegroups.com
Can anyone explain to me what's happening here? Something about creating a anonymous function?

The problem is not creating the anonymous function but that it closes over the matrix argument. The closure passed on to blah will keep a reference to matrix until blah returns. This won't happen if the anonymous function takes the matrix as a parameter:

(defn blah [func v] (func v))
(defn load-exp [matrix]
  (blah
    (fn [m] (dorun (map identity m))) matrix))
 

Sean Corfield

unread,
Sep 10, 2013, 11:26:27 PM9/10/13
to clo...@googlegroups.com
FWIW, Brian and I were looking at this off-list and we changed (dorun
(map identity matrix)) to (doseq [x matrix] (identity x)) and that
seemed to work just fine - even in Brian's more complicated case.
Given that dorun specifically says it doesn't hold on to the head, I
would have expected the two to behave identically.

I also suspected the closure over the matrix argument as being the
root cause but was puzzled when using doseq instead made the problem
go away...

Sean
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

Brian Craft

unread,
Sep 10, 2013, 11:55:42 PM9/10/13
to clo...@googlegroups.com
Ah. So the root of my problem is that the jdbc/transaction macro creates a closure in my function..
(defmacro transaction

Brian Craft

unread,
Sep 11, 2013, 12:00:58 AM9/11/13
to clo...@googlegroups.com
(defmacro transaction
  [& body]
  `(transaction* (fn [] ~@body)))

I'm not sure how to avoid that. The anonymous function created here doesn't take parameters.

Armando Blancas

unread,
Sep 11, 2013, 12:40:16 AM9/11/13
to clo...@googlegroups.com
I also suspected the closure over the matrix argument as being the
root cause but was puzzled when using doseq instead made the problem
go away...


Right, it doesn't seem to be a hold in the closure, unless the compiler could tell when to release it, which is the case when the code is macro expanded (doseq) or placed inline, but not when you call a function like dorun. That's really kind of puzzling.

David Powell

unread,
Sep 11, 2013, 4:08:33 AM9/11/13
to clo...@googlegroups.com

jvisualvm has an innocuous button called "Dump Memory" or something.
You'd expect it to write out a core dump or something, but actually it opens up a GUI which lets you interactively explore all of the objects on the heap.  It is pretty amazing.  Much better than jhat, which I've found to be really flakey.

Good for finding Classloader leaks too, or just generally finding where all your memory has gone via the Compute Retained Sizes option.

Christophe Grand

unread,
Sep 11, 2013, 5:22:56 AM9/11/13
to clojure

On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft <craft...@gmail.com> wrote:
(defmacro transaction
  [& body]
  `(transaction* (fn [] ~@body)))

I'm not sure how to avoid that. The anonymous function created here doesn't take parameters.

The fix for this is:
(defmacro transaction
  [& body]
  `(transaction* (^:once fn* [] ~@body)))

It should be the default for all one-shot fns.

Christophe

--
On Clojure http://clj-me.cgrand.net/
Clojure Programming http://clojurebook.com
Training, Consulting & Contracting http://lambdanext.eu/

Brian Craft

unread,
Sep 11, 2013, 10:36:23 AM9/11/13
to clo...@googlegroups.com, chris...@cgrand.net
ugh. Can't find documentation for this. What does it do?

Christophe Grand

unread,
Sep 11, 2013, 10:39:48 AM9/11/13
to Brian Craft, clojure
^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs.

Brian Craft

unread,
Sep 11, 2013, 11:59:50 AM9/11/13
to clo...@googlegroups.com, Brian Craft, chris...@cgrand.net
This appears to have no effect on the problem. I tested with jdbc/transaction, and with Sean's simple example:

user=> (defn f [g] (g))
#'user/f
user=> (defn t1 [n c] (f (fn [] (dorun (map identity c)))))
#'user/t1
user=> (t1 0 (range 1000000))
java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0)
user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x)))))
#'user/t2
user=> (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c)))))
#'user/t1
user=> (t1 0 (range 1000000))
java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0)

Brian Craft

unread,
Sep 11, 2013, 12:00:50 PM9/11/13
to clo...@googlegroups.com, Brian Craft, chris...@cgrand.net
(started lein with LEIN_JVM_OPTS=-Xmx10M lein repl, to make it easy to see a full heap)

Alex Miller

unread,
Sep 11, 2013, 1:27:04 PM9/11/13
to clo...@googlegroups.com, Brian Craft, chris...@cgrand.net
I have nothing to add for the problem itself (sorry) but am very interested in the *process* of answering this question. Presuming there are things to document here, I would love to see someone create a wiki page (on http://dev.clojure.org/display/community/Home)  or a clojure-doc note or some place to capture tips and techniques for debugging problems like this.

Alex

Sean Corfield

unread,
Sep 11, 2013, 1:57:09 PM9/11/13
to clo...@googlegroups.com
Just to confirm, (t2 0 (range 1000000)) -- using doseq instead of
dorun -- does NOT run out of memory, correct?

Brian Craft

unread,
Sep 11, 2013, 2:20:18 PM9/11/13
to clo...@googlegroups.com
I did start with visualvm. I posted a screenshot in an earlier thread. However I'm unable to make sense of its output. jhat pointed straight to the closure with the reference. visualvm gave me a thousand cascading widgets to expand with names that were meaningless to me, none of which pointed back to the closure. I'm looking at it again now, trying to find another leak. The references for a char[] start at "String" and go back to "cache (Java frame)", with no symbol anywhere that I can relate to the program being run. I don't know what to do with this. The Strings are from another seq: lines from a file via line-seq.

Unfortunately, jhat just hangs on this object, so I have no working instrumentation.

Brian Craft

unread,
Sep 11, 2013, 2:20:46 PM9/11/13
to clo...@googlegroups.com
Correct, I forgot to paste that part. ;)

Brian Craft

unread,
Sep 11, 2013, 2:29:57 PM9/11/13
to clo...@googlegroups.com
Attaching a screenshot. Is this reference chain useful somehow? I can recognize the contents of the string from the input file. The input file is 4G, and apparently lines from line-seq are lingering, so I'm still blowing the heap.
string references.png

Christophe Grand

unread,
Sep 11, 2013, 4:45:41 PM9/11/13
to clojure
I don't get the same results:

$ LEIN_JVM_OPTS=-Xmx20M lein repl
nREPL server started on port 61221 on host 127.0.0.1
REPL-y 0.2.1
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)


user=> (defn f [g] (g))
#'user/f
user=> (defn t1 [n c] (f (fn [] (dorun (map identity c)))))
#'user/t1
user=> (t1 0 (range 1000000))

OutOfMemoryError GC overhead limit exceeded  java.lang.Long.valueOf (Long.java:577)

user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x)))))
#'user/t2
user=> (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c)))))
#'user/t1
user=>  (t1 0 (range 1000000))
nil
user=> (t2 0 (range 1000000))

OutOfMemoryError GC overhead limit exceeded  clojure.lang.ChunkBuffer.chunk (ChunkBuffer.java:29)

BUT this is because the previous OOM left the JVM in a dirty state: try to reorder your expressions:

$ LEIN_JVM_OPTS=-Xmx20M lein repl
nREPL server started on port 61245 on host 127.0.0.1
REPL-y 0.2.1
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)


user=> (defn f [g] (g))
#'user/f
user=> (defn t2 [n c] (f (fn [] (doseq [x c] (identity x)))))
#'user/t2
user=> (t2 0 (range 1000000))
nil

^^this last one failed in the previous run.

I'm not quite sure about why the doseq version works -- I would have to research a bit. My gut feeling is that doseq is based on loop and loops are lifted into ^:once fn* by the compiler in some cases. https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L5951

Brian Craft

unread,
Sep 12, 2013, 1:55:28 PM9/12/13
to clo...@googlegroups.com, chris...@cgrand.net
I think my monkey-patch of jdbc/transaction didn't take. Doing user/sourcery on it shows your version, but doing a macroexpand-all shows the original.

My code calls via korma as kdb:

=> (clojure.walk/macroexpand-all '(kdb/transaction nil))
(if (clojure.java.jdbc/find-connection) (clojure.java.jdbc/transaction* (fn* [] nil)) (clojure.java.jdbc/with-connection* (korma.db/get-connection (clojure.core/deref korma.db/_default)) (fn* ([] (clojure.java.jdbc/transaction* (fn* [] nil))))))

I thought I could get it to take by doing a (require 'clojure.java.jdbc) and applying the patch before doing a require on korma, but that doesn't seem to help.

(require 'clojure.java.jdbc)
; XXX monkey-patch jdbc to avoid head retention bug
(in-ns 'clojure.java.jdbc)
(defmacro transaction
  [& body]
  `(transaction* (^:once fn* [] ~@body)))

(ns  <blahblah> (:require [korma.db as kdb]))

.. and restart the repl, but macroexpand-all still gives me the result above.

Brian Craft

unread,
Sep 12, 2013, 2:20:11 PM9/12/13
to clo...@googlegroups.com, chris...@cgrand.net
or perhaps it's that macroexpand drops the ^:once

Brian Craft

unread,
Sep 12, 2013, 2:37:08 PM9/12/13
to clo...@googlegroups.com, chris...@cgrand.net
After patching both transaction and with-connection (which is also used by korma.db/transaction, and also creates a closure), I can pass in the seq w/o leaking.

Thanks, Christophe.

Sean Corfield

unread,
Sep 12, 2013, 3:27:01 PM9/12/13
to clo...@googlegroups.com
The latest java.jdbc snapshot (of 0.3.0) includes this fix so you can
either get it via Leiningen from the sonatype snapshots repo or git
clone it and do mvn install (or lein install since java.jdbc has a
project.clj file now).

I'll try to cut an official alpha5 release shortly once I've reviewed
the open tickets and updated the changes doc etc.

Sean
Reply all
Reply to author
Forward
0 new messages