Stubbornly eager results in clojure.java.jdbc

263 views
Skip to first unread message

Luke Burton

unread,
Jun 16, 2017, 9:15:13 PM6/16/17
to clo...@googlegroups.com

Riddle me this:


Wherein I synthesize a large table in Postgres, then attempt to lazily load the table, discarding each row as I receive it. I tried *many* permutations and experiments, but settled on these two tests to illustrate my point. Which is that I simply can't get it to work with clojure.java.jdbc.

test1, according to all my research and reading of the source code involved, should consume the query results lazily. It does not, and I can't for the life of me figure out why. Traffic starts to stream in, and the heap is overwhelmed almost immediately. I've deliberately set the heap to 1 GB.

test2 uses a technique I borrowed wholesale from Ghadi Shayban in JDBC-99, which is to have ResultSet implement IReduceInit. It consumes a nominal amount of memory. I've verified it's actually doing something by putting counters in, and using YourKit to watch about 20 MB/s of traffic streaming into the JVM. It's brilliant, it doesn't even break 200 MB total heap usage.

I used YourKit to track where the memory is being retained for test1. Initially I made the mistake of not setting the fetchSize, so I saw an ArrayList inside the driver holding the reference. The driver documentation confirms that autoCommit must be disabled and the fetchSize set to some non-zero number.

After making that change, YourKit confirmed that the GC root holding all the memory was the stack local variable "rs". At least I think it did, as a non-expert in this domain. I tried disassembling the functions using no.disassemble and the IntelliJ decompiler but I'm not really at the point where I understand what to look for.

So my questions are:

1) what am I doing wrong with clojure.java.jdbc?

Note some things I've already tried:

* using row-fn instead of result-set-fn
* using prepared statements
* explicitly setting auto-commit false on the connection
* declaring my result-set-fn with (^{:once true} *fn […]) (I did not see a change in the disassembly when using this)
* probably other things I am forgetting

2) in these situations where you suspect that the head of a lazy sequence is being retained, how do you reason about it? I'm kind of lucky this one blew the heap so quickly, who knows how much of my production code might burning memory unnecessarily but not quite as fatally. Do you disassemble the functions and observe some smoking gun? How do you peek under the covers to see where the problem is? 

Luke.

Luke Burton

unread,
Jun 19, 2017, 7:20:23 PM6/19/17
to Clojure

Anyone have any insights here? Really the most important thing I'm trying to learn is 2) how to identify when a lazy seq head is being retained, other than waiting for it to become bad enough that your program OOMs.


--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

James Reeves

unread,
Jun 19, 2017, 8:47:45 PM6/19/17
to clo...@googlegroups.com
This might be a bug in java.jdbc. The code that passes the result set seq to the function is:

(^{:once true} fn* [rset]
  ((^{:once true} fn* [rs]
     (result-set-fn (if as-arrays?
                      (cons (first rs)
                            (map row-fn (rest rs)))
                      (map row-fn rs))))
   (result-set-seq rset opts)))

I'm wondering if this function holds onto the head of the seq, since it's bound to "rs".


For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
James Reeves

Lubomir Konstantinov

unread,
Jun 20, 2017, 2:48:32 PM6/20/17
to Clojure, ja...@booleanknot.com
^{:once true}

should deal with this. Try the following simplified test case with and without it:

(defn query [result-fn]
  (let [x (for [n (range 1e6)] (make-array Object 100000)) 
        f (^:once fn* [rs] (result-fn rs))] (f x)))

(defn testq []
  (let [myfn (fn [rs] (doseq [r rs] nil))]
    (query myfn)))

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
James Reeves

r0man

unread,
Jun 23, 2017, 10:38:56 AM6/23/17
to Clojure
Hi Luke,

which database are you using? I had the same issue with MySQL recently. At the end I got it working with clojure.java.jdbc.
I don't have the code at hand, but according to the MySQL docs you have to set the fetch size to Integer.MIN_VALUE.

https://dev.mysql.com/doc/connector-j/5.1/en/connector-j-reference-implementation-notes.html

r0man

Luke Burton

unread,
Jun 23, 2017, 3:49:22 PM6/23/17
to Clojure

Postgres, as mentioned in the mail and the linked source code. The problem at this point doesn't appear to be options given to the driver, since I show two implementations using the same driver options. One processes the results lazily, one does not. 

Now, I'm calling two different methods in clojure.java.jdbc, so there's a chance the driver options aren't being propagated correctly to the implementation in one of the methods. It's already soaked up way too much of my time (and I like the elegance of the reducible-result-set alternative) so I haven't revisited the code to explore further.


Luke Burton

unread,
Jun 23, 2017, 5:03:37 PM6/23/17
to Clojure, ja...@booleanknot.com

Well, shoot. I went back and revisited this because it was bugging me … I looked at the code generated with and without usage of ^:once fn* and that led me down the right path. TL;DR – the problem appears to be locals clearing always disabled in Cursive REPL. This is just my current hypothesis based on testing and not 100% certainty. I guess I'll put this in a blog post or something once I validate this hypothesis and get some feedback from Colin (1.6.0-eap1-2017.2 if you're watching, Colin ;)

For mailing list posterity, here's the difference, using Lubomir's examples and the decompiler in IntelliJ, between ^:once fn* and regular fn (or fn*)

Note the inner-bad function:

(defn bad []
  (let [x (for [n (range)] (make-array Object 10000)), f (fn* inner-bad [] (nth x 1e6))] (f)))

The inner-bad class generated is:

public final class core$bad$inner_bad__431 extends AFunction {
  Object x;
  public static final Object const__1 = 1000000.0D;

  public core$bad$inner_bad__431(Object var1) {
    this.x = var1;
  }

  public Object invoke() {
    return RT.nth(this.x, RT.intCast((Number)const__1));
  }
}

Note the inner-good function:

(defn good []
  (let [x (for [n (range)] (make-array Object 10000)), f (^:once fn* inner-good [] (nth x 1e6))] (f)))

The inner-good class generated is:

public final class core$good$inner_good__414 extends AFunction {
  Object x;
  public static final Object const__1 = 1000000.0D;

  public core$good$inner_good__414(Object var1) {
    this.x = var1;
  }

  public Object invoke() {
    Object var10000 = this.x;
    this.x = null;
    return RT.nth(var10000, RT.intCast((Number)const__1));
  }
}

You can see pretty clearly the difference is that each time the function is invoked in the "good" case, the local reference to this.x is cleared, meaning this function really is only good for one invocation. No wonder it's undocumented!

=> (let [z 1000
      f (^:once fn* one-shot [x] (println x))
      p (f z)]
  (p)
  (p))

1000
CompilerException java.lang.NullPointerException

Anyway, I saw this and jumped to the thought: "locals!! I have fallen victim to uncleared locals!!"

So I started some tests, going back to my previous code and trying the JDBC result set that wasn't being lazy before. I put the test1 invocation into -main and tried:
  • lein run – no OOM.
  • lein repl, then invoking (-main) – no OOM.
  • lein repl :headless, then connecting via lein repl :connect "localhost:54321" and invoke (-main) – no OOM.
  • Cursive nREPL via Leiningen (locals clearing, debug REPL) – OOM!!
  • Cursive nREPL via Leiningen (no locals clearing, debug REPL) – OOM!!
  • Cursive nREPL via Leiningen (no locals clearing, regular REPL) – OOM!!
  • Cursive nREPL via Leiningen (no locals clearing, regular REPL) – OOM!!
  • lein run, with -Dclojure.compiler.disable-locals-clearing=true – OOM!!
I feel slightly better for knowing I can point the finger of blame at disabled locals clearing. It would be great to have this fixed in Cursive, considering that I run quite memory intensive applications and I'm sure locals clearing being permanently disabled has caused me to engineer around some non-existent problems. I have not tested CIDER or any other REPL, YMMV.

Luke.

Sean Corfield

unread,
Jun 23, 2017, 5:26:35 PM6/23/17
to Clojure Mailing List, ja...@booleanknot.com

This is excellent news as far as I’m concerned because it shows there’s no specific bug in clojure.java.jdbc that is fundamentally causing the OOM problem you’re seeing!

 

(that’s not to say there aren’t _other_ bugs in clojure.java.jdbc and the idea of the reducible result set definitely has appeal, which is why there’s a JIRA issue for it, but it doesn’t appear to be necessary in this instance).

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

Luke Burton

unread,
Jun 23, 2017, 6:37:38 PM6/23/17
to Clojure, ja...@booleanknot.com
On Jun 23, 2017, at 2:26 PM, Sean Corfield <se...@corfield.org> wrote:

This is excellent news as far as I’m concerned because it shows there’s no specific bug in clojure.java.jdbc that is fundamentally causing the OOM problem you’re seeing!

I'm relieved too, given that I use clojure.java.jdbc extensively :) 

(that’s not to say there aren’t _other_ bugs in clojure.java.jdbc and the idea of the reducible result set definitely has appeal, which is why there’s a JIRA issue for it, but it doesn’t appear to be necessary in this instance).

I've been trying to find a use case for the reducible result set, other than avoiding the memory issues I was experiencing, and there aren't too many that spring to mind. Probably the main one is mentioned in the JIRA, which is cleaning up clojure.java.jdbc internals. You could drop the need for sprinkling  ^:once everywhere.

One idea: as we have the constraint that regular result sets cannot be passed around, and must be consumed entirely in the context of their JDBC connection, perhaps they could be wrapped somewhat like `eduction` and represent the 'recipe' for a query. At the start of reduce/iterator it would attempt to execute the query it owns. That would be kind of handy. Such a thing could be passed around to reduce or transduce, and would implement IReduceInit and clojure.lang.Sequential (apropos post from Alex: http://insideclojure.org/2015/01/18/reducible-generators/)

Another option that achieves similar goals is simply feeding the results to a channel. I seem to code this up myself on every new project :) clojure.java.jdbc internals could handle the bookkeeping of keeping the ResultSet alive until fully consumed, at which point the channel is closed. I'm sure this isn't the first such request, and that there are probably good reasons for it not having happened as yet.

Luke.

Colin Fleming

unread,
Jun 24, 2017, 6:45:03 AM6/24/17
to clo...@googlegroups.com
It's definitely a worry that this happens consistently in the Cursive REPL, but I don't see how. I just started up a REPL in Cursive in normal and debug mode, and locals clearing does seem to be correctly initialised:



Here starting in debug mode:


​​
Your tests above also seem to indicate that this happens in the Cursive REPL whether locals clearing is enabled or not.

How are you compiling those classes? Are they being compiled by IntelliJ or in the lein process? Are you AOTing? If you start your REPL and then send the namespace to the REPL again (causing the code to be recompiled under the options set in the runtime), does that help?



For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages