[ANN] clj-memory meter – measure the memory used by arbitrary objects

[Alexander Yakushev]

I'm happy to release the first version of clj-memory-meter. It's a thin wrapper around jamm which allows measuring how much space an arbitrary object occupies. clj-memory-meter is usable from the REPL, can be loaded on-demand, and doesn't require to launch JVM with special parameters.

Blog post with a little more detail and usage examples: http://clojure-goes-fast.com/blog/introspection-tool-object-memory-meter/

To start, using add [com.clojure-goes-fast/clj-memory-meter "0.1.0"], or run the following with clj:

$ clj -Sdeps "{:deps {com.clojure-goes-fast/clj-memory-meter {:mvn/version \"0.1.0\"}}}"

Clojure 1.9.0

;; Nevermind the warning, it doesn't break the library.
user=> (require '[clj-memory-meter.core :as mm])

objc[59881]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/bin/java (0x1089814c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x10d9dc4e0). One of the two will be used. Which one is undefined.

nil

user=> (mm/measure (vec (range 100000)))

"2.8 MB"

user=> (mm/measure (list 1 2 3 4 5) :debug true)

root [clojure.lang.PersistentList] 320 bytes (40 bytes)

  |

  +--_first [java.lang.Long] 24 bytes (24 bytes)

  |

  +--_rest [clojure.lang.PersistentList] 256 bytes (40 bytes)

    |

    +--_first [java.lang.Long] 24 bytes (24 bytes)

    |

    +--_rest [clojure.lang.PersistentList] 192 bytes (40 bytes)

      |

      +--_first [java.lang.Long] 24 bytes (24 bytes)

      |

      +--_rest [clojure.lang.PersistentList] 128 bytes (40 bytes)

        |

        +--_first [java.lang.Long] 24 bytes (24 bytes)

        |

        +--_rest [clojure.lang.PersistentList] 64 bytes (40 bytes)

          |

          +--_first [java.lang.Long] 24 bytes (24 bytes)

"320 B"

I hope this library will be useful for you. Enjoy!

[Matching Socks]

Unrealized lazy sequence...

user=> (def mm (filter even? (range 100000)))
#'user/mm
user=> (mm/measure mm)
"160 B"

Realized...

user=> (count mm)
50000
user=> (mm/measure mm)
"1.8 MB"

Structural sharing?

user=> (def nn (drop 25000 mm))
#'user/nn
user=> (mm/measure nn)
"1.8 MB"
user=> (mm/measure [mm nn])
"1.8 MB"

Can we free some of that memory?

user=> (def mm nil)
#'user/mm
user=> (mm/measure nn)
"1.8 MB"
user=> (System/gc)
nil
user=> (mm/measure nn)
"1.8 MB"

[Alexander Yakushev]

user=> (count nn)
25000

user=> (mm/measure nn)
"940.3 KB"

;; Still shared though!

[numb...@gmail.com]

Got error when requiring:

<pre>

clj -Sdeps "{:deps {com.clojure-goes-fast/clj-memory-meter {:mvn/version \"0.1.0\"}}}"
Clojure 1.9.0

user=> (require '[clj-memory-meter.core :as mm])

CompilerException java.lang.reflect.InvocationTargetException, compiling:(clj_memory_meter/core.clj:50:1)

user=> (mm/measure (vec (range 100000)))

CompilerException java.lang.RuntimeException: No such namespace: mm, compiling:(NO_SOURCE_PATH:2:1)
user=>

</pre>

[stardiviner]           <Hack this world!>      GPG key ID: 47C32433
IRC(freeenode): stardiviner                     Twitter:  @numbchild
Key fingerprint = 9BAA 92BC CDDD B9EF 3B36  CB99 B8C4 B8E5 47C3 2433
Blog: http://stardiviner.github.io/

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sean Corfield]

Could you try that again and immediately after getting the exception from the require, type in:

 

              *e

 

(instead of trying the mm/measure call)

 

In the exception that prints out, look for the Caused by portion of the exception. That’s going to be the underlying exception that will help us help you debug this.

 

Also, what version of Java are you using? (java -version)

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

---

From: clo...@googlegroups.com <clo...@googlegroups.com> on behalf of numb...@gmail.com <numb...@gmail.com>
Sent: Tuesday, March 6, 2018 8:38:34 PM
To: clo...@googlegroups.com
Subject: Re: [ANN] clj-memory meter – measure the memory used by arbitrary objects

 

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

[Juraj Martinka]

Works great for me. Thank you very much for publishing this!

Previously, I tried to use jol-cli but it was much more cumbersome.

[numb...@gmail.com]

Java Version:
java version "9.0.4"
Java(TM) SE Runtime Environment (build 9.0.4+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)

System: Arch Linux

<pre>

<code>

user=> (require '[clj-memory-meter.core :as mm])

CompilerException java.lang.reflect.InvocationTargetException, compiling:(clj_memory_meter/core.clj:50:1)

user=> *e
#error {
 :cause "Can not attach to current VM"
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message "java.lang.reflect.InvocationTargetException, compiling:(clj_memory_meter/core.clj:50:1)"
   :at [clojure.lang.Compiler load "Compiler.java" 7391]}
  {:type java.lang.reflect.InvocationTargetException
   :message nil
   :at [jdk.internal.reflect.NativeMethodAccessorImpl invoke0 "NativeMethodAccessorImpl.java" -2]}
  {:type java.io.IOException
   :message "Can not attach to current VM"
   :at [sun.tools.attach.HotSpotVirtualMachine <init> "HotSpotVirtualMachine.java" 75]}]
 :trace
 [[sun.tools.attach.HotSpotVirtualMachine <init> "HotSpotVirtualMachine.java" 75]
  [sun.tools.attach.VirtualMachineImpl <init> "VirtualMachineImpl.java" 56]
  [sun.tools.attach.AttachProviderImpl attachVirtualMachine "AttachProviderImpl.java" 58]
  [com.sun.tools.attach.VirtualMachine attach "VirtualMachine.java" 207]
  [jdk.internal.reflect.NativeMethodAccessorImpl invoke0 "NativeMethodAccessorImpl.java" -2]
  [jdk.internal.reflect.NativeMethodAccessorImpl invoke "NativeMethodAccessorImpl.java" 62]
  [jdk.internal.reflect.DelegatingMethodAccessorImpl invoke "DelegatingMethodAccessorImpl.java" 43]
  [java.lang.reflect.Method invoke "Method.java" 564]
  [clj_memory_meter.core$mk_vm invokeStatic "core.clj" 42]
  [clj_memory_meter.core$mk_vm invoke "core.clj" 38]
  [clj_memory_meter.core$load_jamm_agent invokeStatic "core.clj" 45]
  [clj_memory_meter.core$load_jamm_agent invoke "core.clj" 44]
  [clj_memory_meter.core$eval1822 invokeStatic "core.clj" 50]
  [clj_memory_meter.core$eval1822 invoke "core.clj" 50]
  [clojure.lang.Compiler eval "Compiler.java" 6927]
  [clojure.lang.Compiler load "Compiler.java" 7379]

  [clojure.lang.RT loadResourceScript "RT.java" 372]
  [clojure.lang.RT loadResourceScript "RT.java" 363]
  [clojure.lang.RT load "RT.java" 453]
  [clojure.lang.RT load "RT.java" 419]
  [clojure.core$load$fn__5677 invoke "core.clj" 5893]
  [clojure.core$load invokeStatic "core.clj" 5892]
  [clojure.core$load doInvoke "core.clj" 5876]
  [clojure.lang.RestFn invoke "RestFn.java" 408]
  [clojure.core$load_one invokeStatic "core.clj" 5697]
  [clojure.core$load_one invoke "core.clj" 5692]
  [clojure.core$load_lib$fn__5626 invoke "core.clj" 5737]
  [clojure.core$load_lib invokeStatic "core.clj" 5736]
  [clojure.core$load_lib doInvoke "core.clj" 5717]
  [clojure.lang.RestFn applyTo "RestFn.java" 142]
  [clojure.core$apply invokeStatic "core.clj" 648]
  [clojure.core$load_libs invokeStatic "core.clj" 5774]
  [clojure.core$load_libs doInvoke "core.clj" 5758]
  [clojure.lang.RestFn applyTo "RestFn.java" 137]
  [clojure.core$apply invokeStatic "core.clj" 648]
  [clojure.core$require invokeStatic "core.clj" 5796]
  [clojure.core$require doInvoke "core.clj" 5796]
  [clojure.lang.RestFn invoke "RestFn.java" 408]
  [user$eval1800 invokeStatic nil 1]
  [user$eval1800 invoke nil 1]
  [clojure.lang.Compiler eval "Compiler.java" 6927]
  [clojure.lang.Compiler eval "Compiler.java" 6890]
  [clojure.core$eval invokeStatic "core.clj" 3105]
  [clojure.core$eval invoke "core.clj" 3101]
  [clojure.main$repl$read_eval_print__7408$fn__7411 invoke "main.clj" 240]
  [clojure.main$repl$read_eval_print__7408 invoke "main.clj" 240]
  [clojure.main$repl$fn__7417 invoke "main.clj" 258]
  [clojure.main$repl invokeStatic "main.clj" 258]
  [clojure.main$repl doInvoke "main.clj" 174]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [clojure.tools.nrepl.middleware.interruptible_eval$evaluate$fn__5508 invoke "interruptible_eval.clj" 87]
  [clojure.lang.AFn applyToHelper "AFn.java" 152]
  [clojure.lang.AFn applyTo "AFn.java" 144]
  [clojure.core$apply invokeStatic "core.clj" 646]
  [clojure.core$with_bindings_STAR_ invokeStatic "core.clj" 1881]
  [clojure.core$with_bindings_STAR_ doInvoke "core.clj" 1881]
  [clojure.lang.RestFn invoke "RestFn.java" 425]
  [clojure.tools.nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 85]
  [clojure.tools.nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 55]
  [clojure.tools.nrepl.middleware.interruptible_eval$interruptible_eval$fn__5553$fn__5556 invoke "interruptible_eval.clj" 224]
  [clojure.tools.nrepl.middleware.interruptible_eval$run_next$fn__5548 invoke "interruptible_eval.clj" 192]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.util.concurrent.ThreadPoolExecutor runWorker "ThreadPoolExecutor.java" 1167]
  [java.util.concurrent.ThreadPoolExecutor$Worker run "ThreadPoolExecutor.java" 641]
  [java.lang.Thread run "Thread.java" 844]]}
user=>

</code>

</pre>

[stardiviner]           <Hack this world!>      GPG key ID: 47C32433
IRC(freeenode): stardiviner                     Twitter:  @numbchild
Key fingerprint = 9BAA 92BC CDDD B9EF 3B36  CB99 B8C4 B8E5 47C3 2433
Blog: http://stardiviner.github.io/

--

[Alexander Yakushev]

Looks like it's because of JDK9. I created a Github issue, let's take it there: https://github.com/clojure-goes-fast/clj-memory-meter/issues/1

[Kimmo Koskinen]

We just tried this library with a colleague while trying to track down a memory leak when parsing a jdbc result set and saw a stunning victory :)

With clj-memory-meter we were able to see that lazy parsing of json strings left references to quite large buffers (in comparison to actual data) to each item in the sequence, while strict parsing naturally removed the problem.

clj-memory-meter showed it's usefulness, so a big thanks for the library! :)

[Alexander Yakushev]

Love to hear that!