[Caml-list] memory profiling

[Christoph Bauer]

Hi,

what is the best option to do memory profiling with ocaml?
Is there a patch of ocaml-memprof for 3.11.0? What about
objsize?

Thanks,

Christoph Bauer

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[dmitry grebeniuk]

2009/5/5 Christoph Bauer <christo...@lmsintl.com>:

> what is the best option to do memory profiling with ocaml?
> Is there a patch of ocaml-memprof for 3.11.0? What about
> objsize?

If you want to use objsize with ocaml 3.11, you should get
the new version of objsize -- 0.12:
http://forge.ocamlcore.org/frs/?group_id=3
OCaml has new heap since 3.11, and old versions won't work.

objsize has an unresolved make-related problem with building
on msvc/win32 (object/library file extensions, to be specific), so
one should build objsize manually on msvc (not a hard thing.
but I'll fix it in near future).

[Sylvain Le Gall]

Hello,

On 05-05-2009, Christoph Bauer <christo...@lmsintl.com> wrote:
> Hi,
>
> what is the best option to do memory profiling with ocaml?
> Is there a patch of ocaml-memprof for 3.11.0? What about
> objsize?
>

I use a more simple approach (though I have used objsize to estimate
some datastructure size, but only in the toplevel): GC allocation rate.

You can override a little ocaml-benchmark to measure the allocation rate
of the GC. This gives you a pretty good understanding on the fact you
are allocating too much or not.

Regards,
Sylvain Le Gall

ps: here is a part of my benchmarkExt.ml file

(** Benchmark extension
@author Sylvain Le Gall
*)

open Benchmark;;

type t =
{
benchmark: Benchmark.t;
memory_used: float;
}
;;

let gc_wrap f x =
(* Extend sample to add GC stat *)
let add_gc_stat memory_used samples =
List.map
(fun (name, lst) ->
name,
List.map
(fun bt ->
{
benchmark = bt;
memory_used = memory_used;
}
)
lst
)
samples
in

(* Call throughput1 and add GC stat *)
let () =
print_string "Cleaning memory before benchmark"; print_newline ();
Gc.full_major ()
in
let allocated_before =
Gc.allocated_bytes ()
in
let samples =
f x
in
let () =
print_string "Cleaning memory after benchmark"; print_newline ();
Gc.full_major ()
in
let memory_used =
((Gc.allocated_bytes ()) -. allocated_before)
in
add_gc_stat memory_used samples
;;

let throughput1
?min_count ?style
?fwidth ?fdigits
?repeat ?name
seconds
f x =

(* Benchmark throughput1 as it should be called *)
gc_wrap
(throughput1
?min_count ?style
?fwidth ?fdigits
?repeat ?name
seconds f) x
;;

let throughputN
?min_count ?style
?fwidth ?fdigits
?repeat
seconds name_f_args =
List.flatten
(List.map
(fun (name, f, args) ->
throughput1
?min_count ?style
?fwidth ?fdigits
?repeat ~name:name
seconds f args)
name_f_args)
;;

let latency1
?min_cpu ?style
?fwidth ?fdigits
?repeat n
?name f x =
gc_wrap
(latency1
?min_cpu ?style
?fwidth ?fdigits
?repeat n
?name f) x
;;

let latencyN
?min_cpu ?style
?fwidth ?fdigits
?repeat
n name_f_args =
List.flatten
(List.map
(fun (name, f, args) ->
latency1
?min_cpu ?style
?fwidth ?fdigits
?repeat ~name:name
n f args)
name_f_args)
;;

[Jon Harrop]

On Tuesday 05 May 2009 15:39:21 Sylvain Le Gall wrote:
> Hello,
>
> On 05-05-2009, Christoph Bauer <christo...@lmsintl.com> wrote:
> > Hi,
> >
> > what is the best option to do memory profiling with ocaml?
> > Is there a patch of ocaml-memprof for 3.11.0? What about
> > objsize?
>
> I use a more simple approach (though I have used objsize to estimate
> some datastructure size, but only in the toplevel): GC allocation rate.
>
> You can override a little ocaml-benchmark to measure the allocation rate
> of the GC. This gives you a pretty good understanding on the fact you
> are allocating too much or not.

FWIW, I automated the same approach using camlp4 and intend to write it up in
the OCaml Journal. The results were quite good although run-time performance
was seriously degraded (~50x slower) and it would have been nice to visualize
the results in an IDE.

--
Dr Jon Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?e

[Jon Harrop]

On Tuesday 05 May 2009 13:45:18 dmitry grebeniuk wrote:
> 2009/5/5 Christoph Bauer <christo...@lmsintl.com>:
> > what is the best option to do memory profiling with ocaml?
> > Is there a patch of ocaml-memprof for 3.11.0? What about
> > objsize?
>
> If you want to use objsize with ocaml 3.11, you should get
> the new version of objsize -- 0.12:
> http://forge.ocamlcore.org/frs/?group_id=3

> OCaml has new heap since 3.11...

Can anyone elaborate on this?

--
Dr Jon Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?e

_______________________________________________

[Richard Jones]

On Sat, Jan 09, 2010 at 01:25:52PM +0000, Jon Harrop wrote:
> On Tuesday 05 May 2009 13:45:18 dmitry grebeniuk wrote:
> > 2009/5/5 Christoph Bauer <christo...@lmsintl.com>:
> > > what is the best option to do memory profiling with ocaml?
> > > Is there a patch of ocaml-memprof for 3.11.0? What about
> > > objsize?
> >
> > If you want to use objsize with ocaml 3.11, you should get
> > the new version of objsize -- 0.12:
> > http://forge.ocamlcore.org/frs/?group_id=3
> > OCaml has new heap since 3.11...
>
> Can anyone elaborate on this?

Not sure about "new heap", but the way that heap pages are tracked
changed from 3.10 -> 3.11. In 3.10 a flat bitmap was used. This was
unsuitable for 64 bit address spaces[1] and in 3.11 a sparse structure
is used (a hash table).

Rich.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9

--
Richard Jones
Red Hat