Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

[Caml-list] using BigArray to mmap huge binary file

23 views
Skip to first unread message

Dário Abdulrehman

unread,
Nov 9, 2006, 1:34:54 PM11/9/06
to caml...@yquem.inria.fr
Hi,

I have a huge binary data file that I would like to memory map for intensive
reading (no writing) with the best possible performance.
I read some discussions about using BigArrays to achieve this in Ocaml. I
would like to know how this solution compares with the C language mmap.
Please bear in mind I am a total newbie to Ocaml but I have good Common Lisp
experience.

Thanks.

Richard Jones

unread,
Nov 9, 2006, 1:51:17 PM11/9/06
to Dário Abdulrehman

We use Bigarray to map geographical data. It works. As far as I'm
aware ArrayX.map_file is really just a wrapper around mmap, and read
accesses are just indirections on a pointer.

Depends a lot on what format your data is in. Ours is stored as a
huge array of 16-bit signed ints and Bigarray supports that naturally.
If your data was in an unsupported format, or if it wasn't strictly an
array, you might end up doing contortions.

Rich.

--
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Internet Marketing and AdWords courses - http://merjis.com/courses - NEW!
Merjis blog - http://blog.merjis.com - NEW!

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

malc

unread,
Nov 9, 2006, 2:16:07 PM11/9/06
to Richard Jones
On Thu, 9 Nov 2006, Richard Jones wrote:

> On Thu, Nov 09, 2006 at 06:30:50PM +0000, D?rio Abdulrehman wrote:
>> I have a huge binary data file that I would like to memory map for intensive
>> reading (no writing) with the best possible performance.
>> I read some discussions about using BigArrays to achieve this in Ocaml. I
>> would like to know how this solution compares with the C language mmap.
>> Please bear in mind I am a total newbie to Ocaml but I have good Common Lisp
>> experience.
>
> We use Bigarray to map geographical data. It works. As far as I'm
> aware ArrayX.map_file is really just a wrapper around mmap, and read
> accesses are just indirections on a pointer.

I beg to differ, to the best of my knowlege every reference to bigarray
is bounds checked, even with -unsafe. So, while map_file does indeed look
like a wrapper around mmap, read/write is a bit more than an indirected
memory fetch.

--
vale

0 new messages