Issue 79 in google-sparsehash: Is it possible to use memory mapping to load / save a dense hash map?

[google-s...@googlecode.com]

Status: New
Owner: ----

New issue 79 by huas...@gmail.com: Is it possible to use memory mapping to
load / save a dense hash map?
http://code.google.com/p/google-sparsehash/issues/detail?id=79

What steps will reproduce the problem?

If a dense hash map contains a large number of key-value pairs (both POD),
is it possible to use a memory mapping file to load / save it?

The data eats memory a log, and I need to load them in multiple processes.
If memory mapping is supported, the footprint can be shared between
processes, which saves a lot of memory.

I'm not familiar with dense hash map's implementation, just ask here for
whether it is possible, or someone has tried this.

Thank you in advance.

What is the expected output? What do you see instead?

What version of the product are you using? On what operating system?

Please provide any additional information below.

[google-s...@googlecode.com]

Updates:
Labels: Type-Enhancement Priority-Medium

Comment #1 on issue 79 by csilv...@gmail.com: Is it possible to use memory

mapping to load / save a dense hash map?
http://code.google.com/p/google-sparsehash/issues/detail?id=79

An interesting idea! You'd have a read-only hashtable then, is that right?

This is not possible right now. I'm wondering if you could do something
similar using shared memory (which I recognize is a lot more finicky to
work with) but even that I think would be pretty tricky, and involve
writing a custom allocator.

If you end up trying to write something like this, I'd be happy to look at
a patch! I don't know how intrusive such a change would be. In practice,
you'd probably want just to mmap the table array (in
densehashtable.h:dense_hashtable), and have each process own its own copy
of all the other data (everything else is small). Then you'd want to write
your own version of Serialize() and Unserialize() that mmap in the table or
something.

[google-s...@googlecode.com]

Comment #2 on issue 79 by huas...@gmail.com: Is it possible to use memory

mapping to load / save a dense hash map?
http://code.google.com/p/google-sparsehash/issues/detail?id=79

If the table array value type is POD (without pointer type), it's straight
forward to save the table array's memory across processes using either mmap
or shared memory.

I'll try to implement Serialize() and Unserialize() with mmap first, and if
success, put the patch here. Please help me review it then.

[google-s...@googlecode.com]

Comment #3 on issue 79 by baibaic...@gmail.com: Is it possible to use

memory mapping to load / save a dense hash map?
http://code.google.com/p/google-sparsehash/issues/detail?id=79

Hi
Have you considered to handle concurrent-write issue? Or Does it just
support read?

[google-s...@googlecode.com]

Comment #4 on issue 79 by huas...@gmail.com: Is it possible to use memory

mapping to load / save a dense hash map?
http://code.google.com/p/google-sparsehash/issues/detail?id=79

plan to support mmapped read-only densehash, no concurrent-write.