[boost] interprocess with unordered_map
Joel Young
Folks,
I've built a unordered_map in a memory mapped file. The map goes from:
typedef boost::interprocess::allocator<
char,
boost::interprocess::managed_mapped_file::segment_manager
> char_allocator_t;
typedef boost::interprocess::basic_string<
char,
std::char_traits<char>,
char_allocator_t
>
typedef std::pair< key_type, pay_type> val_type;
typedef boost::interprocess::allocator<
val_type,
boost::interprocess::managed_mapped_file::segment_manager> allocator_t;
typedef boost::unordered_map<
key_type,
pay_type,
boost::hash<key_type>,
std::equal_to<key_type>,
allocator_t> map_type;
Everything works fine on creation. But when I later go to read from the map in
a different program where the mapped file is opened read only, I have to do
gymcrackery like:
// A temporary string that allocates from the mapped file
struct shm_clean { // cleanup shared memory on stack-unwind
shm_clean() {
boost::interprocess::shared_memory_object::remove("StupidSharedMemory");
}
~shm_clean() {
boost::interprocess::shared_memory_object::remove("StupidSharedMemory");
}
} cleaner;
boost::interprocess::managed_shared_memory
stupid(boost::interprocess::create_only ,"StupidSharedMemory" ,500);
key_type key_val(stupid.get_segment_manager());
to create a temporary to allow me to search since the maps find() method will
only take a key_type and interprocess::basic_string WILL NOT implicitly
construct from a std::string, nor will it implicitly construct from a char* to
itself.
Can the interprocess::basic_string be modified to allow these conversions?
I hate having to do:
key_val = "fruitcake";
db.find(key_val);
Suggestions?
Thanks!
Joel
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Ion Gaztañaga
> Can the interprocess::basic_string be modified to allow these conversions?
>
> I hate having to do:
>
> key_val = "fruitcake";
> db.find(key_val);
interprocess::basic_string is now boost::container::basic_string, which
follows std::basic_string interface. And you always need an allocator to
place the string in shared memory. I don' think it could be implicitly
constructible from char* or std::string.
Ion
Olaf van der Spek
> El 15/03/2012 3:14, Joel Young escribió:
>>
>> Can the interprocess::basic_string be modified to allow these conversions?
>>
>> I hate having to do:
>>
>> key_val = "fruitcake";
>> db.find(key_val);
>
>
> interprocess::basic_string is now boost::container::basic_string, which
> follows std::basic_string interface. And you always need an allocator to
> place the string in shared memory. I don' think it could be implicitly
> constructible from char* or std::string.
But there's no (good) reason for the input to be required to be in
shared memory.
--
Olaf
Ion Gaztañaga
> 2012/3/15 Ion Gaztañaga<igazt...@gmail.com>:
>> El 15/03/2012 3:14, Joel Young escribió:
>>>
>>> Can the interprocess::basic_string be modified to allow these conversions?
>>>
>>> I hate having to do:
>>>
>>> key_val = "fruitcake";
>>> db.find(key_val);
>>
>>
>> interprocess::basic_string is now boost::container::basic_string, which
>> follows std::basic_string interface. And you always need an allocator to
>> place the string in shared memory. I don' think it could be implicitly
>> constructible from char* or std::string.
>
> But there's no (good) reason for the input to be required to be in
> shared memory.
A shared memory allocator is designed to allocate from a managed shared
memory segment. And this requires a base address of that shared memory
segment . And even with std::string you need to build a temporary string
to compare it with the ones stored in the container.
Do you have any proposal or alternative idea?
Ion
Olaf van der Spek
> El 15/03/2012 23:18, Olaf van der Spek escribió:
>>
>> 2012/3/15 Ion Gaztañaga<igazt...@gmail.com>:
>>>
>>> El 15/03/2012 3:14, Joel Young escribió:
>>>>
>>>>
>>>> Can the interprocess::basic_string be modified to allow these
>>>> conversions?
>>>>
>>>> I hate having to do:
>>>>
>>>> key_val = "fruitcake";
>>>> db.find(key_val);
>>>
>>>
>>>
>>> interprocess::basic_string is now boost::container::basic_string, which
>>> follows std::basic_string interface. And you always need an allocator to
>>> place the string in shared memory. I don' think it could be implicitly
>>> constructible from char* or std::string.
>>
>>
>> But there's no (good) reason for the input to be required to be in
>> shared memory.
>
>
> A shared memory allocator is designed to allocate from a managed shared
> memory segment. And this requires a base address of that shared memory
> segment . And even with std::string you need to build a temporary string to
> compare it with the ones stored in the container.
>
> Do you have any proposal or alternative idea?
Yes, do a direct compare between your key type and const char*
--
Olaf
Joel
>
> El 15/03/2012 23:18, Olaf van der Spek escribió:
> > 2012/3/15 Ion Gaztañaga<igaztanaga <at> gmail.com>:
> >> El 15/03/2012 3:14, Joel Young escribió:
> >>>
> >>> Can the interprocess::basic_string be modified to allow these conversions?
> >>>
> >>> I hate having to do:
> >>>
> >>> key_val = "fruitcake";
> >>> db.find(key_val);
> >>
> >> place the string in shared memory. I don' think it could be implicitly
> >> constructible from char* or std::string.
My code above used the assignment operator to convert it. The constructor can
do it also. If not, then at least provide a convenience function to do it!
> > But there's no (good) reason for the input to be required to be in
> > shared memory.
Exactly. I've got my container in the mapped file or shared memory. Why should
I want to clutter up the shared memory with a temporary? Especially if I'm just
doing a find in a read-only chunk of memory!
> A shared memory allocator is designed to allocate from a managed shared
> memory segment. And this requires a base address of that shared memory
> segment . And even with std::string you need to build a temporary string
> to compare it with the ones stored in the container.
But the temporary string isn't used for anything! I create a temporary string,
allocated into a DIFFERENT memory pool and it works just fine. If I can create
an arbitrary basic_string with an allocator to a random place to serve as a
temporary, the library can darn well do it too!
Joel
Ion Gaztañaga
>> Do you have any proposal or alternative idea?
>
> Yes, do a direct compare between your key type and const char*
I must be missing something, as I think that comparison must be done by
boost::unordered.
Ion
Ion Gaztañaga
Ok, but this is a different issue. We still need to create a temporary
string (and allocate dynamic memory, in shared memory, heap, etc... ) so
you don't get any free comparison. Your original statement was:
"I have to do gymcrackery like:
key_type key_val(stupid.get_segment_manager());
to create a temporary to allow me to search since the maps find() method
will only take a key_type and interprocess::basic_string WILL NOT
implicitly construct from a std::string, nor will it implicitly
construct from a char* to itself.
Can the interprocess::basic_string be modified to allow these conversions?"
No basic_string change will change the fact that you need to create a
temporary (explicit or implicit) to find a value, as unordered requires
it. Changing basic_string with implicit conversions to save some
keystrokes will do much more harm than good. And you can reuse that
explicit temporary if you search several keys in a loop obtaining much
better performance.
If we want to achieve direct comparison, Boost.Unordered needs to go
Boost.Intrusive or Boost.Multiindex route, and offer additional search
overloads that can find "equivalent keys":
http://www.boost.org/doc/libs/1_49_0/libs/multi_index/doc/reference/ord_indices.html#set_operations
I think the annoyance of writing a temporary string is much better than
changing boost::container::basic_string (which knows nothing about
shared memory as it's an std::string-like class) with conversions that
will bring a lot of overload problems.
The only point that I consider negative is that we need to allocate
memory in a shared memory segment to do a comparison. Maybe we have no
write access to shared memory (open_read_only flag) and having to
allocate memory will ruin this advantage.
A solution for this is to find a way to explicitly build an special
temporary of type basic_string< T, boost::interprocess::allocator<> >
that will hold the string in heap. I won't make this implicit (say, a
default constructed boost::interprocess::allocator) as otherwise tons of
new users will start having problems thinking they've placed a string in
shm when they are actually building it in the heap).
Ion
Olaf van der Spek
> El 17/03/2012 5:53, Joel escribió:
> If we want to achieve direct comparison, Boost.Unordered needs to go
> Boost.Intrusive or Boost.Multiindex route, and offer additional search
> overloads that can find "equivalent keys":
>
> http://www.boost.org/doc/libs/1_49_0/libs/multi_index/doc/reference/ord_indices.html#set_operations
>
> http://www.boost.org/doc/libs/1_49_0/doc/html/intrusive/advanced_lookups_insertions.html#intrusive.advanced_lookups_insertions.advanced_lookups
That route looks like a good one. Avoiding the need for expensive
temps is great.
--
Olaf
Joel
> > If we want to achieve direct comparison, Boost.Unordered needs to go
> > Boost.Intrusive or Boost.Multiindex route, and offer additional search
> > overloads that can find "equivalent keys":
Olaf van der Spek <ml <at> vdspek.org> writes:
> That route looks like a good one. Avoiding the need for expensive
> temps is great.
Perhaps map and flat_map need this as well? Should bugs be filed?
Thanks!
Joel
Olaf van der Spek
>> 2012/3/17 Ion Gaztañaga <igaztanaga <at> gmail.com>:
>> > If we want to achieve direct comparison, Boost.Unordered needs to go
>> > Boost.Intrusive or Boost.Multiindex route, and offer additional search
>> > overloads that can find "equivalent keys":
>
> Olaf van der Spek <ml <at> vdspek.org> writes:
>> That route looks like a good one. Avoiding the need for expensive
>> temps is great.
>
> Perhaps map and flat_map need this as well? Should bugs be filed?
All find() might be affected, but it may be a conscious design decision.
You could file bugs and find out.
--
Olaf
Ion Gaztañaga
>> 2012/3/17 Ion Gaztañaga<igaztanaga<at> gmail.com>:
>>> If we want to achieve direct comparison, Boost.Unordered needs to go
>>> Boost.Intrusive or Boost.Multiindex route, and offer additional search
>>> overloads that can find "equivalent keys":
>
> Olaf van der Spek<ml<at> vdspek.org> writes:
>> That route looks like a good one. Avoiding the need for expensive
>> temps is great.
>
> Perhaps map and flat_map need this as well? Should bugs be filed?
Yes, please. At least in Boost.Container, as those containers are based
on Boost.Intrusive, implementation should be pretty easy.
Best,
Ion
Joel
> El 18/03/2012 21:07, Joel escribió:
> > Perhaps map and flat_map need this as well? Should bugs be filed?
> Yes, please. At least in Boost.Container, as those containers are based
> on Boost.Intrusive, implementation should be pretty easy.
I went ahead and filed.
Containers:
https://svn.boost.org/trac/boost/ticket/6710
Interprocess:
https://svn.boost.org/trac/boost/ticket/6711
For the interprocess one, I messed up and didn't specify the right component.
I'm not seeing how to changed it as I don't have an account.
Joel
Ion Gaztañaga
>
>> El 18/03/2012 21:07, Joel escribió:
>>> Perhaps map and flat_map need this as well? Should bugs be filed?
>
> Ion Gaztañaga<igaztanaga<at> gmail.com> writes:
>> Yes, please. At least in Boost.Container, as those containers are based
>> on Boost.Intrusive, implementation should be pretty easy.
>
> I went ahead and filed.
>
> Containers:
> https://svn.boost.org/trac/boost/ticket/6710
>
> Interprocess:
> https://svn.boost.org/trac/boost/ticket/6711
>
> For the interprocess one, I messed up and didn't specify the right component.
> I'm not seeing how to changed it as I don't have an account.
Ok, thanks.
Ion
Joel
> Ion Gaztañaga <igaztanaga <at> gmail.com> writes:
> > Yes, please. At least in Boost.Container, as those containers are based
> > on Boost.Intrusive, implementation should be pretty easy.
>
> I went ahead and filed.
>
> Containers:
> https://svn.boost.org/trac/boost/ticket/6710
>
> Interprocess:
> https://svn.boost.org/trac/boost/ticket/6711
Ticket 6711 was closed "won't fix"
https://svn.boost.org/trac/boost/ticket/6711 :
> Closing as "won't fix", since boot unordered is an implementation
> of the standard unordered containers, and this would be a break
> from the standard. It would also be worse when using std::equal_to
> or boost::hash since it would have to cast for every call to those
> function objects rather than just once for the call to operator[].
Perhaps the interprocess docs can be updated to show how to create a simple
allocator from the heap for easy creation of these temporary copiers and how to
cleanly provide the thunking between "normal" equivalent types (e.g. strings)
and those used in memory regions (basic_string).
Ion Gaztañaga
> Perhaps the interprocess docs can be updated to show how to create a simple
> allocator from the heap for easy creation of these temporary copiers and how to
> cleanly provide the thunking between "normal" equivalent types (e.g. strings)
> and those used in memory regions (basic_string).
Interprocess allocators would require changes to support that and I need
to think about it, but I definitely think it should ease this task, as
it is demanded by users.
Thanks for your reports and comments,
Ion
Olaf van der Spek
> El 21/03/2012 22:56, Joel escribió:
>
>> Perhaps the interprocess docs can be updated to show how to create a
>> simple
>> allocator from the heap for easy creation of these temporary copiers and
>> how to
>> cleanly provide the thunking between "normal" equivalent types (e.g.
>> strings)
>> and those used in memory regions (basic_string).
>
>
> Interprocess allocators would require changes to support that and I need to
> think about it, but I definitely think it should ease this task, as it is
> demanded by users.
There's still no technical reason to require this copy, is there?
Can't we come up with a solution that avoids this copy?
--
Olaf
Ion Gaztañaga
> 2012/3/23 Ion Gaztañaga<igazt...@gmail.com>:
>> El 21/03/2012 22:56, Joel escribió:
>>
>>> Perhaps the interprocess docs can be updated to show how to create a
>>> simple
>>> allocator from the heap for easy creation of these temporary copiers and
>>> how to
>>> cleanly provide the thunking between "normal" equivalent types (e.g.
>>> strings)
>>> and those used in memory regions (basic_string).
>>
>>
>> Interprocess allocators would require changes to support that and I need to
>> think about it, but I definitely think it should ease this task, as it is
>> demanded by users.
>
> There's still no technical reason to require this copy, is there?
> Can't we come up with a solution that avoids this copy?
Without changing the container the copy (either implicit or explicit) is
unavoidable. In those cases, maybe we can avoid a shared memory
allocation (which will be always slower than a heap allocation) for that
copy.
Ion
Olaf van der Spek
> El 23/03/2012 12:57, Olaf van der Spek escribió:
>>
>> 2012/3/23 Ion Gaztañaga<igazt...@gmail.com>:
>>>
>>> El 21/03/2012 22:56, Joel escribió:
>>>
>>>> Perhaps the interprocess docs can be updated to show how to create a
>>>> simple
>>>> allocator from the heap for easy creation of these temporary copiers and
>>>> how to
>>>> cleanly provide the thunking between "normal" equivalent types (e.g.
>>>> strings)
>>>> and those used in memory regions (basic_string).
>>>
>>>
>>>
>>> Interprocess allocators would require changes to support that and I need
>>> to
>>> think about it, but I definitely think it should ease this task, as it is
>>> demanded by users.
>>
>>
>> There's still no technical reason to require this copy, is there?
>> Can't we come up with a solution that avoids this copy?
>
>
> Without changing the container the copy (either implicit or explicit) is
> unavoidable. In those cases, maybe we can avoid a shared memory allocation
> (which will be always slower than a heap allocation) for that copy.
What's stopping us from changing the container?
--
Olaf