Memory mapped files and shared memory for C++

[Alexander Zaitsev]

Hello.

In 2006 Ion Gaztañaga (Boost.Interprocess and other great libraries author, email: igaztanaga at gmail dot com) wrote proposal about memory mapped files and shared memory for C++ (Document number: N2044=06-0114). But the proposal is forgotten now.

I want to reborn it, because i think it's very important thing for C++.

Please read the proposal and discuss about it.

GitHub link for reading: Github

Also the identical files attached to the topic in .pdf and .odt formats.

P.S. I think that dividing the proposal on 2 parts (memory mapped files and shared memory will be good idea).

[Thiago Macieira]

On Friday, 21 July 2017 16:26:20 PDT Alexander Zaitsev wrote:
> Hello.
>
> In 2006 Ion Gaztañaga (Boost.Interprocess and other great libraries author,
> email: igaztanaga at gmail dot com) wrote proposal about memory mapped
> files and shared memory for C++ (Document number: N2044=06-0114). But the
> proposal is forgotten now.
>
> I want to reborn it, because i think it's very important thing for C++.
>
> Please read the proposal and discuss about it.

Looks interesting, but falls short because it leaves extremely important
things out.

* needs an exception-free API

* Execute permissions are missing. Your classes support only as read-only,
read-write or invalid. But memory mapped objects have more flags, especially
the execute permission. In fact, the ability to *deny* execution is an
important security feature.

* "copy on write" is probably the wrong name for the "private" property. It's
unintuitive that it would apply to a read-only region, but it does. It
indicates whether you want to see changes made by other processes or not. I
advise changing the name back to "private" vs "shared".

* std::size_t may be the wrong type for the map size. It's valid for all
modern architectures, but in theory it just has to be wide enough to hold the
size of the biggest object. We may want instead to change the wording of this
type's definition (though note it's also in POSIX).

* Please update the address parameter to be nullptr, not 0, and update the
get_address() documentation. Do all OSes guarantee they will map where asked?

(We can probably safely assume that mapping at nullptr is not supported by the
C++ standard)

* For offset_t, you may want to use the filesystem API's offset type.

* For the flush() function, you need a way to explain whether it is needed or
not. Most memory mappings don't need it.
If you were thinking of msync(), then you're missing the arguments to that
function.

* I'm missing an equivalent of madvise() / posix_madvise()

* std::file_mapping needs a way to refer to an already-open file, without
needing to know its name (assuming it even has one). O_TMPFILE and memfd files
on Linux don't.

* what happens to the memory region if the std::file_mapping object is
destroyed? On one hand, RAII principles would say the mapping is undone; on
another, that's not how the low-level OS works and it's often useful to retain
a mapping forever, without having to keep a (polymorphic) object around that
will never, ever be used again.

* std::shared_memory_object needs to specify which type of shared memory it
is. On Unix systems, we have two: POSIX (sys/mman.h, shm_open()) and System V
(sys/shm.h, shmget()), with Linux having recently added a third (memfd). They
are completely disjoint namespaces and are identified differently.

* POSIX shared memory has the semantics of a file, including the access
permissions. Please merge with the filesystem API where they've looked into
that. I suspect Windows ones are similar.

* the System V shared memory uses a key_t object for identification, not a
"const char *name". You can use a simple narrow-character string, but you need
to provide an "int proj_id" so it can be passed through ftok(3).

* just like path names, on Windows shared memory segments are not narrow
characters. You need to provide a wchar_t overload for Windows.

* Using truncate() for growing is weird. I know the POSIX API does that, but
bad prior art is not good reason. What did the filesystem API choose? If given
the option, I'd recommend "resize".

* Please write something about what happens if you map a size larger than the
object's size. Our experience in Qt is that some systems allow this,
automatically growing the object that backs it, while others will just crash
your application. It's ok to write that it's implementation-defined, but you
need to write it.

* for that matter, please update the API with the ability to seal memory-
mappable objects. See https://lwn.net/Articles/593918/,
http://man7.org/linux/man-pages/man2/memfd_create.2.html. This was added to
Linux specifically to avoid DoS attacks where processes of different privileges
are operating on the same shared memory object. Of course, implementation-
defined whether this is supported, but it's an important security feature we
should encourage application writers to use.

* you'll want to investigate cross-process thread signalling (semaphores) too.
Eventually, we'll need to discuss std::mutex on memory-mapped regions.

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center

[Alexander Zaitsev]

Thank you for a lot of suggestions. I will think about your comment.

суббота, 22 июля 2017 г., 4:28:48 UTC+3 пользователь Thiago Macieira написал:

[Alexander Zaitsev]

Did you use before Boost.Interprocess library?

суббота, 22 июля 2017 г., 4:28:48 UTC+3 пользователь Thiago Macieira написал:

On Friday, 21 July 2017 16:26:20 PDT Alexander Zaitsev wrote:

[Thiago Macieira]

On Saturday, 22 July 2017 16:11:20 PDT Alexander Zaitsev wrote:
> Did you use before Boost.Interprocess library?

No.

[Alexander Zaitsev]

Are there any people who are interested in adding file mapping to C++ standard?

суббота, 22 июля 2017 г., 2:26:20 UTC+3 пользователь Alexander Zaitsev написал:

[Ville Voutilainen]

On 8 August 2017 at 18:20, Alexander Zaitsev <zamaz...@gmail.com> wrote:
> Are there any people who are interested in adding file mapping to C++
> standard?

If by "file mapping" you mean being able to do the equivalent of mmap,
absolutely.

>
> суббота, 22 июля 2017 г., 2:26:20 UTC+3 пользователь Alexander Zaitsev
> написал:
>>
>> Hello.
>>
>> In 2006 Ion Gaztañaga (Boost.Interprocess and other great libraries
>> author, email: igaztanaga at gmail dot com) wrote proposal about memory
>> mapped files and shared memory for C++ (Document number: N2044=06-0114). But
>> the proposal is forgotten now.
>>
>> I want to reborn it, because i think it's very important thing for C++.
>>
>> Please read the proposal and discuss about it.
>>
>> GitHub link for reading: Github
>>
>> Also the identical files attached to the topic in .pdf and .odt formats.
>>
>> P.S. I think that dividing the proposal on 2 parts (memory mapped files
>> and shared memory will be good idea).
>

> --
> You received this message because you are subscribed to the Google Groups
> "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to std-proposal...@isocpp.org.
> To post to this group, send email to std-pr...@isocpp.org.
> To view this discussion on the web visit
> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/458462c6-779f-4d26-a944-e995a3a0753f%40isocpp.org.

[Alexander Zaitsev]

Yes, i mean cross-platform "mmap". Also i suggest dividi the proposal into two parts: file_mapping and shared memory. BTW, Javascript already has shared memory support in the Standard.

вторник, 8 августа 2017 г., 18:26:35 UTC+3 пользователь Ville Voutilainen написал:

[Thiago Macieira]

On Tuesday, 8 August 2017 08:20:01 PDT Alexander Zaitsev wrote:
> Are there any people who are interested in adding file mapping to C++
> standard?

Besides the person who replied to you?

[Alexander Zaitsev]

Yes, of course. Your comment is very-very useful :-)

среда, 9 августа 2017 г., 0:53:56 UTC+3 пользователь Thiago Macieira написал:

[Niall Douglas]

Yes, I have what I intend to propose for standardisation as part of AFIO at https://ned14.github.io/afio/classafio__v2__xxx_1_1section__handle.html and https://ned14.github.io/afio/classafio__v2__xxx_1_1map__handle.html. AFIO is past one Boost peer review, it still needs to see a second peer review before it would land before WG21. But I think you'll find the design and level of abstraction is the correct one.

Niall

[Alexander Zaitsev]

Wow! AFIO looks pretty good! When AFIO will become Boost.AFIO? I think will be better if you will try to standardise file_mapping with your AFIO library.

среда, 9 августа 2017 г., 2:11:14 UTC+3 пользователь Niall Douglas написал:

[Niall Douglas]

On Thursday, August 10, 2017 at 12:24:53 AM UTC+1, Alexander Zaitsev wrote:

Wow! AFIO looks pretty good! When AFIO will become Boost.AFIO? I think will be better if you will try to standardise file_mapping with your AFIO library.

Thanks! Though most of the credit for the design goes to the Boost peer review it had in 2015.

I expect the absolute earliest it could land in the Boost peer review queue would be 2018, more likely 2019. Given every API returns a result<T>, it has zero chance until Outcome is into Boost first, and that is probably 2018 realistically speaking.

Anyway, point is there are people on the problem of standardising file mapping, but I think you'll need to be patient. Until then, AFIO is all green on its test suite on all platforms, and is ready for use.

Niall