NPTL Shared mutex failure in 32- + 64-bit applications
Guido Ostkamp
I am using 32-bit and 64-bit applications in parallel which I need to
coordinate using POSIX shared mutexes.
While this works very well with the Solaris OS, I have run into
trouble with the RedHat Enterprise Linux 5.5 (RHEL 5.5) OS, which uses
NPTL in glibc.
It appears that the posix_mutex_t struct looks very different when
compiled for 32-bit and 64-bit applications, both in overall size as
well as in the elements and especially their order.
Consequently, when creating such mutex and initializing it in shared
memory mapped by both kind of applications, I can either only use it
for 32-bit or 64-bit ones (depending on by whom it has been created)
but not for both (i.e. after creation in 32-bit application, then lock
and unlock in 32-bit works ok, but 64-bit application is unable to
lock it etc.).
I am really puzzled, why their is no ABI compatibility so that it
could be used mixed as in Solaris OS.
Does anyone know the reasons and what I can do to overcome this
problem?
I can't go back to semaphores as this will have significant negativ
impact on performance.
Any help is appreciated.
Regards
Guido
David Schwartz
bis-30-09-2...@nurfuerspam.de> wrote:
> I am using 32-bit and 64-bit applications in parallel which I need to
> coordinate using POSIX shared mutexes.
Impossible. There is no way applications can use shared memory if
they're using different libraries (or even different versions of the
same library) to access the shared memory structures. There is no way
to ensure the memory layout is the same.
> While this works very well with the Solaris OS, I have run into
> trouble with the RedHat Enterprise Linux 5.5 (RHEL 5.5) OS, which uses
> NPTL in glibc.
You mean it happens to work by luck/accident. So far as I know, there
is no reason to expect it to work.
> It appears that the posix_mutex_t struct looks very different when
> compiled for 32-bit and 64-bit applications, both in overall size as
> well as in the elements and especially their order.
Of course. You are using two completely different threading libraries.
There is no reason their internal structures should have any
similarities at all.
> Consequently, when creating such mutex and initializing it in shared
> memory mapped by both kind of applications, I can either only use it
> for 32-bit or 64-bit ones (depending on by whom it has been created)
> but not for both (i.e. after creation in 32-bit application, then lock
> and unlock in 32-bit works ok, but 64-bit application is unable to
> lock it etc.).
The same would happen if you used different version of the threading
library. And it would happen if you used a non-native threading
library in one and native in the other. Or it would happen if you use
two different non-native threading libraries. Or for a host of other
reasons.
> I am really puzzled, why their is no ABI compatibility so that it
> could be used mixed as in Solaris OS.
I'm not sure why you think of this as an ABI compatibility issue. The
memory layouts of internal library structures are not ABI issues.
They're purely internal to the library.
> Does anyone know the reasons and what I can do to overcome this
> problem?
The reason is obvious -- your two applications are using different
threading libraries, so you can't assume a 'pthread_mutex_t *' in one
application will point to something the other considers to be a
'pthread_mutex_t'.
> I can't go back to semaphores as this will have significant negative
> impact on performance.
>
> Any help is appreciated.
If you need this very unusual feature, you'll need to code it
yourself. Your expectation that it would "just work, somehow" seems
completely unreasonable to me.
DS
Marcel Müller
> I am using 32-bit and 64-bit applications in parallel which I need to
> coordinate using POSIX shared mutexes.
No way. See http://www.osnews.com/thread?59948
Look for IPC techniques. Common portable solutions simply use local sockets.
> I can't go back to semaphores as this will have significant negativ
> impact on performance.
If IPC performance is so important to you, then you should not use a mix
of 34 bit and 64 bit code. The thunking layers when using shared
resources will not improve things.
Marcel
Guido Ostkamp
> On May 27, 3:14ï¿œpm, "Guido Ostkamp" <gueltig-
> bis-30-09-2...@nurfuerspam.de> wrote:
>> I am using 32-bit and 64-bit applications in parallel which I need to
>> coordinate using POSIX shared mutexes.
>
> Impossible. There is no way applications can use shared memory if
> they're using different libraries (or even different versions of the
> same library) to access the shared memory structures. There is no
> way to ensure the memory layout is the same.
I have to disagree with that. I have been working on a multi OS
Telecommunication platform for more than a decade now, and we never
had any problems to access structures located in shared memory by
32-bit and 64-bit applications or libraries at the same time.
You just have take care to implement the necessary fillers, padding
etc. so that every structure element is located at the same binary
offset and overall size is the same, as well as that alignment fits.
Furthermore, we are not using "different" libraries here. They are
just the 32-bit and the 64-bit versions of the same GLIBC version.
With respect to this, Sun has obviously done a good job in Solaris
libpthread - I don't think at all that this happens to work just "by
chance".
The POSIX API definition of e.g. pthread_mutexattr_setpshared()
w.r.t. to PTHREAD_PROCESS_SHARED just defines:
"The process-shared attribute is set to PTHREAD_PROCESS_SHARED to
permit a mutex to be operated upon by any thread that has access to
the memory where the mutex is allocated, even if the mutex is
allocated in memory that is shared by multiple processes."
So shared mutex functionality has to work from ANY thread in ANY
process. I am not aware of any exception defined with respect to
32-bit vs. 64-bit applications.
Considering this the current NPTL implementation appears to be not
100% POSIX compliant to me.
> I'm not sure why you think of this as an ABI compatibility issue.
> The memory layouts of internal library structures are not ABI
> issues. They're purely internal to the library.
Maybe you think of ABI only in terms of backward compatibility to
former OS versions, i.e. running an application on a newer system that
has been compiled on an earlier system.
I consider structures to be part of the interfaces (as parameters of
functions or in shared memory). And those should be binary compatible
if used in the same 32-bit and 64-bit API.
I don't know what has led to the decision to mix up struct components
of pthread_mutex_t and add additional elements in 64-bit
implementation of NPTL libpthread.so. I would have expected that in
this case the same changes are also applied to the 32-bit version.
Maybe the NPTL author considered some backward compatibility to old
structs used in 32-bit on earlier NPTL or even linuxthreads
implementations to be more important than implementing full POSIX
compliance.
> If you need this very unusual feature, you'll need to code it
> yourself.
I am investigating whether it would be possible to implement a fixed
private version of NPTL (possibly with changed API and struct names)
so that this could work.
For this, I have started building the original unchanged source RPM of
GLIBC 2.5 for the RHEL 5.5. I am using. Unfortunately, although the
build went fine, I am unable to just use the produced libpthread.so
(which should have same version compared to the one located in /lib64)
as the simple test program already crashes on loading the library.
So after 'rpmbuild -bb' I grabbed only the libpthread lib, copied it
into an empty dir, setup LD_LIBRARY_PATH to the dir, verified with ldd
that test program takes the new lib, and called the test program.
According to gdb invoked on the core, it bombs out with SIGSEGV at
/* Transfer the old value from the dynamic linker's internal location. */
*__libc_dl_error_tsd () = *(*GL(dl_error_catch_tsd)) ();
This is called by __init() of the lib (so my test programs code is not
yet involved at this point).
Don't know whether you are the right one to ask, but if you (or anyone
else here) happens to know why, please drop me a note.
Regards
Guido
Guido Ostkamp
> Guido Ostkamp wrote:
>> I am using 32-bit and 64-bit applications in parallel which I need to
>> coordinate using POSIX shared mutexes.
>
> No way. See http://www.osnews.com/thread?59948
The link only points to a thread stating that with kernel 2.6.x
support for shared mutexes is now present, I could not find any
relation w.r.t. 32-bit vs. 64-bit interworking.
> If IPC performance is so important to you, then you should not use a
> mix of 34 bit and 64 bit code. The thunking layers when using shared
> resources will not improve things.
The mixed systems are unfortunately required, as we have customers
that cannot go full 64-bit for various reasons.
Regards
Guido
David Schwartz
bis-30-09-2...@nurfuerspam.de> wrote:
> > Impossible. There is no way applications can use shared memory if
> > they're using different libraries (or even different versions of the
> > same library) to access the shared memory structures. There is no
> > way to ensure the memory layout is the same.
> I have to disagree with that. I have been working on a multi OS
> Telecommunication platform for more than a decade now, and we never
> had any problems to access structures located in shared memory by
> 32-bit and 64-bit applications or libraries at the same time.
I find that extremely hard to believe, but okay.
> You just have take care to implement the necessary fillers, padding
> etc. so that every structure element is located at the same binary
> offset and overall size is the same, as well as that alignment fits.
But why would you even do that? They're two different structures in
two different libraries. Why would you be aligning them to each other?
> Furthermore, we are not using "different" libraries here. They are
> just the 32-bit and the 64-bit versions of the same GLIBC version.
Those are not only different libraries, they are different *platforms*
and completely different ABIs.
> With respect to this, Sun has obviously done a good job in Solaris
> libpthread - I don't think at all that this happens to work just "by
> chance".
Honestly, I'm stunned anyone even attempted this. There is nothing I
can imagine that would lead a reasonable person to expect it to work.
> The POSIX API definition of e.g. pthread_mutexattr_setpshared()
> w.r.t. to PTHREAD_PROCESS_SHARED just defines:
>
> "The process-shared attribute is set to PTHREAD_PROCESS_SHARED to
> permit a mutex to be operated upon by any thread that has access to
> the memory where the mutex is allocated, even if the mutex is
> allocated in memory that is shared by multiple processes."
>
> So shared mutex functionality has to work from ANY thread in ANY
> process. I am not aware of any exception defined with respect to
> 32-bit vs. 64-bit applications.
We are talking about two completely different libraries here!
Interoperation between distinct platforms is totally out of scope for
the POSIX standard. It's insane to even try it.
> Considering this the current NPTL implementation appears to be not
> 100% POSIX compliant to me.
I guess we'll have to agree to disagree. Nothing in POSIX requires
different platforms to interoperate.
> > I'm not sure why you think of this as an ABI compatibility issue.
> > The memory layouts of internal library structures are not ABI
> > issues. They're purely internal to the library.
> Maybe you think of ABI only in terms of backward compatibility to
> former OS versions, i.e. running an application on a newer system that
> has been compiled on an earlier system.
> I consider structures to be part of the interfaces (as parameters of
> functions or in shared memory). And those should be binary compatible
> if used in the same 32-bit and 64-bit API.
There is no "same 32-bit and 64-bit API". There is a 32-bit ABI and
there is a 64-bit API. They are not intended nor expected to be
compatible or similar in any way. They are different *platforms*.
> Maybe the NPTL author considered some backward compatibility to old
> structs used in 32-bit on earlier NPTL or even linuxthreads
> implementations to be more important than implementing full POSIX
> compliance.
POSIX doesn't say anything about completely different POSIX
implementations interoperating. Sharing memory between different
platforms is totally of scope. POSIX simply doesn't say anything at
all about interoperation of 32-bit and 64-bit platforms on the same
machine. It would be perfectly legal to 100% isolate them if the
implementation wanted to.
> > If you need this very unusual feature, you'll need to code it
> > yourself.
> I am investigating whether it would be possible to implement a fixed
> private version of NPTL (possibly with changed API and struct names)
> so that this could work.
Why not just implement your own shared mutexes? How much capability do
you need? Do you need condition variables? Do you need priorities? If
all you need is a shared, fast lock, it's not hard to do.
DS
Ersek, Laszlo
> On May 28, 1:58�pm, "Guido Ostkamp" <gueltig-
> bis-30-09-2...@nurfuerspam.de> wrote:
[snip]
>> The POSIX API definition of e.g. pthread_mutexattr_setpshared() w.r.t.
>> to PTHREAD_PROCESS_SHARED just defines:
>>
>> "The process-shared attribute is set to PTHREAD_PROCESS_SHARED to
>> permit a mutex to be operated upon by any thread that has access to the
>> memory where the mutex is allocated, even if the mutex is allocated in
>> memory that is shared by multiple processes."
>>
>> So shared mutex functionality has to work from ANY thread in ANY
>> process. I am not aware of any exception defined with respect to 32-bit
>> vs. 64-bit applications.
[snip]
> Nothing in POSIX requires different platforms to interoperate.
To wit,
"Conforming applications shall not attempt to link together object files
compiled for different programming models. Applications shall also be
aware that binary data placed in shared memory or in files might not be
recognized by applications built for other programming models."
http://www.opengroup.org/onlinepubs/9699919799/utilities/c99.html
[snip]
Cheers,
lacos
Marcel Müller
>> No way. See http://www.osnews.com/thread?59948
>
> The link only points to a thread stating that with kernel 2.6.x
> support for shared mutexes is now present, I could not find any
> relation w.r.t. 32-bit vs. 64-bit interworking.
Hmm, seems that something went wrong.
I wanted to show only one post:
"Pthreads can't work as an IPC mechanism because it requires that all
threads belong to the same process (the process in IPC)."
>> If IPC performance is so important to you, then you should not use a
>> mix of 34 bit and 64 bit code. The thunking layers when using shared
>> resources will not improve things.
>
> The mixed systems are unfortunately required, as we have customers
> that cannot go full 64-bit for various reasons.
However, you cannot even expect shared memory to work over 32/64 bit
boundaries flawlessly on different platforms. Usual shared memory
implementations guarantee that the virtual address is the same in all
processes. Otherwise shared libraries with code relocation won't work.
Obviously a 32 bit application cannot address virtual memory of 64 bit
code in general.
Marcel
Guido Ostkamp
>
> "Pthreads can't work as an IPC mechanism because it requires that
> all threads belong to the same process (the process in IPC)."
I am talking about _mutexes_ here, not about threads.
Shared mutexes (i.e. mutexes that reside in shared memory visible to
multiple procesess) are generally supported in Solaris and Linux.
> However, you cannot even expect shared memory to work over 32/64 bit
> boundaries flawlessly on different platforms. Usual shared memory
> implementations guarantee that the virtual address is the same in
> all processes. Otherwise shared libraries with code relocation won't
> work. Obviously a 32 bit application cannot address virtual memory
> of 64 bit code in general.
A 32-bit app is generally unable to map a shared memory segment of a
size that exceeds its available mappable address space.
However, if the shared memory segment can be mapped in 32-bit apps,
you can map the same segment in 64-bit apps as well (at least in
Solaris, and I would expect this to work in other platforms as well).
Regards
Guido
Guido Ostkamp
> On May 28, 1:58ï¿œpm, "Guido Ostkamp" <gueltig-
> bis-30-09-2...@nurfuerspam.de> wrote:
>> I have to disagree with that. I have been working on a multi OS
>> Telecommunication platform for more than a decade now, and we never
>> had any problems to access structures located in shared memory by
>> 32-bit and 64-bit applications or libraries at the same time.
>
> I find that extremely hard to believe, but okay.
We started off with 32-bit apps and libs but when systems became more
powerful and thus could handle more operations per second we also
needed to store much more data and ran out of address space in 32-bit
for selected tasks. Solaris 64-bit Support on Sparc hardware was
available, so the system got extended by a 64-bit library layer and
for selected cases 64-bit internal apps as well.
When I remember correctly, this started when Solaris 8 was current,
quite a while ago.
>> You just have take care to implement the necessary fillers, padding
>> etc. so that every structure element is located at the same binary
>> offset and overall size is the same, as well as that alignment
>> fits.
>
> But why would you even do that? They're two different structures in
> two different libraries. Why would you be aligning them to each
> other?
When you need to provide high speed multithreaded data access by
multiple processes, you have to store the data in RAM (think of
Hashes, B-Trees, Heaps etc. in overall up to Gigs of RAM) and avoid
all unnecessary syscalls.
64-bit apps are able to map the data directly into their address
space. 32-bit apps can map it up to a certain amount (until you run
out of address space).
For small amount of data, both 32-bit and 64-bit apps must be able to
map the data directly in SHM and work with it in parallel.
Check out the Solaris 64 bit developer's guide available at
<http://dlc.sun.com/pdf/816-5138/816-5138.pdf> and find section
"Interprocess Communication", on page 60 and you will see that it is
perfectly legal to have shared memory etc. interworking between 32-bit
and 64-bit apps.
And based on this I feel it is perfectly normal to expect that fast
synchronization of such data access through shared mutexes works.
> Honestly, I'm stunned anyone even attempted this. There is nothing I
> can imagine that would lead a reasonable person to expect it to
> work.
I hope I've demonstrated that this is a normal expectation.
> Why not just implement your own shared mutexes? How much capability
> do you need? Do you need condition variables? Do you need
> priorities? If all you need is a shared, fast lock, it's not hard to
> do.
I think we would need the pthread_mutex* and pthread_cond* stuff.
Seems that we need to dig into sources because NPTL fails on this.
Regards
Guido
David Schwartz
bis-30-09-2...@nurfuerspam.de> wrote:
> However, if the shared memory segment can be mapped in 32-bit apps,
> you can map the same segment in 64-bit apps as well (at least in
> Solaris, and I would expect this to work in other platforms as well).
I would imagine 32-bit apps and 64-bit apps could map memory backed by
the same file. But I wouldn't expect the 32-bit and 64-bit SysV shm
libraries to use identifiers from the same namespace. (They might, but
it's not reasonable to just expect it.)
DS
Chris Friesen
While it may not be guaranteed by the standard that 32-bit and 64-bit
apps could both safely use process-shared mutexes, it can certainly be
argued that this ability is useful.
It does seem a bit harsh to have a system comprised of many different
processes that happens to use process-shared mutexes and just because
one of the processes wants to use more than 4GB of memory the whole
system (or at least a significant part of it) needs to be redesigned.
Chris
Guido Ostkamp
> I would imagine 32-bit apps and 64-bit apps could map memory backed
> by the same file. But I wouldn't expect the 32-bit and 64-bit SysV
> shm libraries to use identifiers from the same namespace. (They
> might, but it's not reasonable to just expect it.)
Just to avoid confusion for other readers: I assume with 'identifier'
you mean to the 'shared memory key' (shmkey) and not the 'shared
memory identifier' (shmid) which you get by using shmget() API.
Because it of course plays no role whether 32-bit and 64-bit shmget()
APIs return different shmid for the same shmkey, as long as they later
refer to the same shared memory segment that can be mapped using
shmat().
There are not much commercially important un*xoid OS around. We
already discussed Solaris in that other subthread.
Check out HP-UX Interopability Whitepaper at
<http://docs.hp.com/en/938/iop.pdf>, pg. 24, and you will see that all
shared memory segments created by 32-bit apps are automatically
available for 64-bit apps. If 64-bit apps want to create a shareable
one, they need to set a flag with shmget(), that's all. Otherwise
shmkey appears to be identifying the segment as usual.
Regards
Guido
Chris Friesen
> Check out HP-UX Interopability Whitepaper at
> <http://docs.hp.com/en/938/iop.pdf>, pg. 24, and you will see that all
> shared memory segments created by 32-bit apps are automatically
> available for 64-bit apps. If 64-bit apps want to create a shareable
> one, they need to set a flag with shmget(), that's all. Otherwise
> shmkey appears to be identifying the segment as usual.
The fact that it happens to work on HP-UX and Solaris doesn't mean that
it's required by the standard...just that they've chosen to make it work
in a way that is arguably more useful than the minimum requirements.
The behaviour is implementation-defined in both cases. (And apparently
Windows does something similar where mutexes etc. can be shared between
64 and 32-bit apps.)
I've opened a bug report with glibc, it'll be interesting to see what
they say:
http://sourceware.org/bugzilla/show_bug.cgi?id=11651
Chris
David Schwartz
> It does seem a bit harsh to have a system comprised of many different
> processes that happens to use process-shared mutexes and just because
> one of the processes wants to use more than 4GB of memory the whole
> system (or at least a significant part of it) needs to be redesigned.
It sounds harsh when you put it that way, but you're leaving out the
key detail that the switch to being able to use more than 4GB of
memory requires also using a different threading library. The only
"complete redesign" required is to keep every application using the
same threading library -- which was always understood to be a
requirement of process-shared mutexes.
DS
Casper H.S. Dik
>When I remember correctly, this started when Solaris 8 was current,
>quite a while ago.
Solaris 7. ('99)
Casper
--
Expressed in this posting are my opinions. They are in no way related
to opinions held by my employer, Sun Microsystems.
Statements on Sun products included here are not gospel and may
be fiction rather than truth.
Marcel Müller
> Marcel Mï¿œller <news.5...@spamgourmet.com> wrote:
>> I wanted to show only one post:
>>
>> "Pthreads can't work as an IPC mechanism because it requires that
>> all threads belong to the same process (the process in IPC)."
>
> I am talking about _mutexes_ here, not about threads.
>
> Shared mutexes (i.e. mutexes that reside in shared memory visible to
> multiple procesess) are generally supported in Solaris and Linux.
This is an extension, that is not portable.
>> However, you cannot even expect shared memory to work over 32/64 bit
>> boundaries flawlessly on different platforms. Usual shared memory
>> implementations guarantee that the virtual address is the same in
>> all processes. Otherwise shared libraries with code relocation won't
>> work. Obviously a 32 bit application cannot address virtual memory
>> of 64 bit code in general.
>
> A 32-bit app is generally unable to map a shared memory segment of a
> size that exceeds its available mappable address space.
This is not sufficient. Also the virtual address has to be free. On 32
bit systems (without 16 bit compatibility) usually the virtual address
space is divided into a private arena and a shared arena. The size of
all together is 4GB. Usually 2GB private memory + 2GB shared memory, but
this is implementation defined and in most cases adjustable by a kernel
parameter. So any request to shared memory of 32 bit apps is taken from
the shared arena, and all shared memory objects together may not exceed
the size of the shared arena. Also fragmentation of the virtual address
space takes place. This may prevent the allocation of larger shared
memory objects if the server is running for a long time.
A 64 bit process will in general allocate the shared memory from the 64
bit shared arena, which is of course significantly larger. Otherwise the
shared memory of all 64 bit processes would be restricted to 2GB (or so)
too. So in general a 32 bit process cannot access shared memory from a
64 bit process - not even small pieces - because the virtual address of
the memory objects does not fit into 32 bit.
Of course, the memory manager may have parameters for 64 bit processes
to explicitly allocate shared memory from the 32 bit shared arena.
Sometimes this parameter is default on some platforms in the transition
period. (E.g. IBM did something like that with OS/2 to achieve 16 bit
protected mode compatibility, that only can access 1GB.)
> However, if the shared memory segment can be mapped in 32-bit apps,
> you can map the same segment in 64-bit apps as well (at least in
> Solaris, and I would expect this to work in other platforms as well).
This is no problem as long as the 32 bit shared arena is also part of
the shared arena in the 64 bit memory model. Otherwise it is impossible.
But it is a common practice to exclude the smaller memory addresses from
the use by application with a larger address space. So a 64 bit app will
not use the first 4 GB, unless there is a good reason to do so. The
compatibility to 32 bit shared memory objects may be a good reason.
E.g. the 64 bit memory layout could look like
00000000 00000000 - 00000000 7FFFFFFF := reserved
00000000 80000000 - 00000000 FFFFFFFF := 32 bit shared arena
00000001 00000000 - 7FFFFFFF FFFFFFFF := 64 bit private arena
80000000 00000000 - FFFFFFFF FFFFFFFF := 64 bit shared arena
In reality it is even more complicated. See e.g. here
http://www.tar.hu/wininternals/ch07lev1sec4.html
Marcel
Marcel Müller
> I've opened a bug report with glibc, it'll be interesting to see what
> they say:
Works as designed?
Marcel
David Schwartz
bis-30-09-2...@nurfuerspam.de> wrote:
> Just to avoid confusion for other readers: I assume with 'identifier'
> you mean to the 'shared memory key' (shmkey) and not the 'shared
> memory identifier' (shmid) which you get by using shmget() API.
> Because it of course plays no role whether 32-bit and 64-bit shmget()
> APIs return different shmid for the same shmkey, as long as they later
> refer to the same shared memory segment that can be mapped using
> shmat().
Yep. There's no reason to expect a 'key_t' to be the same size in both
the 32-bit and 64-bit APIs.
DS
Natarajan Krishnaswami
> Honestly, I'm stunned anyone even attempted this. There is nothing I
> can imagine that would lead a reasonable person to expect it to work.
Not even OS vendor documentation? :)
> We are talking about two completely different libraries here!
Two different libraries, written by the same people, often built from
the same source code, mediating access to the same kernel and kernel
resources, for which there may be esthetic or user pressure to
maintain interoperability for some subset of functionalityy (e.g,
interprocess sync objects).
The OP's view that proceses using these should be able to interoperate
isn't a priori insane. For example, it was an intentional design
point for z/OS's process shared mutexes and CV's that they permit
sharing across 64/32 (31)-bit processes. Apparently Solaris went the
same way, too.
> Nothing in POSIX requires different platforms to interoperate.
It's not a POSIX issue since (at least) one of the processes was not
built using the implementation's required config and compiler options.
If that's been made to work anyway, it's as an extension, albeit
empirically one that multiple vendors have chosen to implement.
N.
roger
> On May 28, 1:58 pm, "Guido Ostkamp" <gueltig-
>
> bis-30-09-2...@nurfuerspam.de> wrote:
> > > Impossible. There is no way applications can use shared memory if
> > > they're using different libraries (or even different versions of the
> > > same library) to access the shared memory structures. There is no
> > > way to ensure the memory layout is the same.
> > I have to disagree with that. I have been working on a multi OS
> > Telecommunication platform for more than a decade now, and we never
> > had any problems to access structures located in shared memory by
> > 32-bit and 64-bit applications or libraries at the same time.
>
> I find that extremely hard to believe, but okay.
Why is that hard to believe?
Shared memory is just memory, like any other.
The only difference is that it may appear in more than one process,
most probably at different virtual addresses. It doesn't matter
whether
you use SYSV shared memory or simply mmap(MAP_SHARED).
And it doesn't matter whether the different processes are 32-bit or
64-bit processes (assuming your OS supports both types of processes).
And you need some mechanism to protect the integrity of the shared
memory, semaphores, mutexes, rwlocks, whatever.
> > You just have take care to implement the necessary fillers, padding
> > etc. so that every structure element is located at the same binary
> > offset and overall size is the same, as well as that alignment fits.
>
> But why would you even do that? They're two different structures in
> two different libraries. Why would you be aligning them to each other?
>
> > Furthermore, we are not using "different" libraries here. They are
> > just the 32-bit and the 64-bit versions of the same GLIBC version.
>
> Those are not only different libraries, they are different *platforms*
> and completely different ABIs.
Lack of imagination, David.
> > With respect to this, Sun has obviously done a good job in Solaris
> > libpthread - I don't think at all that this happens to work just "by
> > chance".
No, it didn't happen just by chance.
From the beginning, the Solaris locking primitives were designed
to work in private or shared memory, even to be storable on stable
storage such a file that can be mmap()ed whenever you want it.
And they were designed to work in shared memory with both 32-bit
and 64-bit processes accessing them at the same time.
The Solaris implementation of Posix message queues uses
mmap()ed shared memory in this way. The same message queue
can be opened and used by any mixture of 32-bit and 64-bit
processes you care to employ. The integrity of the message queue
is maintained by process-shared semaphores and mutexes contained
in the message queue itself, in the shared memory containing the
message queue. Obviously, no pointers are stored in the shared
memory.
It is all done by indexes into the memory relative to the virtual
address
at which the message queue is mapped.
> > > If you need this very unusual feature, you'll need to code it
> > > yourself.
> > I am investigating whether it would be possible to implement a fixed
> > private version of NPTL (possibly with changed API and struct names)
> > so that this could work.
I agree with the OP here.
NPTL could use some improvement here, at least on systems
that support both 32-bit and 64-processes at the same time.
Roger Faulkner
Ersek, Laszlo
this thread.
If I understand correctly, the OP has multiple processes. Those processes
communicate through shared memory. The OP laid out his own structures so
that they are 32-bit/64-bit aware (or oblivious); the problem seems to
manifest with types declared by the GNU C library that are to be placed in
shared memory, like process-shared mutexes.
The OP mentioned in the beginning that he shuns SYSV (XSI) semaphores. I
don't seem to recall his reason for that. Why are XSI semaphores
unsuitable? That way, processes would only have to handle shared memory
structures that they have complete control over. They would get XSI
semaphores (semaphore sets) via ftok()/semget().
XSI semaphores appeared in SUSv1 (XPG4v2). A more modern interface that
might be usable for the same purpose is Realtime Semaphores, eg.
sem_open() ("initialize and open a named semaphore"), defined first in
SUSv2 (XPG5). Ulrich Drepper writes in
<http://people.redhat.com/drepper/posix-option-groups.html>, "glibc 2.3
with NPTL has full support for semaphores, including named semaphores and
inter-process semaphores".
... The "idea" behind the above is that a semaphore initialized to 1 can
double as basic (PTHREAD_MUTEX_NORMAL, non-recursive), otherwise correctly
(and "conservatively") used mutex. A very coarse correspondence list with
Realtime Semaphores (one should check the specifics, eg. interruptibility
by signals and so on):
init - sem_open() with O_CREAT and initial value 1u
lock - sem_wait()
trylock - sem_trywait()
unlock - sem_post()
timedlock - sem_timedwait()
SUSv4 XBD 4.11 "Memory Synchronization" lists all relevant XSI Semaphore
and Realtime Semaphore interfaces.
I can't offer a real substitute for process-shared condition variables
now, but a separate semaphore (initialized to 0u) might help. The
"queueing" (non-idempotent) nature of the semaphore might come in handy
even, because we don't have the atomicity of pthread_cond_wait() releasing
the mutex and blocking on the condvar; in the loop we separately have to
increment the mutex-semaphore (release the lock) and then decrement the
condvar-semaphore.
for (;;) {
sem_wait(mutex_sem);
if (predicate_ok) {
break;
}
sem_post(mutex_sem);
/* if the producer posts "cond_sem" here, nothing is lost */
sem_wait(cond_sem);
}
move_out_for_consumption();
sem_post(mutex_sem);
Due to the "queueing" nature of "cond_sem", the producer may experience a
problem with sem_post(), if there is a limit on the semaphore value and
that limit was reached earlier -- although I don't see an error condition
dedicated to that, or a reference to {SEM_VALUE_MAX} in sem_post()'s spec.
I'm not sure if such a "max value exceeded" error would be an issue
though, at least one notification would still remain pending.
Additionally, previously posted and now superfluous "cond_sem" increments
could be processed by the consumer as spurious condvar wakeups (which
can't be avoided anyway).
I won't try to invent a clunky replacement for condvar *broadcasts*.
In the end, I'm just curious why the OP excludes semaphores.
Cheers,
lacos