[SciPy-user] shared memory machines
Gideon Simpson
scipy? How did you do it?
-gideon
_______________________________________________
SciPy-user mailing list
SciPy...@scipy.org
http://projects.scipy.org/mailman/listinfo/scipy-user
Gael Varoquaux
> Has anyone been able to take advantage of shared memory machines with
> scipy? How did you do it?
I am not sure I understand your question. You want to do parallel
computing and share the arrays between processes, is that it?
Gaël
Gideon Simpson
system, not a commodity cluster. In these shared memory
configurations, were I using compiled code, I'd be able to use OpenMP
to take advantage of the additional cores/processors. I'm wondering
if anyone has looked at ways to take advantage of such configurations
with scipy.
-gideon
Gael Varoquaux
> Yes, but I'm talking about when you have a multiprocessor/multicore
> system, not a commodity cluster. In these shared memory
> configurations, were I using compiled code, I'd be able to use OpenMP
> to take advantage of the additional cores/processors. I'm wondering
> if anyone has looked at ways to take advantage of such configurations
> with scipy.
I use the multiprocessing module:
http://docs.python.org/library/multiprocessing.html
I also have some code to share arrays between processes. I'd love to
submit it for integration with numpy, but first I'd like it to get more
exposure so that the eventual flaws in the APIs are found. I am attaching
it.
Actually I wrote this code a few months ago, and now that I am looking at
it, I realise that the SharedMemArray should probably be a subclass of
numpy.ndarray, and implement the full array signature. I am not sure if
this is possible or not (ie if it will still be easy to have
multiprocessing share the data between processes or not). I don't really
have time for polishing this right, anybody wants to have a go?
Gaël
Sturla Molden
> scipy? How did you do it?
I have either used OpenML in C or Fortran 90 extension modules, or
multiprocessing in Python. If you have lengthy calculations in extension
libraries you can also use Python threads, given that your extension
releases the GIL.
I have been working on a multiprocessing + NumPy cookbook tutorial. For
now the unfinished draft is here:
http://folk.uio.no/sturlamo/python/multiprocessing-tutorial.pdf
It only covers shared memory programming, though. I will also add how to
use Queues for message-passing. Many prefer message-passing to shared
memory. You avoid performance problems due to 'false sharing', and there
is often less resource contention.
The difference betwwen threading and multiprocessing should also be better
covered. Both are applicable, but in defferent contexts. And with
threading you can also choose between 'shared-objects' and message-passing
(Queue.Queue).
Sturla Molden
Sturla Molden
> Actually I wrote this code a few months ago, and now that I am looking at
> it, I realise that the SharedMemArray should probably be a subclass of
> numpy.ndarray, and implement the full array signature. I am not sure if
> this is possible or not (ie if it will still be easy to have
> multiprocessing share the data between processes or not). ¨
You can use multiprocessing.Array to allocate shared memory, and use its
buffer to create an ndarray with numpy.frombuffer.
Basically multiprocessing can use whatever can be pickled. ndarrays copy
their contents when pickled, and subclasses seem to inherit this
behaviour. Note that this is perfectly okay if you are happy with a
message-passing approach to parallel computing.
When using mp.Array as shared memory, the object must be passed to
multiprocessing.Process on instantiation. This is because of handle
inheritance. Therefore you cannot pass an instance of mp.Array through a
mp.Queue or mp.Pipe. If we use named share memory (System V IPC) instead
of BSD mmap, we can probably get around this. But support for this li
lacking in Python and SciPy.
Sturla Molden
Gael Varoquaux
> I have been working on a multiprocessing + NumPy cookbook tutorial. For
> now the unfinished draft is here:
> http://folk.uio.no/sturlamo/python/multiprocessing-tutorial.pdf
Hey, it's a very interested document. It seems that you have quite a lot
of insight on these problems.
I hadn't realized that a numpy array with the memory alocated as shared
memory would be automaticaly shared by multiprocessing (I tried, and to
my surprise, it works).
So it seems that shmem_as_ndarray (the implementation of which is fairly
similar in your code and in mine), and probably probably some array
creation helper like empty_shmem, is all we need to use multiprocessing
with numpy. Do you concur?
I also like a lot your code to figure out the number of processor. It is
very useful in a multiprocessing scientific computing package. However my
limitation is more often than not memory. Do you have cross platform code
to analyse the percent of memory used, and the absolute amount of memory
free?
I think I should write empty_shmem, to complete hide the multiprocessing
Array, delete my useless SharedMemArray class, integrate your number of
processor function, and recirculate my code, if it is OK with you. In a
few iterations we can propose this for integration in numpy.
Cheers,
Gaël
Robert Kern
<gael.va...@normalesup.org> wrote:
> I think I should write empty_shmem, to complete hide the multiprocessing
> Array, delete my useless SharedMemArray class, integrate your number of
> processor function, and recirculate my code, if it is OK with you. In a
> few iterations we can propose this for integration in numpy.
Here's mine, FWIW. It goes down directly to the multiprocessing.heap
code underlying the Array stuff. On Windows, the objects transfer via
pickle while under UNIX, they must be inherited. Windows mmap objects
can be pickled while UNIX mmap objects can't. Like Sturla says, we'd
have to use named shared memory to get around this on UNIX.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
Gael Varoquaux
> On Mon, Feb 2, 2009 at 00:38, Gael Varoquaux
> <gael.va...@normalesup.org> wrote:
> > I think I should write empty_shmem, to complete hide the multiprocessing
> > Array, delete my useless SharedMemArray class, integrate your number of
> > processor function, and recirculate my code, if it is OK with you. In a
> > few iterations we can propose this for integration in numpy.
> Here's mine, FWIW. It goes down directly to the multiprocessing.heap
> code underlying the Array stuff. On Windows, the objects transfer via
> pickle while under UNIX, they must be inherited. Windows mmap objects
> can be pickled while UNIX mmap objects can't. Like Sturla says, we'd
> have to use named shared memory to get around this on UNIX.
Well, you know way more than I do about this. But I fear I am
miss-understanding something. Does what you are saying means that an
'empty_shmem', that would create a multiprocessing Array, and expose it
as a numpy array, is bound to fail under windows? My experiments seem to
show that this works under Linux, and this would be a very simple way of
doing shared memory. We could have a numpy.multiprocessing, with all kind
of constructors for arrays (empty, zeros, ones, *_like, and maybe
'array') that would be shared between process.
Am I out of my mind, and will this fail utterly?
Sturla Molden
> Well, you know way more than I do about this. But I fear I am
> miss-understanding something. Does what you are saying means that an
> 'empty_shmem', that would create a multiprocessing Array, and expose it
> as a numpy array, is bound to fail under windows?
Linux:
You can create shared memory using BSD mmap or System V IPC.
Multiprocessing does the former. Shared memory created via BSD mmap is
"unnamed". Thus, it has to be created in the parent prior prior to the
call to fork(), otherwise the child cannot get access to it. That is why
mp.Array must be created prior to mp.Process (the latter calls os.fork).
Windows:
There is no fork(). Shared memory can be named or unnamed. In the second
case, it is passed to the spawned process via handle inheritance. This is
what multiprocessing does. Again, the consequence is that it must med
created prior to the creation of mp.Process. In this case it must actually
be passed as an argument to to mp.Process when it is instantiated.
However:
If we had an ndarray that used named shared memory as buffer, it would be
more convinient on Windows and Linux alike. Any process can map this
segment if it knows its name. It would only pickle the name of the shared
segment (as well as dtype and shape), and could thus be messaged between
processes using mp.Queue. Currently we can only send copies of private
memory arrays via mp.Queue.
> My experiments seem to
> show that this works under Linux, and this would be a very simple way of
> doing shared memory. We could have a numpy.multiprocessing, with all kind
> of constructors for arrays (empty, zeros, ones, *_like, and maybe
> 'array') that would be shared between process.
>
> Am I out of my mind, and will this fail utterly?
It will work. But we should use named shared memory (which requires some C
or Cython coding), not BSD mmap as mp.Array currently does. Also we must
override how ndarrays are pickled.
Sturla Molden
Robert Kern
<gael.va...@normalesup.org> wrote:
> On Mon, Feb 02, 2009 at 12:51:51AM -0600, Robert Kern wrote:
>> On Mon, Feb 2, 2009 at 00:38, Gael Varoquaux
>> <gael.va...@normalesup.org> wrote:
>> > I think I should write empty_shmem, to complete hide the multiprocessing
>> > Array, delete my useless SharedMemArray class, integrate your number of
>> > processor function, and recirculate my code, if it is OK with you. In a
>> > few iterations we can propose this for integration in numpy.
>
>> Here's mine, FWIW. It goes down directly to the multiprocessing.heap
>> code underlying the Array stuff. On Windows, the objects transfer via
>> pickle while under UNIX, they must be inherited. Windows mmap objects
>> can be pickled while UNIX mmap objects can't. Like Sturla says, we'd
>> have to use named shared memory to get around this on UNIX.
>
> Well, you know way more than I do about this. But I fear I am
> miss-understanding something. Does what you are saying means that an
> 'empty_shmem', that would create a multiprocessing Array, and expose it
> as a numpy array, is bound to fail under windows?
[These first two paragraphs are basically what Sturla says in his
response. He's faster on the Send button than I am. :-)]
Almost. On Windows, the subprocesses inherit nothing. All objects must
be passed through pickles. Passing the Array works, but passing the
ndarray won't because the ndarray pickler will pass-by-value. My
approach registers a new pickler for ndarrays that recognizes my
shared-memory ndarrays and makes a pickle that just references the
shared memory. You could replicate that using Array as the memory
allocator, but I think my approach which uses the "raw" allocators
underneath Array is more straightforward.
On UNIX, Arrays and the stuff underneath it don't pickle because the
underlying mmap is not named. We'd need to wrap the appropriate APIs
in order to do this. If you can arrange your program such that the
arrays get inherited, you're fine because you don't need to pickle
anything, but you can't pass these ndarrays through Queues and such.
I've tried using the shm module, which does wrap those APIs, but I've
never been able to get the memory to actually share unless if the
subprocess inherits it.
http://nikitathespider.com/python/shm/
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
Sturla Molden
> Almost. On Windows, the subprocesses inherit nothing. All objects must
> be passed through pickles. Passing the Array works, but passing the
> ndarray won't because the ndarray pickler will pass-by-value.
Almost. A subprocess can be specified to inherit its parent's handles. The
parent must then pass the value of the handle to the subprocess, e.g. via
the stdin pipe. This is how mp.Array works on Windows.
> On UNIX, Arrays and the stuff underneath it don't pickle because the
> underlying mmap is not named.
It is the same on Windows. Named shared memory is the cure in both cases.
The advantage of named shared memory is that it can be created after the
subprocesses are spawned/forked.
Sturla Molden
Gael Varoquaux
> > Am I out of my mind, and will this fail utterly?
> It will work. But we should use named shared memory (which requires some C
> or Cython coding), not BSD mmap as mp.Array currently does. Also we must
> override how ndarrays are pickled.
I just wanted to say that I am still interested in exploring this a bit
deeper, but I got swamped suddenly. Besides, as Robert and you have sown,
there is more than I thought to it.
Cheers,
Gaël
Robert Kern
<gael.va...@normalesup.org> wrote:
> On Mon, Feb 02, 2009 at 06:24:18PM +0100, Sturla Molden wrote:
>> > Am I out of my mind, and will this fail utterly?
>
>> It will work. But we should use named shared memory (which requires some C
>> or Cython coding), not BSD mmap as mp.Array currently does. Also we must
>> override how ndarrays are pickled.
>
> I just wanted to say that I am still interested in exploring this a bit
> deeper, but I got swamped suddenly. Besides, as Robert and you have sown,
> there is more than I thought to it.
BTW, Philip Semanchuk, the maintainer of the aforementioned shm
module, contacted Sturla and myself offlist to point out two, more
up-to-date, modules which provide named shared memory on UNIX systems:
http://semanchuk.com/philip/sysv_ipc/
http://semanchuk.com/philip/posix_ipc/
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
Gael Varoquaux
> BTW, Philip Semanchuk, the maintainer of the aforementioned shm
> module, contacted Sturla and myself offlist to point out two, more
> up-to-date, modules which provide named shared memory on UNIX systems:
> http://semanchuk.com/philip/sysv_ipc/
> http://semanchuk.com/philip/posix_ipc/
Interesting. I wonder how to use these. I would really like to see shared
memory in numpy by itself at some point. I did not look at the code as it
is GPL, from what I see.
The core idea, from what I understand, would be to use the POSIX shm_open
call to expose some named shared to numpy using eg from_buffer. Or can we
simply make it point to the pointer of an existing array using shmat, if
is is contiguous? That would avoid a copy (if contiguous).
Finally, to make sure share memory works with multiprocessing, we would
have to override pickling so that the pickling and unpicking are done
simply by storing the name of the shared memory object or retrieving it.
This is risky, because actual persistence would be destroyed.
Under Window we would use CreateSharedMemory to perform the same trick
using CreateFileMapping and MapViewOfFile?
Sounds fun.
Gaël
Brian Granger
at all, but I guess I have some reading to do. One important question
though:
Can these mechanisms be used to create shared memory amongst processes
that are started in a completely independent manner. That is, they
are not fork()'d.
If so, then we should develop a shared memory version of numpy arrays
that will work in any multiple-process setting. I am thinking
multiprocessing *and* the IPython.kernel.
Cheers,
Brian
Philip Semanchuk
> This is quite interesting indeed. I am not familiar with this stuff
> at all, but I guess I have some reading to do. One important question
> though:
> Can these mechanisms be used to create shared memory amongst processes
> that are started in a completely independent manner. That is, they
> are not fork()'d.
> If so, then we should develop a shared memory version of numpy arrays
> that will work in any multiple-process setting. I am thinking
> multiprocessing *and* the IPython.kernel.
Hi all,
I'm the author of the aforementioned IPC modules and I thought I'd
jump in even though I'm not a numpy guy.
Yes, one can use IPC objects (Sys V or POSIX) in completely
independent processes. There's a demo that comes along with both
modules that demonstrates that. I guess numpy isn't GPLed? You could
still download either one of the above packages and run the demo to
observe the process independence.
Gaël, AFAIK shared memory is guaranteed to be contiguous. I'm making
my assumption based on the fact that neither the Sys V nor POSIX API
has any references to accessing different chunks of memory. It's
treated as one logical block. In fact, the POSIX API for creating
shared memory (shm_open) simply returns a file descriptor that one
accesses as a memory mapped file:
http://www.opengroup.org/onlinepubs/000095399/functions/shm_open.html
HTH
Philip
Robert Kern
> Brian Granger wrote:
>
>> This is quite interesting indeed. I am not familiar with this stuff
>> at all, but I guess I have some reading to do. One important question
>> though:
>> Can these mechanisms be used to create shared memory amongst processes
>> that are started in a completely independent manner. That is, they
>> are not fork()'d.
>> If so, then we should develop a shared memory version of numpy arrays
>> that will work in any multiple-process setting. I am thinking
>> multiprocessing *and* the IPython.kernel.
>
> Hi all,
> I'm the author of the aforementioned IPC modules and I thought I'd
> jump in even though I'm not a numpy guy.
>
> Yes, one can use IPC objects (Sys V or POSIX) in completely
> independent processes. There's a demo that comes along with both
> modules that demonstrates that. I guess numpy isn't GPLed?
No, we're BSD-licensed.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
Andrew Straw
stuff. I wouldn't attempt to resurrect the complete working module, but
cut and paste at will:
http://code.astraw.com/projects/motmot/browser/trunk/pycamiface/src/_cam_iface_shm.pyx?rev=328
(This was from a wrapper of a camera driver that used shared memory
since the camera driver was very badly behaved and couldn't be trusted
to run in the same process. I have since stopped using this code and
wouldn't have time to get it working again, but it did open and use
shared memory quite nicely on linux.)
Also I found this web site very useful:
http://www.ecst.csuchico.edu/~beej/guide/ipc/
Brian Granger
Gael Varoquaux
> If so, then we should develop a shared memory version of numpy arrays
> that will work in any multiple-process setting. I am thinking
> multiprocessing *and* the IPython.kernel.
I am +1 on that, obviously. I'd love to see a 'fork'-based IPython
version, though :).
Sturla Molden
> Can these mechanisms be used to create shared memory amongst processes
> that are started in a completely independent manner. That is, they
> are not fork()'d.
Yes it can. If we know the name of the segment (an integer on Unix, a
string on Windows), it can be mapped into any process. Similarly for
named semaphores.
This is different from the locks ans shared memory in multiprocessing,
which must be shared through forking (Unix) or handle inheritance
(Windows), and therefore created prior to instantiation of
multiprocessing.Process. Otherwise, there is no valid handle to inherit.
The question remains: should we base this on Cython or C (there is very
little coding to do), or some third party extension, e.g. Philip
Semanchuk's POSIX IPC and Mark Hammond's pywin32? I am thinking that at
least for POSIX IPC, GPL is a severe limitation. Also we need some
automatic clean up, which can only be accomplished with an extension
object (that is, __dealloc__ in Cython will always be called, as opposed
to __del__ in Python). In Pywin32 there is a PyHANDLE object that
automatically calls CloseHandle when it is collected. But I don't think
Semanchuk's POSIX IPC module will do the same. And avoiding dependencies
on huge projects like pywin32 is also good.
Sturla Molden
Gael Varoquaux
> The question remains: should we base this on Cython or C (there is very
> little coding to do), or some third party extension, e.g. Philip
> Semanchuk's POSIX IPC and Mark Hammond's pywin32? I am thinking that at
> least for POSIX IPC, GPL is a severe limitation. Also we need some
> automatic clean up, which can only be accomplished with an extension
> object (that is, __dealloc__ in Cython will always be called, as opposed
> to __del__ in Python). In Pywin32 there is a PyHANDLE object that
> automatically calls CloseHandle when it is collected. But I don't think
> Semanchuk's POSIX IPC module will do the same. And avoiding dependencies
> on huge projects like pywin32 is also good.
I am all for avoiding external dependencies (especially if they are GPL).
multiprocessing is in the standard library, I would like to be able to do
shared memory parallel computing with only numpy and the standard
library. Actually I can see a near future where some algorithms of scipy
could have the option of using multiple cores (I am thinking of eg
non-parametric statistics).
The __dealloc__ argument is a very good one for going with Cython. In
addition I really like the feeling of Cython code. And am I wrong in
thinking that it would make the transition to Python3 easier?
Gaël
Philip Semanchuk
On Feb 6, 2009, at 10:15 AM, Sturla Molden wrote:
> On 2/6/2009 1:34 AM, Brian Granger wrote:
>
>> Can these mechanisms be used to create shared memory amongst
>> processes
>> that are started in a completely independent manner. That is, they
>> are not fork()'d.
>
> Yes it can. If we know the name of the segment (an integer on Unix, a
> string on Windows), it can be mapped into any process. Similarly for
> named semaphores.
A small correction -- SysV IPC objects are referred to with an integer
key. POSIX IPC objects are referred to with a file system-ish name
e.g. "/my_semaphore".
> The question remains: should we base this on Cython or C (there is
> very
> little coding to do), or some third party extension, e.g. Philip
> Semanchuk's POSIX IPC and Mark Hammond's pywin32? I am thinking that
> at
> least for POSIX IPC, GPL is a severe limitation. Also we need some
> automatic clean up, which can only be accomplished with an extension
> object (that is, __dealloc__ in Cython will always be called, as
> opposed
> to __del__ in Python). In Pywin32 there is a PyHANDLE object that
> automatically calls CloseHandle when it is collected. But I don't
> think
> Semanchuk's POSIX IPC module will do the same. And avoiding
> dependencies
> on huge projects like pywin32 is also good.
You are correct that posix_ipc doesn't close handles when deallocated.
THis is a deliberate choice -- the documentation says that closing the
handle makes the IPC object no longer available to the *process*. So
if one has multiple handles to an IPC object (say, inside multiple
threads), closing one would invalidate them all. But as I write this,
I'm wondering if that's not just a documentation bug and something
with which I ought to experiment.
bye
Philip
Sturla Molden
> You are correct that posix_ipc doesn't close handles when deallocated.
> THis is a deliberate choice -- the documentation says that closing the
> handle makes the IPC object no longer available to the *process*. So
> if one has multiple handles to an IPC object (say, inside multiple
> threads), closing one would invalidate them all. But as I write this,
> I'm wondering if that's not just a documentation bug and something
> with which I ought to experiment.
I have been thinking about this as well. I am mostly familiar with
Windows so excuse my terminology: We don't want an array to call
CloseHandle() on a mapped segment that another array is still using. The
effect would be global to the process. Thus, we would either need to
maintain some sort of global reference count for all mapped shared
resources, or make duplicates of the handle. On Windows there is a
function called DuplicateHandle() that will do this. I am not sure about
Unix.
Sturla Molden
Sturla Molden
> I have been thinking about this as well. I am mostly familiar with
> Windows so excuse my terminology: We don't want an array to call
> CloseHandle() on a mapped segment that another array is still using. The
> effect would be global to the process. Thus, we would either need to
> maintain some sort of global reference count for all mapped shared
> resources, or make duplicates of the handle. On Windows there is a
> function called DuplicateHandle() that will do this. I am not sure about
> Unix.
On Unix we could possibly use a WeakValueDictionary with name as key and
handle as value. And then let the handle object close itself when it is
collected. So an array could first look for an open handle in the
dictionary, before trying to map a new one. And since we are doing all
this in Cython, the GIL will take care of the synchronization. This
would also work on Windows, but there we have DuplicateHandle as another
option.
S.M.
Philip Semanchuk
On Feb 6, 2009, at 10:38 AM, Sturla Molden wrote:
> On 2/6/2009 4:26 PM, Philip Semanchuk wrote:
>
>> You are correct that posix_ipc doesn't close handles when
>> deallocated.
>> THis is a deliberate choice -- the documentation says that closing
>> the
>> handle makes the IPC object no longer available to the *process*. So
>> if one has multiple handles to an IPC object (say, inside multiple
>> threads), closing one would invalidate them all. But as I write this,
>> I'm wondering if that's not just a documentation bug and something
>> with which I ought to experiment.
>
> I have been thinking about this as well. I am mostly familiar with
> Windows so excuse my terminology: We don't want an array to call
> CloseHandle() on a mapped segment that another array is still using.
> The
> effect would be global to the process. Thus, we would either need to
> maintain some sort of global reference count for all mapped shared
> resources, or make duplicates of the handle. On Windows there is a
> function called DuplicateHandle() that will do this. I am not sure
> about
> Unix.
On Unix, one can duplicate a file handle with a call to dup().
Note that the doc for shm_unlink() says this:
"If one or more references to the shared memory object exist when the
object is unlinked, the name shall be removed before shm_unlink()
returns, but the removal of the memory object contents shall be
postponed until all open and map references to the shared memory
object have been removed."
Furthermore (and this is where it gets tricky):
"Even if the object continues to exist after the last shm_unlink(),
reuse of the name shall subsequently cause shm_open() to behave as if
no shared memory object of this name exists (that is, shm_open() will
fail if O_CREAT is not set, or will create a new shared memory object
if O_CREAT is set)."
You'd have to do your testing very carefully to see if dup() really
increments the kernel's reference count on a shared memory segment.
Sturla Molden
> You'd have to do your testing very carefully to see if dup() really
> increments the kernel's reference count on a shared memory segment.
Ok, in that case it is probably better to let Python take care of the
reference counting.
S.M.
Sturla Molden
Ok, so this is approximately what I had in mind for Windows. It is a
named mutex and shared memory that is pickled by name (given that I read
the Python manuals on pickling extension objects correctly...)
It still lacks an ndarray subclass that is pickled without making a copy
of the buffer, and also a malloc similar to multiprocessing.
And similar Cython code has to be written for posix...
But this is a start. If anyone feel like contributing, please do.
S.M.
Gael Varoquaux
>
> Ok, so this is approximately what I had in mind for Windows. It is a named
> mutex and shared memory that is pickled by name (given that I read the
> Python manuals on pickling extension objects correctly...)
>
> It still lacks an ndarray subclass that is pickled without making a copy of
> the buffer, and also a malloc similar to multiprocessing.
>
> And similar Cython code has to be written for posix...
OK, I've given it try, but it seems that my sheer incompetence on these
matters is about to be revealed. Running the attached test code, I get a
bus error. The output of test.py is:
{'/c0aa50edb5a04371b8414ef16a49a4fa': (3070545920L, 409600)}
Buffer created
Array created
3070545920
[1] 9882 bus error python test.py
I am quite clueless as to where this comes from (I can see different
posibilities) and how to debug this.
Once again, this is from sheer incompetence, but I have never mmaped
files throught the C API, and my days of C, especially memory allocation
in C, are very far.
I am posting this on the mailing list hoping that someone will have an
idea as to what I am doing wrong. Once this work, we can start looking at
making this clean to have posix and windows implementations work
together.
Gaël
Philip Semanchuk
Hi Gaël,
I believe one must call ftruncate() on the file handle returned by
shm_open(). Look at the example at the bottom of this page:
http://www.opengroup.org/onlinepubs/000095399/functions/shm_open.html
I hope this info is useful.
Philip
Gael Varoquaux
> Hi Gaël,
> I believe one must call ftruncate() on the file handle returned by
> shm_open(). Look at the example at the bottom of this page:
> http://www.opengroup.org/onlinepubs/000095399/functions/shm_open.html
Hurray, that was it! The code snippet at the end of this page is very
clear. Thank you for the pointer.
> I hope this info is useful.
It really was. Thanks a lot. I need to do a few more checks, but I
believe I have a first version of some code sharing arrays by name.
Gaël
Gael Varoquaux
> It really was. Thanks a lot. I need to do a few more checks, but I
> believe I have a first version of some code sharing arrays by name.
OK, I have a first working version under Unix (attached, with trivial
test case).
Now we need to make it so that the ndarray can be used in the
mutliprocessing function call, rather than the buffer object. In other
words we need to create an object that behaves as an ndarray, but
implements a different pickling method. What do people suggest as a best
approach here? Subclassing ndarray?
Cheers,
Gaël
Sturla Molden
> What do people suggest as a best
> approach here? Subclassing ndarray?
I have been working on that. Basically using what Robert Kern posted a
while ago and ripping out some code from multiprocessing's heap object.
S.M.
Gael Varoquaux
> On 2/9/2009 9:23 AM, Gael Varoquaux wrote:
> > What do people suggest as a best
> > approach here? Subclassing ndarray?
> I have been working on that. Basically using what Robert Kern posted a
> while ago and ripping out some code from multiprocessing's heap object.
Fantastic. I have to worry about other things for a little while
(real-work related), so I won't be competing without you to find a good
solution :).
Cheers,
Gaël
Sturla Molden
>> It really was. Thanks a lot. I need to do a few more checks, but I
>> believe I have a first version of some code sharing arrays by name.
>
> OK, I have a first working version under Unix (attached, with trivial
> test case).
By the way, how is memory reclaimed under your Posix code?
On Windows, a memory mapping is removed when there is no open handles to
it. That is what the Handle object does (i.e. preventing a sytem wide
memory leak).
On System V IPC a shared segment it has to be marked for removal, i.e.
there are no reference counting in the kernel as in Windows. So I was
thinking out marking it for removal when the attachment count is zero.
But as you have used Posix V IPC I have no idea. Just make sure it does
not produce a global memory leak.
S.M.
Gael Varoquaux
> On System V IPC a shared segment it has to be marked for removal, i.e.
> there are no reference counting in the kernel as in Windows. So I was
> thinking out marking it for removal when the attachment count is zero.
> But as you have used Posix V IPC I have no idea. Just make sure it does
> not produce a global memory leak.
Hum, I believe you are right, and I have produced just that.
This means that we probably need a shared reference counter :(. Sounds
tedious to implement. Do people have any suggestions on how to implement
this? I can see several possibilities:
* Using multiprocessing to share the dictionnary of shared map
addresses, but this induces a tight coupling with multiprocessing, and
I am not sure we want this.
* Sharing this dictionnary via a C structure, ie to do our own
implementation of a shared state.
* Add the ref count information in the shared array. For instance the
first byte could be the ref count. This sounds the easiest option,
but I am probably not seeing some of the problems that will arize
from this approach.
I think I am going to take a stab at option three, tonight or later in
the week, but please, wise people of the list, give me feedback on what
you think might work.
Gaël
Sturla Molden
> This means that we probably need a shared reference counter :(. Sounds
> tedious to implement.
On System V, you can get the attachment count using shmctl with
IPC_STAT. Then after calling shmdt, checking the count and marking for
removal if it is zero:
int cleanup(int shmid)
{
int ierr;
struct shmid_ds buf;
ierr = shmctl(shmid, IPC_STAT, &buf);
if(ierr < 0) goto error;
if (buf.shm_nattch == 0) {
ierr = shmctl(shmid, IPC_RMID, NULL);
if(ierr < 0) goto error;
}
return 0
error:
return errno;
}
S.M.
Philip Semanchuk
On Feb 9, 2009, at 6:56 AM, Sturla Molden wrote:
> On 2/9/2009 12:38 PM, Gael Varoquaux wrote:
>
>> This means that we probably need a shared reference counter :(.
>> Sounds
>> tedious to implement.
>
> On System V, you can get the attachment count using shmctl with
> IPC_STAT. Then after calling shmdt, checking the count and marking for
> removal if it is zero:
>
> int cleanup(int shmid)
> {
> int ierr;
> struct shmid_ds buf;
> ierr = shmctl(shmid, IPC_STAT, &buf);
> if(ierr < 0) goto error;
> if (buf.shm_nattch == 0) {
> ierr = shmctl(shmid, IPC_RMID, NULL);
> if(ierr < 0) goto error;
> }
> return 0
> error:
> return errno;
> }
Unfortunately POSIX IPC doesn't report that information.
Since I'm not a numpy user I'm a little lost as to how you're using
the shared memory here, but I gather that it is effectively "magic" to
a numpy user? i.e., he doesn't have any idea that a shared memory
segment is being created on his behalf? If that's the case I don't see
any way around reference counting.
Gael Varoquaux
> Since I'm not a numpy user I'm a little lost as to how you're using
> the shared memory here, but I gather that it is effectively "magic" to
> a numpy user? i.e., he doesn't have any idea that a shared memory
> segment is being created on his behalf?
My goal would be that he shouldn't have to know, or to care. It should be
as much transparent as possible.
> If that's the case I don't see any way around reference counting.
Thanks for your input, it is valued a lot.
Gaël
Philip Semanchuk
Gaël,
I notice that the size of the shared memory segment is set to "pages"
* PAGESIZE. Who determines the value of "pages"? And what happens if
the numpy object you're storing in the segment grows beyond that size?
AFAIK ftruncate() can only be called *once* to resize the segment.
That's true on OS X, anyway, so it's probably true elsewhere.
I once wrote some code to implement a shared dict using shared memory,
and this was a problem I ran into. What happens when an item grows?
The solution I eventually developed was to have one shared memory
segment for metadata and a collection of other shared memory segments
to hold the actual data. The metadata segment stored a (pickled) free
space map and if a request was made to store an item that was larger
than any free space I had, I'd allocate a new segment of the
appropriate size. Otherwise, I'd stick it in the smallest piece of
free space that it would fit into in an existing segment.
You can perhaps see where this is leading -- once one is tracking free
space slots and so forth, one needs to think about memory compaction,
too, because sooner or later items will get deleted from the dict and
if nothing new is inserted all of that free space is sitting around
going to waste.
Also, is it consistent with your license to use code from Python
itself? If so, then I have another minor suggestion.
Cheers
Philip
Robert Kern
> Also, is it consistent with your license to use code from Python
> itself? If so, then I have another minor suggestion.
Yup.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
Sturla Molden
> Unfortunately POSIX IPC doesn't report that information.
I'll suggest we use System V IPC instead, as it does report a ref count.
Code example attached. It compiles with Cython but I have not done any
testing except that.
My suggestion is to spawn a thread in the creator process to monitor the
attachment count for the segment, and mark it for removal when it has
dropped to zero. There is a __dealloc__ in a Handle object that does the
shmdt, and then Python should do the refcounting (similar to what is
done for CloseHandle in Windows).
We have to figure out what to do with ctrl-c. It is a source of trouble.
With a daemonic GC thread it could cause a leak, with a non-daemonic GC
thread it may hang forever (which is also a leak). So I opted for a
daemonic GC thread.
I also have a version of the Windows sharedmem with a small bugfix (I
forgot to unmap the segment before closing the handle). I had to remove
the mutex from the Windows code. It can be put in a separate module. We
should also have a lock with a named Sys V semaphore.
> Since I'm not a numpy user I'm a little lost as to how you're using
> the shared memory here, but I gather that it is effectively "magic" to
> a numpy user? i.e., he doesn't have any idea that a shared memory
> segment is being created on his behalf? If that's the case I don't see
> any way around reference counting.
We are going to use multiple processes as if they were threads. It is
basically a hack to work around Python's GIL (global interpreter lock).
Basically we want to create ndarray's with the same interface as before,
except that they have shared memory as data. For example,
import numpy
a = numpy.zeros((4,1024), order='F', dtype=float)
import scipy
a = scipy.sharedmem.zeros((4,1024), order='F', dtype=float)
should do the same, except that the latter uses shared memory. And when
it is sent through a multiprocessing.Queue, only the segment name,
offset, shape and dtype gets pickled. In the former case, a copy of the
whole data buffer is made. Right now we are just creating the shared
memory buffer to use as backend.
In multiprocessing you will find an object called mp.Array. We can wrap
its buffer with an ndarray, but it cannot be passes through a mp.Queue.
In other words, all shared memory must be allocated in advance. And that
is what we don't want.
Sturla Molden
Philip Semanchuk
On Feb 9, 2009, at 11:24 AM, Robert Kern wrote:
> On Mon, Feb 9, 2009 at 09:28, Philip Semanchuk
> <phi...@semanchuk.com> wrote:
>> Also, is it consistent with your license to use code from Python
>> itself? If so, then I have another minor suggestion.
>
> Yup.
I'm not sure how prevalent the getpagesize() API is. You might want to
consider using the following code (from Python's mmapmodule.c) to get
the page size.
#ifdef MS_WINDOWS
#include <windows.h>
static int
my_getpagesize(void)
{
SYSTEM_INFO si;
GetSystemInfo(&si);
return si.dwPageSize;
}
#endif
#ifdef UNIX
#include <sys/mman.h>
#include <sys/stat.h>
#if defined(HAVE_SYSCONF) && defined(_SC_PAGESIZE)
static int
my_getpagesize(void)
{
return sysconf(_SC_PAGESIZE);
}
#else
#define my_getpagesize getpagesize
#endif
#endif /* UNIX */
Sturla Molden
> I once wrote some code to implement a shared dict using shared memory,
> and this was a problem I ran into.
We should have that removed. The actual allocation will be rounded up to
a multiple of the page size. So to prevent a leak we should round up
before allocating and reporting the actual size.
> What happens when an item grows?
We don't want an array to grow or move once it has been created. But a
process should be allowed to create subarray views.
S.M.
Sturla Molden
> I'm not sure how prevalent the getpagesize() API is. You might want to
> consider using the following code (from Python's mmapmodule.c) to get
> the page size.
I think we can just use mmap.PAGESIZE :)
S.M.
Philip Semanchuk
On Feb 9, 2009, at 11:42 AM, Sturla Molden wrote:
> On 2/9/2009 4:07 PM, Philip Semanchuk wrote:
>
>> Unfortunately POSIX IPC doesn't report that information.
>
> I'll suggest we use System V IPC instead, as it does report a ref
> count. Code example attached. It compiles with Cython but I have not
> done any testing except that.
>
> My suggestion is to spawn a thread in the creator process to monitor
> the attachment count for the segment, and mark it for removal when
> it has dropped to zero. There is a __dealloc__ in a Handle object
> that does the shmdt, and then Python should do the refcounting
> (similar to what is done for CloseHandle in Windows).
If you're destroying the segment when the attach count drops to zero,
why not check that immediately after the call to shmdt()?
> key = ftok( <char *> (self.name), 0)
ftok() should probably be avoided as it returns duplicate keys:
http://nikitathespider.com/python/shm/#ftok
I'd recommend using a random number generator instead. I believe a
key_t is guaranteed to fit into an int, so you could generate a random
number anywhere from 1 to INT_MAX, taking care not to step on the
value IPC_PRIVATE (unless you want to assume that that is always
#defined to 0).
Sturla Molden
> If you're destroying the segment when the attach count drops to zero,
> why not check that immediately after the call to shmdt()?
I thought it was only the owner/creator that was allowed to do that?
> ftok() should probably be avoided as it returns duplicate keys:
> http://nikitathespider.com/python/shm/#ftok
Oh :(
In that case I could rewrite the object to pickle the shmid instead of a
random name (uuid string) on System V.
> I'd recommend using a random number generator instead. I believe a
> key_t is guaranteed to fit into an int, so you could generate a random
> number anywhere from 1 to INT_MAX, taking care not to step on the
> value IPC_PRIVATE (unless you want to assume that that is always
> #defined to 0).
I am not sure how big the problem is, as I pass an uuid as filename to ftok.
S.M.
Sturla Molden
> I have been working on that. Basically using what Robert Kern posted a
> while ago and ripping out some code from multiprocessing's heap object.
Here is a first draft. The ftok issue is not fixed.
I am not sure if Robert Kern's use of copy_reg affects ndarrays in
general, or just the ones we create here.
There are probably a few bugs to kill.
And we need a setup script.
S.M.
Robert Kern
> On 2/9/2009 12:20 PM, Sturla Molden wrote:
>
>> I have been working on that. Basically using what Robert Kern posted a
>> while ago and ripping out some code from multiprocessing's heap object.
>
> Here is a first draft. The ftok issue is not fixed.
>
> I am not sure if Robert Kern's use of copy_reg affects ndarrays in general,
> or just the ones we create here.
It affects ndarrays in general. The reduce function should ideally be
written to detect whether the ndarray is shared, or is a view
eventually leading back to a shared ndarray, or is just a regular
ndarray.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
Philip Semanchuk
On Feb 9, 2009, at 1:05 PM, Sturla Molden wrote:
> On 2/9/2009 6:49 PM, Philip Semanchuk wrote:
>
>> If you're destroying the segment when the attach count drops to zero,
>> why not check that immediately after the call to shmdt()?
>
> I thought it was only the owner/creator that was allowed to do that?
Yes, sort of. Sys V IPC objects are owned by users, not processes, so
if user foo creates a semaphore in one process and destroys it in
another, that's OK. I just verified this on OS X. Unless portions of
your SciPy application will be running under different users, I think
the matter of which process destroys an IPC object is irrelevant.
>> ftok() should probably be avoided as it returns duplicate keys:
>> http://nikitathespider.com/python/shm/#ftok
>
> Oh :(
>
> In that case I could rewrite the object to pickle the shmid instead
> of a
> random name (uuid string) on System V.
But you need the key, not the id, to pass to shmget() to get a handle
to an existing IPC object.
>> I'd recommend using a random number generator instead. I believe a
>> key_t is guaranteed to fit into an int, so you could generate a
>> random
>> number anywhere from 1 to INT_MAX, taking care not to step on the
>> value IPC_PRIVATE (unless you want to assume that that is always
>> #defined to 0).
>
> I am not sure how big the problem is, as I pass an uuid as filename
> to ftok.
I'm not sure how big the problem is either. All I know is that in my
experience, ftok() returned the same key for different files in the
same directory. I realized, therefore, that my code needed to handle
the case where ftok() didn't generate a useful key. Since I needed a
second, more reliable method of key generation, why use ftok() at all?
If I were you, rather than trying to figure out how broken ftok() is
(and it might be broken in different ways on different platforms), I'd
just abandon it altogether. It's not as if generating a random number
instead is difficult. In fact, it's easier. Instead of generating a
random uuid and passing that to ftok(), eliminate the middleman and
generate a random key yourself.
Sturla Molden
> On Feb 9, 2009, at 1:05 PM, Sturla Molden wrote:
> Yes, sort of. Sys V IPC objects are owned by users, not processes, so
> if user foo creates a semaphore in one process and destroys it in
> another, that's OK. I just verified this on OS X. Unless portions of
> your SciPy application will be running under different users, I think
> the matter of which process destroys an IPC object is irrelevant.
My programs will certainly not do that (but I use Windows anyway). I have
one process that spawns workers, and they run with the same user. I think
that covers 99% of all use for this extension.
But others may have a more complex design, so the safest method is to let
the creator kill the segment. But it comes at the expense of a thread.
Otherwise I could just assume the user is the same and do the check after
shmdt. As I use Windows I have no personal preference here. Not using a
thread avoids some of the ctrl-c issue, and it will be a bit faster. But
then all processes sharing memory must run with the same user, otherwise
there will be leaks and havoc.
>> In that case I could rewrite the object to pickle the shmid instead
>> of a
>> random name (uuid string) on System V.
>
> But you need the key, not the id, to pass to shmget() to get a handle
> to an existing IPC object.
Yes, but if we use the shmid, we have the handle. So we can pass that
integer from one process to another. At least that is what my old book on
Linux programming describes. Since you say I need to use shmget, can I
assume this method is not valid on all Unix incarnations?
> If I were you, rather than trying to figure out how broken ftok() is
> (and it might be broken in different ways on different platforms), I'd
> just abandon it altogether. It's not as if generating a random number
> instead is difficult. In fact, it's easier. Instead of generating a
> random uuid and passing that to ftok(), eliminate the middleman and
> generate a random key yourself.
It has to be a unique key for the system, not just a random number. So I
could try to call shmget multiple times with IPC_EXCL until it succeeds.
Then I'll have to check why it failed as well.
This is the first time I have found Windows to be the less annoying system.
Thanks for your help. :-)
Sturla Molden
Sturla Molden
Just a small update: I have removed ftok and done what Philip Semanchuk
suggested. The System V version now uses numpy's random integer generator
to create a key (and if it fails, checks errno for EEXIST). Clean up can
be done using threads or without threads; keep your paws off os.setuid if
you set gc_thread to False.
S.M.
Robert Kern
> Ok, the work is basically done :)
>
> What remains is testing/debugging and a setup script.
>
> Perhaps we should move this debate to scipy-dev? I feel like I am spamming
> this list...
<shrug> Whatever. It would be nice, though, if you hosted the files
somewhere, perhaps under source control, instead of passing them
around in attachments.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
Philip Semanchuk
On Feb 9, 2009, at 4:41 PM, Sturla Molden wrote:
>>> In that case I could rewrite the object to pickle the shmid instead
>>> of a
>>> random name (uuid string) on System V.
>>
>> But you need the key, not the id, to pass to shmget() to get a handle
>> to an existing IPC object.
>
> Yes, but if we use the shmid, we have the handle. So we can pass that
> integer from one process to another. At least that is what my old
> book on
> Linux programming describes. Since you say I need to use shmget, can I
> assume this method is not valid on all Unix incarnations?
No, sorry, ignore what I said. I was not thinking clearly.
>> If I were you, rather than trying to figure out how broken ftok() is
>> (and it might be broken in different ways on different platforms),
>> I'd
>> just abandon it altogether. It's not as if generating a random number
>> instead is difficult. In fact, it's easier. Instead of generating a
>> random uuid and passing that to ftok(), eliminate the middleman and
>> generate a random key yourself.
>
> It has to be a unique key for the system, not just a random number.
> So I
> could try to call shmget multiple times with IPC_EXCL until it
> succeeds.
> Then I'll have to check why it failed as well.
Exactly. It's a pain in the arse but it is what must be done.
> This is the first time I have found Windows to be the less annoying
> system.
Indeed, that's a rare occurrence. The POSIX API is better than Sys V
in that there's a much larger "key" space, so large that collisions
between randomly generated ids are statistically...hmmm, maybe I
should watch my mouth on a statistically-oriented mailing list. =)
Under POSIX, an IPC object's name is a string that starts with a
slash. Under FreeBSD (the most restrictive API I've encountered), the
name is limited to 14 filename-permissible characters. Subtracting the
leading slash, that's space for 13 alphanumeric characters plus
underscore and dot. Filenames permit 52 upper and lowercase characters
plus 10 digits plus 2 punctuation characters = 64 characters. 64**13
is a lot more choices than INT_MAX. So OK, I stand by my statement.
How does the Windows API resolve name/key collisions?
> Thanks for your help. :-)
Glad to be able to provide it. (Ni är välkomna.)
Cheers
Philip
Gael Varoquaux
> On 2/9/2009 5:48 PM, Philip Semanchuk wrote:
> > I'm not sure how prevalent the getpagesize() API is. You might want to
> > consider using the following code (from Python's mmapmodule.c) to get
> > the page size.
> I think we can just use mmap.PAGESIZE :)
Good point :). I was using getpagesize, from unistd.h.
Gaël
Sturla Molden
> How does the Windows API resolve name/key collisions?
Right. On Windows a name is a string, and an UUIDs should be unique to
the system. If a name exists, CreateFileMapping fails and GetLastError
returns ERROR_INVALID_HANDLE. If the object alredy exist in the process,
CreateFileMapping returns a valid handle but GetLastError returns
ERROR_ALREADY_EXISTS. I'll put som some tests for that to be pedantic,
albeit UUIDs should be unique.
Sturla Molden
Sturla Molden
The Windows version seems to be working correctly on my computers.
I will put updates here:
http://folk.uio.no/sturlamo/python/sharedmem.zip
Testing (particularly on Unix and friends) is appreciated. If you have
comments, encounter bugs, or have corrections, please send them to my email.
usage:
import sharedmem as shm
Now shm.zeros, shm.ones, and shm.empty should work like their numpy
equivalents. They are pickled and depickled by name (hidden from sight),
meaning only metadata is stored in the pickle. Call .copy() if you need
the pickle to contain a copy of the data as well. Unlike
multiprocessing.Array, these shared memory arrays can be sent through a
multiprocessing.Queue, tcp, or any other IPC you may think of. That's it.
I am done spamming this list with this for now.
Regards,
Gael Varoquaux
> Ok, the work is basically done :)
Congratulation, Sturla!
I must admit that I am not too much enthousiastic about the
thread+polling to do the cleaning up. I don't really understand why it is
necessary. If the view of the array is the last to be decref, than
'buf.shm_nattch' should be 0, and as a result the freeing up of the
memory can happen in the dealloc. Or did I miss something?
Cheers,
Gaël
Gael Varoquaux
> What remains is testing/debugging and a setup script.
I did a setup script, and I had to change a few detail because Cython was
unhappy with the names of the modules (I suspect local imports happening
instead of absolute ones).
I had to add a __weakref__ attribute to the handle, to make it so that it
can be weakref'd.
Now I am stuck because shared memory allocation is not working. This
boils down to the following traceback:
Traceback (most recent call last):
File "test.py", line 4, in <module>
a = shmem.shared_zeros(10)
File "ndarray.py", line 135, in shared_zeros
arr = shared_empty(shape, dtype, order)
File "ndarray.py", line 126, in shared_empty
wrapper = heap.BufferWrapper(nbytes)
File "array_heap.py", line 168, in __init__
block = BufferWrapper._heap.malloc(size)
File "array_heap.py", line 148, in malloc
(arena, start, stop) = self._malloc(size)
File "array_heap.py", line 70, in _malloc
arena = Arena(length)
File "array_heap.py", line 37, in __init__
self.buffer = SharedMemoryBuffer(size)
File "sharedmemory_sysv.pyx", line 170, in
sharedmemory_sysv.SharedMemoryBuffer.__init__ (sharedmemory_sysv.c:1400)
raise OSError, "Failed to attach shared memory: permission denied"
OSError: Failed to attach shared memory: permission denied
Basically this means that the shmat on line 167 of sharedmemory_sysv.pyx
is failing. I don't really know why, but I suspect this might be
something stupid.
I need to go to bed now, and I probably won't have time to look at that
at all before thursday evening. Maybe I will be in luck and someone more
clever than me will have time to look at that in the mean time :).
Cheers,
Gaël
Philip Semanchuk
One problem I see is that the call to shmget() specifies no
permissions. The third param to shmget() should contain two sets of
bitwise params OR-ed together. The first set is IPC_CREAT and
IPC_EXCL, the second set is the permissions. So you might want to
change line 156 to this:
shmid = shmget(key, buf_size, IPC_CREAT | IPC_EXCL | 0600)
or this:
shmid = shmget(key, buf_size, IPC_CREAT | IPC_EXCL | 0666)
http://www.opengroup.org/onlinepubs/009695399/functions/shmget.html
Robert Kern
I believe that was the error I kept running into when I was futzing
around with this.
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
-- Umberto Eco
Gael Varoquaux
> One problem I see is that the call to shmget() specifies no
> permissions. The third param to shmget() should contain two sets of
> bitwise params OR-ed together. The first set is IPC_CREAT and
> IPC_EXCL, the second set is the permissions. So you might want to
> change line 156 to this:
> shmid = shmget(key, buf_size, IPC_CREAT | IPC_EXCL | 0600)
> or this:
> shmid = shmget(key, buf_size, IPC_CREAT | IPC_EXCL | 0666)
Indeed, Philip, that was it. Thanks a lot for your help.
Gaël
Gael Varoquaux
So I found a few more simple errors in the code and fixed them (code
attached). The garbage collector thread lock multiprocessing. I am not
sure why. I disabled it just to see what would happen. I added a few
print statement to try and debug eventual memory leaks.
I think I have a memory leak, judging from the different prints. I am not
sure though, and I wonder if there is a good way of checking this, other
than running the test code in a big loop and checking if the test box
eventually dies. Valgrind does report some 'possibly lost' blocks that
increase with the size of the array. Anybody has a suggestion on how to
debug this?
Cheers,
Gaël
Sturla Molden
> I did a setup script, and I had to change a few detail because Cython was
> unhappy with the names of the modules (I suspect local imports happening
> instead of absolute ones).
>
> I had to add a __weakref__ attribute to the handle, to make it so that it
> can be weakref'd.
Thanks Gael
I've noticed you used the version I posted to the list, and not the
latest on the web. So there is a lot of debugging you missed. I'll do a
quick merge of what you posted with mine.
I inherited via a Python class to allow a weakref to a Handle. Your
solution is cleaner.
As for the clean-up thread:
A shared segment has an owner on Linux. Only the owner or superuser can
mark it for deletion. Someone else but the owner may be the last to
detach, and then marking for deletion will fail. I think we should
remove the thread and raise an exception if marking for deletion fails.
We cannot completely foolproof the clean-up against use of os.setuid
anyway.
Sturla Molden
Sturla Molden
> I shouldn't be working on that, but this is way more fun :).
>
> So I found a few more simple errors in the code and fixed them
Here is my working Windows version from yesterday:
http://folk.uio.no/sturlamo/python/sharedmem.zip
Sturla Molden
> So I found a few more simple errors in the code and fixed them (code
> attached). The garbage collector thread lock multiprocessing.
def __dealloc__(SharedMemoryBuffer self):
print 'Calling __dealloc__ on buffer at %s' \
% <unsigned long> self.mapped_address #DBG
self.handle.dealloc()
Why do you do this? The Handle should self destruct. Anyway, this is
evil and will possibly case multiprocessing to hang, as well as segfaults.
Sturla
Sturla Molden
> So I found a few more simple errors in the code and fixed them (code
> attached).
Gael, I tried to merge your changes with mine. My Python code worked
yesterday (albeit not the version you've debugging), so I guess it still
does.
The gc thread is removed.
Sturla
Gael Varoquaux
> def __dealloc__(SharedMemoryBuffer self):
> print 'Calling __dealloc__ on buffer at %s' \
> % <unsigned long> self.mapped_address #DBG
> self.handle.dealloc()
> Why do you do this? The Handle should self destruct. Anyway, this is
> evil and will possibly case multiprocessing to hang, as well as segfaults.
This was for debugging. I do not understand why my test code shows only
one call to __dealloc__ (see below), and I am trying to figure out why. I
fear this has more to do with Python's garbage collector.
I agree this is evil. However, if I don't add this code, the __dealloc__
method of the handler does not seem get called in my example.
Here is what worries me:
I run this test code:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
import ndarray as shmem
import numpy as np
def modify_array(ary):
ary[:3] = 1
print 'Array address in sub program %s' % ary.ctypes.data
from multiprocessing import Pool
def main():
a = shmem.shared_zeros(10)
p = Pool()
print 'Array address in main program %s' % a.ctypes.data
print a
job = p.apply_async(modify_array, (a, ))
p.close()
p.join()
print a
main()
import gc
gc.collect()
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I get the following output:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Array address in main program 47294723575808
[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
Array address in sub program 47294723575808
Calling __dealloc__ on buffer at 47294723575808
Deallocated memory at 47294723575808
[ 1. 1. 1. 0. 0. 0. 0. 0. 0. 0.]
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The two messages about deallocation are debug prints that I inserted in
the two __dealloc__ methods. It seems to me that the array 'a' in the
main program has not been dellocated. I thus believe that there is a
memory leak (I haven't been able to really confirm). It seems to me that
the __dealloc__ method of 'a' does not get called in the main program. I
have also just added print of pid (not in above example), and the two
calls to __dealloc__ do happen in the child process. Finally, if I do not
call explictely __dealloc__ for the handler in the dealloc of the buffer,
I do not see it being called.
So I am wondering if we are not being tricked by the fact that Python
calls the __del__ method lazily, in particular when quitting. Maybe the
solution to this problem is to add an exit hook (seems like that's what
other people did when faced with this problem:
http://www.python.org/search/hypermail/python-recent/0635.html, follow up
is also interresting:
http://www.python.org/search/hypermail/python-recent/0636.html), however
this is not terribly robust. I wonder how mutliprocessing deals with this
problem.
By the way, I have just found a trivial bug: if I call shared_zeros with
1e5 as an argument, the code does not realise it should process this as
an int. I suggest that shared_empty also accepts floats in the 'magic'
cast from numbers to tuple for the shape, as this is what numpy does.
Gaël
Sturla Molden
if __name__ == "__main__":
main()
in you testing code for multiprocessing to work correctly. Leaving it
out is a source of mysterious errors. In fact on Windows, leaving it out
creates something similar to a fork bomb.
> So I am wondering if we are not being tricked by the fact that Python
> calls the __del__ method lazily, in particular when quitting. Maybe the
> solution to this problem is to add an exit hook (seems like that's what
> other people did when faced with this problem:
> http://www.python.org/search/hypermail/python-recent/0635.html, follow up
> is also interresting:
> http://www.python.org/search/hypermail/python-recent/0636.html), however
> this is not terribly robust. I wonder how mutliprocessing deals with this
> problem.
multiprocessing.util.Finalize is an exit hook. That should do the
clean-up in the main program. It clean up the BufferWrapper object,
which owns the SharedMemoryBuffer.
As long as the Heap object is destroyed, it will clean up. Try to put in
some printing to see if the buffer is marked for removal. Do not use a
Cython's print statement but something else (e.g. printf from stdio.h).
Sturla
Sturla Molden
> As long as the Heap object is destroyed,
eh, Handle object.
cdef extern from "stdio.h":
void printf(char *str)
cdef class Handle:
""" Automatic shared segment deattachment
- without this object we would need to do reference
counting manually, as shmdt is global to the process.
Do not instantiate this class, except from within
SharedMemoryBuffer.__init__.
"""
cdef int shmid
cdef object name
cdef object cleanup
cdef object __weakref__
def __init__(Handle self, shmid, name):
self.shmid = <int> shmid
self.name = name
def gethandle(Handle self):
return int(self.shmid)
def __dealloc__(Handle self):
self.dealloc()
def dealloc(Handle self):
cdef shmid_ds buf
cdef int _shmid= <int> self.shmid
cdef void *addr
cdef int ierr
try:
ma, size = __mapped_addresses[ self.name ]
addr = <void *>(<unsigned long> ma)
ierr = shmdt(addr)
if (ierr < 0): raise MemoryError, "shmdt failed."
del __mapped_addresses[ self.name ]
print "Deallocated memory at %s" % ma #DBG
except KeyError:
print __mapped_addresses #DBG
print self.name #DBG
print 'KeyError' #DBG
#pass
# this may happen and is not a problem
if (shmctl(_shmid, IPC_STAT, &buf) == -1):
raise OSError, \
"IPC_STAT failed, you could have a global memory leak!"
if (buf.shm_nattch == 0):
if( shmctl(_shmid, IPC_RMID, NULL) == -1 ):
raise OSError, \
"IPC_RMID failed, you have a global memory leak!"
else:
printf("shared segment removed\n")
S.M.
Gael Varoquaux
> As long as the Heap object is destroyed, it will clean up. Try to put in
> some printing to see if the buffer is marked for removal. Do not use a
> Cython's print statement but something else (e.g. printf from stdio.h).
I have put in some more print statements. I have the fealing that it is
not cleaned up. I am attaching my test code, and the modified
sharedmemory_sysv.pyx for debug. The output is the following:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Array address in main program 47782588157952 (pid: 18024)
[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
Array address in sub program 47782588157952 (pid: 18033)
Calling __dealloc__ on buffer at 47782588157952, in pid 18033
Checking for deallocating of memory at 47782588157952
Not deallocating: 8 attached segments
[ 1. 1. 1. 0. 0. 0. 0. 0. 0. 0.]
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I do have the feeling that something is not getting garbage-collected.
Gaël
Sturla Molden
> I have put in some more print statements. I have the fealing that it is
> not cleaned up. I am attaching my test code, and the modified
> sharedmemory_sysv.pyx for debug. The output is the following:
I have reproduced the same error in Windows. A suspected, it stems from
using multiprocessing's Finalizer. For some reason it is never called.
Also having the heap object as a class attribute of the BufferWrapper
prevents clean-up with a __del__ method. Whereas having it as a global
variable in the module works ok. The problem is not the Cython extension
code, it is the array_heap.py module.
I think we should not use a malloc at all. If a segment cannot be reused
(Heap.free) until all other handles to it is closed. This is a bug in my
code. The easiest solution is to remove the heap malloc all together. In
that case, the allocator for small arrays will just reuse a shared
segment until it is exhausted, and then discared it. Large arrays will
get their own segment. Otherwise we have to do refcounting for open
handles (manually in Windows) and we are back to thread-based cleanup in
the creator.
Sturla Molden
Sturla Molden
Have put up a new version version of the shared memory ndarrays here:
http://folk.uio.no/sturlamo/python/sharedmem-feb13-2009.zip
usage:
import numpy
import sharedmem as shm
arr = shm.empty(...)
arr = shm.ones(...)
arr = shm.zeros(...)
As for the memory leak reported by Gaël previously: This is a bug in
multiprocessing, not in our code. The offending line is 353 in
multiprocessing/forking.py. It shuts down sub-processes abruptly, by
using os._exit for suicide, preventing any clean-up code from executing.
Change this line to "sys.exit(exitcode)" and it works as expected. The
bug has been reported to Jesse Noller.
Regards,
Sturla Molden
Brian Granger
Brian
Sturla Molden
> Does this work without multiprocessing?
Yes.
Multiprocessing is only used for obtaining the page size. I could move
that to Cython as well.
S.M.
Philip Semanchuk
On Feb 13, 2009, at 1:51 PM, Sturla Molden wrote:
>
>> Does this work without multiprocessing?
>
> Yes.
>
> Multiprocessing is only used for obtaining the page size. I could move
> that to Cython as well.
The page size is available in the mmap module.