MMAP with cython fails

[Karthik Sivaraman]

Hi 

Apologies for the long post. Am a Cython n00b.

Am trying to mmap a named shared memory. Am using cython 0.11.2 with freebsd 8.2 and python 2.6.6.

The following is my simple pyx code:

mmap.pyx

========

mmapDone = False

cdef extern from "sys/_null.h":

    cdef enum:

        NULL "NULL"

cdef extern from "sys/mman.h":

    void * mmap(void *, size_t, int, int, int, off_t)

    int shm_open(char *, int, mode_t)

    cdef enum:

        PROT_READ "PROT_READ"

        MAP_SHARED "MAP_SHARED"

        MAP_FAILED "MAP_FAILED"

cdef extern from "sys/fcntl.h":

    cdef enum:

        O_RDONLY "O_RDONLY"

cdef extern from "sys/stat.h":

    cdef enum:

        S_IRUSR "S_IRUSR"

def mmap_shared_area():

    cdef int fd

    cdef int fsize

    cdef void* mmapStart

    print 'Attempting to shm_open....'

    fd = shm_open("/shared_area", O_RDONLY, S_IRUSR)

    if (fd == -1):

        raise Exception('Unable to open Shared Memory. File Descriptor is -1')

        return (0) 

    else:

        print 'SUCCESS fd = %d' %(fd)

    

    print 'Attempting to MMAP...'    

    fsize = 500 

    mmapStart = <void*> mmap(NULL, fsize, PROT_READ, MAP_SHARED, fd, 0)

    if (MAP_FAILED == <int>mmapStart):

        print 'MMAP failed miserably for no reason'

        return (0)

    mmapDone = True

    return (1)

                

After compiling and generating a shared library - mymmap.so, I test the module using:

test.py

=====

#!/usr/local/bin/python

import mymmap as mm

print 'Returned value %d' %(mm.mmap_shared_area())

Output

=====

Attempting to shm_open....

SUCCESS fd = 3

Attempting to MMAP...

MMAP failed miserably for no reason

Returned value 0

An equivalent C implementation succeeds:

shm_client.c

=========

#include <sys/mman.h>

#include <fcntl.h>

#include <sys/stat.h>

#include <stdio.h>

#include <stdlib.h>

#include <errno.h>

#include <unistd.h>

int main()

{

    int fd; 

    fd = shm_open("/prox_staging_area", O_RDONLY, S_IRUSR | S_IWUSR);

    if (-1 == fd) {

        printf("\n Unable to open shared memory. Error number %d Error string %s\n", errno, strerror(errno));

        exit(1);

    }   

    printf("\n Opened Shared memory with fd %d\n", fd);

    void *mmapStart;

    mmapStart = (void*) mmap(NULL, 500, PROT_READ, MAP_SHARED, fd, 0); 

    printf("\n mmapStart is %d\n", mmapStart);

    if (MAP_FAILED == mmapStart)

        printf("\nUnable to mmap\n");

    sleep(1000);

}

Output of C client

============

 Opened Shared memory with fd 3

 mmapStart is 1155530752

Any ideas why the C client succeeds where as the Cython client fails? Am I missing something?

Thanks

Karthik

[Sturla Molden]

Not sure, but on 64-bit sizeof(void*) > sizeof(int), so cast void* to Py_intptr_t instead.

if (<Py_intptr_t>MAP_FAILED == <Py_intptr_t>mmapStar): 

    raise OSError, "mmap failed"

Can't say that is the error causing the problem though, but it is an error in your code.

:-)

By the way, you can just do this:

import mmap

Now your code will work on Windows as well... Not to mention that mmap.mmap (or numpy.memmap) has already been debugged and tested by thousands around the world.

I have never seen a case where the Python overhead when calling mmap really matters. So my best advice is "don't use C".  :-) 

Sturla

Sendt fra min iPad

--
 
---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Sturla Molden]

By the way, if you are messing with shared memory on Cython, 

feel free to borrow from our code here:

https://github.com/sturlamolden/sharedmem-numpy/tree/master/sharedmem

We have weeded out a lot of problems, inluding system-wide memory leaks 

from multiprocessing skipping clean-up code on System V IPC (which you 

seem to use).

Sturla

Sendt fra min iPad

Den 7. juli 2013 kl. 10:51 skrev Karthik Sivaraman <msg....@gmail.com>:

--

[Karthik Sivaraman]

Thanks for the suggestions Sturla.. Will definitely have a look in your project..

I hunted down the problem (with help from my collegues) to a missing type definition. I missed:

ctypedef int off_t

This causes cython compile to interpret offt_t as a PyObject rather than int

Karthik