GeoDjango in Ubuntu 8.10, Segmentation Fault

[GRoby]

Hello,

I created a GeoDjango application that I have been successfully using
with Ubuntu 8.04.

I would like to have as many packages as possible handled by the OS,
so I am testing it with Ubuntu 8.10, with all packages installed
through Synaptic (including PostGIS, PostGreSQL, Mod-Python, Psycopg2,
Django, etc).

In a Python Shell, I am able to perform queries on my spatial models
successfully. In a web browser, I get a plain white screen if any
code is executed that works with spatially enabled models. Otherwise
everything seems to be working (including an OpenLayers MapServer
view, so I know the DB is loaded correctly).

In the error.log file, I get a single error per attempted page view
like:
[Tue Nov 18 16:13:43 2008] [notice] child pid 21074 exit signal
Segmentation fault (11)
[Tue Nov 18 16:16:59 2008] [notice] child pid 21071 exit signal
Segmentation fault (11)
[Tue Nov 18 16:19:08 2008] [notice] child pid 21328 exit signal
Segmentation fault (11)

If I attempt to run the same code in a Python Shell, the code runs
correctly.

Any ideas, I suspect that it could be a permission or Mod-Python
related issue but am not sure where to go from here.

Thanks!

BTW, if anyone wants a copy of the application, it is available from
http://www.coopgis.org, it is the start of an Electric Utility GIS.

[GRoby]

I replaced packages one at a time with ones from Ubuntu 8.04. The
segmentation faults went away when I switched Apache from 2.2.9 to
2.2.8, so I can just lock the package with the Hardy version for now.

> BTW, if anyone wants a copy of the application, it is available fromhttp://www.coopgis.org, it is the start of an Electric Utility GIS.

[Graham Dumpleton]

Based on comments in a separate thread, GeoDango may have issues with
multithreaded configuration. Which Apache MPM was being used for each
Apache version, prefork or worker? The worker MPM uses threads and so
that could be the culprit.

Graham

[GRoby]

Both of them are using the mpm-worker package. I tested the 2.2.9
version with mpm-prefork and mpm-event, they all gave the same
Segmentation Fault errors (rebooting the server in between installing
each).

Greg


On Nov 18, 6:53 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:

[Info Cascade]

I reported a similar problem in October. Since then, I have tested on a
new server and am experiencing the same problem. This one is on an
CentOS 5.2 EC2 machine configured as below... I wonder if noting the
workaround discovered by GRoby will give the developers any clues as to
what is going on. I guess I'll move to Apache 2.2.8 and see if that helps.
Again, it works fine from the command line, but only seg viols when
using Apache.
Justin was good enough to run a test case I created and was unable to
duplicate the error. I'm wondering now about the exact versions of
Apache and other components that might be causing the problem.

> Linux domU-12-31-39-02-94-32 2.6.16.33-xenU #2 SMP Wed Aug 15 17:27:36
> SAST 2007 x86_64 x86_64 x86_64 GNU/Linux
> mod_python.x86_64 3.2.8-3.1 installed
> python.x86_64 2.4.3-19.el5
> installed
The Apache configuration:
> Server version: Apache/2.2.3
> Server built: Nov 10 2007 12:44:14
> Server's Module Magic Number: 20051105:3
> Server loaded: APR 1.2.7, APR-Util 1.2.7
> Compiled using: APR 1.2.7, APR-Util 1.2.7
> Architecture: 64-bit
> Server MPM: Prefork
> threaded: no
> forked: yes (variable process count)
> Server compiled with....
> -D APACHE_MPM_DIR="server/mpm/prefork"
> -D APR_HAS_SENDFILE
> -D APR_HAS_MMAP
> -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
> -D APR_USE_SYSVSEM_SERIALIZE
> -D APR_USE_PTHREAD_SERIALIZE
> -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
> -D APR_HAS_OTHER_CHILD
> -D AP_HAVE_RELIABLE_PIPED_LOGS
> -D DYNAMIC_MODULE_LIMIT=128
> -D HTTPD_ROOT="/etc/httpd"
> -D SUEXEC_BIN="/usr/sbin/suexec"
> -D DEFAULT_PIDLOG="logs/httpd.pid"
> -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
> -D DEFAULT_LOCKFILE="logs/accept.lock"
> -D DEFAULT_ERRORLOG="logs/error_log"
> -D AP_TYPES_CONFIG_FILE="conf/mime.types"
> -D SERVER_CONFIG_FILE="conf/httpd.conf"

[Justin Bronn]

> Based on comments in a separate thread, GeoDango may have issues with
> multithreaded configuration. Which Apache MPM was being used for each
> Apache version, prefork or worker? The worker MPM uses threads and so
> that could be the culprit.

Yes -- the libraries GeoDjango uses, GEOS and GDAL, are not thread
safe. Thus, it is highly recommended to use the prefork version of
Apache.

> In a Python Shell, I am able to perform queries on my spatial models
> successfully. In a web browser, I get a plain white screen if any
> code is executed that works with spatially enabled models.

I created an Ubuntu 8.10 VM to try and test out this problem.
However, the only way that I could reproduce your exact problem
(segfault upon HTTP request) is when using the mpm-worker (threaded)
Apache. When using the prefork, however, I was able to get both
mod_python and mod_wsgi to work with a simple demonstration app [1].
I think mpm-worker is the default when you do `apt-get install
apache2`, so make sure you have it removed, e.g., `apt-get remove
apache2-mpm-worker; apt-get install apache2-mpm-prefork`.

BUT, I found another issue -- while I could login to and browse the
admin interface, whenever I tried to view a geographic model an
exception would get raised (but no segfault) deep in the admin widgets
[2], crashing the app. While this crash occurs in 8.10 it does _not_
happen in my 8.04 VM. Thus, this leads me to believe that it may be a
manifestation of the same troubles you're experiencing. Moreover,
this admin crash happens with _both_ mod_python and mod_wsgi in 8.10
(mod_wsgi configured with `threads=1`).

Needless to say, this behavior has me perplexed at the moment, and due
to my finals I'm not going to have a lot of time to dig in deeper
until next month. Perhaps there's a clash of the libraries that are
linked to Apache and the ones used by the packaged versions of GEOS/
GDAL, or maybe it's caused by Ubuntu's AppArmor (confined to just CUPS
in 8.04) -- but these are just potential possibilities.

> Justin was good enough to run a test case I created and was unable to
> duplicate the error.  I'm wondering now about the exact versions of
> Apache and other components that might be causing the problem.

I still have the CentOS 5.2 VM, and I'll test it out again, but I'm
not sure yet that this could be the same issue as you reported in
October. It's a different distribution that used significantly older
versions (that also worked in my tests).

[1] http://geodjango.org/hg/world
[2] http://code.djangoproject.com/browser/django/trunk/django/contrib/gis/admin/widgets.py#L43

[GRoby]

Justin,

I reran my tests with the latest version of Apache on my 8.10 test
machine. I am no longer getting the Segfaults with either worker or
prefork. I am now getting an Invalid SRS type "wkt" error (that I
assume you are now running into).

I have tried several things (including downgrading PostGreSQL to 8.3.4
from 8.3.5 and switching my GDAL directory to an invalid location but
have not been able to duplicate the Segfault problem. (I had an
invalid GDAL directory specified when I was testing last week, but
switching it back now gives me the expected OGR failure exception).

Let me know whether any further testing that I can do would be useful
(maybe seeing whether compiling GEOS and GDAL works with the newest
Apache version?)?

Thanks, for now the downgraded Apache version is working without any
problems.

Greg

> [2]http://code.djangoproject.com/browser/django/trunk/django/contrib/gis...

[Justin Bronn]

> I reran my tests with the latest version of Apache on my 8.10 test
> machine.  I am no longer getting the Segfaults with either worker or
> prefork.  I am now getting an Invalid SRS type "wkt" error (that I
> assume you are now running into).

Yup that's the error I got: http://code.djangoproject.com/ticket/9694.
I've also got a patch on there that fixes the SRS exception on 8.10 --
I have a vague notion as to why it works, but no definitive answer.
My only guess as to your non-reproducible segfault is that worker was
being used when you thought you had prefork going.

-Justin

[Info Cascade]

FWIW --

I got this working in CentOS5.2 by removing the default (yum installed)
version of apache, and compiling the latest version, 2.2.10.
While I was at it, I compiled the latest version of mod_python.
That was all it took, and now, no more segmentation violations.

Thanks,
Liam

[rcoup]

On Nov 26, 6:07 am, Justin Bronn <jbr...@gmail.com> wrote:
> I created an Ubuntu 8.10 VM to try and test out this problem.
> However, the only way that I could reproduce your exact problem
> (segfault upon HTTP request) is when using the mpm-worker (threaded)
> Apache.  When using the prefork, however, I was able to get both
> mod_python and mod_wsgi to work with a simple demonstration app [1].
> I think mpm-worker is the default when you do `apt-get install
> apache2`, so make sure you have it removed, e.g., `apt-get remove
> apache2-mpm-worker; apt-get install apache2-mpm-prefork`.

See my experiences with Intrepid/apache/GEOS:
http://groups.google.co.nz/group/modwsgi/browse_thread/thread/a488d2551c4c7df0

The minimal WSGI/script (crashes under mod-python too) i can get to
segfault is:
from django.contrib.gis.geos import GEOSGeometry
g = GEOSGeometry('POINT(0 0)')
foo = g.wkt #segfault

Let me know if you want more testing done.

Rob :)

[rcoup]

Posting to geos-devel, I got this reply from Paul Ramsey:
http://lists.osgeo.org/pipermail/geos-devel/2008-December/003800.html

Leading me to: http://sgillies.net/blog/829/shapely-1-0-8/

"The same problem [segfault] could afflict any python package that
uses Ctypes on 64-bit systems without explicitly marking argument and
return types. "

Looking back through the modwsgi thread and this one, everyone's on
64bit, except the CentOS guy who said it worked ok on 32bit but not on
64bit.

Maybe thats the problem with the ctypes+geos bindings in GeoDjango?
I'm running something-pre-1.0, but I can try it on a trunk checkount
to confirm the same symptoms.

Rob :)

[rcoup]

On Dec 3, 1:40 pm, rcoup <robert.c...@gmail.com> wrote:
> Leading me to:http://sgillies.net/blog/829/shapely-1-0-8/
>
> "The same problem [segfault] could afflict any python package that
> uses Ctypes on 64-bit systems without explicitly marking argument and
> return types. "

The relevant changeset in the shapely lib for the fix is:
http://trac.gispython.org/lab/changeset/1145

Rob :)

[GRoby]

I can confirm that all of my seg faults were all on my 64 bit machine
but could not be duplicated when I tried to on my 32 bit test virtual
machines.

[Andre P LeBlanc]

Confirmed also. only happens on my 64-bit machine, and is not related
to apache at all as I can reproduce it faithfully in ipython.

[rcoup]

Hi,

On Dec 3, 5:21 pm, Andre P LeBlanc <andreplebl...@gmail.com> wrote:
> Confirmed also. only happens on my 64-bit machine, and is not related
> to apache at all as I can reproduce it faithfully in ipython.
>

I tried pretty hard to replicate it in the console and couldn't - none
of our unit tests crashed it, and neither would anything else i tried.
But the minute it was running under mod-python/mod-wsgi it segfaulted
with the simplest method called on a GEOSGeometry object.

Rob :)

[Justin Bronn]

> "The same problem [segfault] could afflict any python package that
> uses Ctypes on 64-bit systems without explicitly marking argument and
> return types. "
>
> Looking back through the modwsgi thread and this one, everyone's on
> 64bit, except the CentOS guy who said it worked ok on 32bit but not on
> 64bit.
>
> Maybe thats the problem with the ctypes+geos bindings in GeoDjango?
> I'm running something-pre-1.0, but I can try it on a trunk checkount
> to confirm the same symptoms.

I realized the problems about not making explicit argument return
types, and made _almost_ all of them explicit over a _year_ ago in
r6653 (by adding the `prototypes` directory where all interactions
with C functions were defined). Good to see that Shapely got around
to eventually doing that, rather than the haphazard fixes they were
implementing up to this point that never really addressed the
underlying issues.

The reason I said "almost" is that I purposely did not do this for
routines that return strings allocated within GEOS. If you specified
the response type as `c_char_p` then ctypes would convert to a Python
object (str) when given to a ctypes error check function. In other
words, because I was doing the proper thing and using the error check
function it is not possible to set the response type to `c_char_p`
_and_ get the pointer address to the string because all you got was a
Python str instead. It's important to get the pointer address,
otherwise I could not free the string memory allocated inside the GEOS
library -- in other words, GeoDjango would leak memory every time one
serialized to WKT if I didn't do it this way (that's bad).

While it's not possible to get the pointer address when using
`c_char_p` as the restype, I've since figured out that if you use a
simple _subclass_ of `c_char_p` and use that for the response type I
can get access to both the string value and the pointer address to be
freed.

I've implemented this patch in my minor refactor of the GEOS interface
for 1.1 [1]. I've also created a ticket [2] and I'll attach a patch
with the fix, which I'll commit to trunk and the 1.0.X branch within a
week.

And now I realized why I couldn't reproduce on CentOS -- I was using a
32-bit version; forgot to ask "what architecture", d'oh.

-Justin

[1] http://geodjango.org/hg/gis-geos
[2] http://code.djangoproject.com/ticket/9747

[Andre P LeBlanc]

I had a large table of zip & postal codes which always crashed after
about 20 minutes of geocoding them in the ipython shell, i did the
same thing several times on the same os, but on a 32-bit machine with
no issues. running under mod-python or mod-wsgi on the 64-bit box
also seems to randomly throw 500's, but not in any consistent way.