[Django] #32600: Multi geometries cause Segmentation fault 11 on Macbook with M1 CPU
Django
-------------------------------------+-------------------------------------
Reporter: Aapo | Owner: nobody
Rista |
Type: Bug | Status: new
Component: GIS | Version: 3.2
Severity: Normal | Keywords: MultiPoint,
| MultiLinestring, MultiPolygon,
Triage Stage: | Segmentation fault, GIS, GeoDjango
Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
I got new Macbook Air (M1, 2020) and when I'm trying to use a project
transferred from previous Intel Macbook Pro, I get Segmentation fault 11
in functions that use MultiPoint, MultiLinestring or MultiPolygon.
How to reproduce:
Create a virtualenv, a Django project and start the shell:
{{{
cd /tmp/ && python3 -m venv venv multisegfault && source
multisegfault/bin/activate && pip install django && django-admin
startproject segfaulttest && cd segfaulttest && python manage.py shell
}}}
Copy paste this line to the shell:
{{{
from django.contrib.gis.geos import MultiLineString; MultiLineString()
}}}
You should get **Segmentation fault: 11**.
It may be possible that **GEOSGeom_createCollection_r** causes crash, but
I don't know how to try it outside Django.
Libraries are installed using HomeBrew and they have versions:
Django 3.2rc1 and 3.1.7
$ gdal-config --version
3.2.2
$ geos-config --version
3.9.1
$ python -V
Python 3.9.2
# SELECT PostGIS_full_version();
POSTGIS="3.1.1 aaf4c79" [EXTENSION] PGSQL="130" GEOS="3.9.1-CAPI-1.14.2"
PROJ="7.2.1" LIBXML="2.9.10" LIBJSON="0.15" LIBPROTOBUF="1.3.3"
WAGYU="0.5.0 (Internal)"
Everything is good on Intel Macbook Pro (same library versions) and on
Ubuntu 20.04 (default versions).
--
Ticket URL: <https://code.djangoproject.com/ticket/32600>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: GIS | Version: 3.2
Severity: Normal | Resolution: duplicate
Keywords: MultiPoint, | Triage Stage:
MultiLinestring, MultiPolygon, | Unreviewed
Segmentation fault, GIS, |
GeoDjango |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => duplicate
Comment:
Duplicate of #32544.
--
Ticket URL: <https://code.djangoproject.com/ticket/32600#comment:1>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Severity: Normal | Resolution: duplicate
Keywords: MultiPoint, | Triage Stage:
MultiLinestring, MultiPolygon, | Unreviewed
Segmentation fault, GIS, |
GeoDjango |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak):
Can you enable `faulthandler` and check the origin of a segmentation
fault?
`catchsegv python -X faulthandler script.py`
--
Ticket URL: <https://code.djangoproject.com/ticket/32600#comment:2>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Severity: Normal | Resolution: duplicate
Keywords: MultiPoint, | Triage Stage:
MultiLinestring, MultiPolygon, | Unreviewed
Segmentation fault, GIS, |
GeoDjango |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Aapo Rista):
I'm still figuring out how to run catchsegv on Mac Os, but here are the
rest:
{{{
$ python -X faulthandler manage.py shell
Python 3.9.2 (default, Mar 24 2021, 20:21:37)
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.contrib.gis.geos import MultiLineString; MultiLineString()
Fatal Python error: Segmentation fault
Current thread 0x00000001009b7d40 (most recent call first):
File "/path/to/Project/venv/lib/python3.9/site-
packages/django/contrib/gis/geos/prototypes/threadsafe.py", line 49 in
__call__
File "/path/to/Project/venv/lib/python3.9/site-
packages/django/contrib/gis/geos/libgeos.py", line 152 in __call__
File "/path/to/Project/venv/lib/python3.9/site-
packages/django/contrib/gis/geos/collections.py", line 56 in
_create_collection
File "/path/to/Project/venv/lib/python3.9/site-
packages/django/contrib/gis/geos/collections.py", line 36 in __init__
File "<console>", line 1 in <module>
File
"/opt/homebrew/Cellar/python@3.9/3.9.2_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py",
line 90 in runcode
File
"/opt/homebrew/Cellar/python@3.9/3.9.2_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py",
line 74 in runsource
File
"/opt/homebrew/Cellar/python@3.9/3.9.2_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py",
line 258 in push
File
"/opt/homebrew/Cellar/python@3.9/3.9.2_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py",
line 232 in interact
File
"/opt/homebrew/Cellar/python@3.9/3.9.2_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/code.py",
line 301 in interact
File "/path/to/Project/venv/lib/python3.9/site-
packages/django/core/management/commands/shell.py", line 82 in python
File "/path/to/Project/venv/lib/python3.9/site-
packages/django/core/management/commands/shell.py", line 100 in handle
File "/path/to/Project/venv/lib/python3.9/site-
packages/django/core/management/base.py", line 398 in execute
File "/path/to/Project/venv/lib/python3.9/site-
packages/django/core/management/base.py", line 354 in run_from_argv
File "/path/to/Project/venv/lib/python3.9/site-
packages/django/core/management/__init__.py", line 413 in execute
File "/path/to/Project/venv/lib/python3.9/site-
packages/django/core/management/__init__.py", line 419 in
execute_from_command_line
File "/path/to/Project/django_project/mydata/manage.py", line 17 in main
File "/path/to/Project/django_project/mydata/manage.py", line 21 in
<module>
Segmentation fault: 11
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32600#comment:3>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Severity: Normal | Resolution: duplicate
Keywords: MultiPoint, | Triage Stage:
MultiLinestring, MultiPolygon, | Unreviewed
Segmentation fault, GIS, |
GeoDjango |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Aapo Rista):
Ok, I found something in Console's Crash Reports (I left out a few hundred
of lines). See GEOSGeom_createCollection_r there.
Again, I think this is not GEOS/GDAL/Django **version** related issue (as
assumed in #32544), but a bug in GEOS/Python/libgeos or something.
{{{
Process: Python [86891]
Path:
/opt/homebrew/*/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python
Identifier: Python
Version: 3.9.2 (3.9.2)
Code Type: ARM-64 (Native)
Parent Process: bash [86834]
Responsible: Terminal [1531]
User ID: 502
Date/Time: 2021-03-27 16:48:34.731 +0200
OS Version: macOS 11.2.3 (20D91)
Report Version: 12
Anonymous UUID: 1F81D35A-CC9F-662C-B696-4BC56365499D
Sleep/Wake UUID: 24C66F23-05FD-4B16-BF9A-9BEF835E38FA
Time Awake Since Boot: 450000 seconds
Time Since Wake: 6200 seconds
System Integrity Protection: enabled
Crashed Thread: 0 Dispatch queue: com.apple.main-thread
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x000000016dda4000
Exception Note: EXC_CORPSE_NOTIFY
Termination Signal: Segmentation fault: 11
Termination Reason: Namespace SIGNAL, Code 0xb
Terminating Process: exc handler [86891]
VM Regions Near 0x16dda4000:
Stack 16cda4000-16dda4000 [ 16.0M] rw-/rwx
SM=ALI thread 0
-->
Submap 180000000-190000000 [256.0M] r--/r--
SM=SHM machine-wide VM submap
Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libgeos_c.1.dylib 0x000000010528873c
GEOSGeom_createCollection_r + 88
1 libgeos_c.1.dylib 0x000000010528872c
GEOSGeom_createCollection_r + 72
2 libffi.dylib 0x0000000194e64050 ffi_call_SYSV +
80
3 libffi.dylib 0x0000000194e6c9d8 ffi_call_int +
944
4 _ctypes.cpython-39-darwin.so 0x0000000103f38684
_ctypes_callproc + 856
5 _ctypes.cpython-39-darwin.so 0x0000000103f330f4 PyCFuncPtr_call
+ 220
6 org.python.python 0x00000001024cbbe0 _PyObject_Call
+ 128
7 org.python.python 0x00000001025c2030
_PyEval_EvalFrameDefault + 40440
8 org.python.python 0x00000001025b7290
_PyEval_EvalCode + 436
9 org.python.python 0x00000001024cbe7c
_PyFunction_Vectorcall + 364
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32600#comment:4>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Severity: Normal | Resolution: invalid
Keywords: MultiPoint, | Triage Stage:
MultiLinestring, MultiPolygon, | Unreviewed
Segmentation fault, GIS, |
GeoDjango |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* resolution: duplicate => invalid
--
Ticket URL: <https://code.djangoproject.com/ticket/32600#comment:5>
Django
on macOS arm64 (M1)
-------------------------------------+-------------------------------------
Reporter: Aapo Rista | Owner: nobody
Component: GIS | Version: dev
Severity: Normal | Resolution:
Keywords: Polygon, | Triage Stage:
MultiPoint, MultiLineString, | Unreviewed
MultiPolygon, Segmentation fault, |
GIS, GeoDjango, GEOS, macOS, M1, |
arm64 |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: closed => new
* cc: Brenton Partridge (added)
* version: 3.2 => dev
* keywords:
MultiPoint, MultiLinestring, MultiPolygon, Segmentation fault, GIS,
GeoDjango
=>
Polygon, MultiPoint, MultiLineString, MultiPolygon, Segmentation
fault, GIS, GeoDjango, GEOS, macOS, M1, arm64
* has_patch: 0 => 1
* resolution: invalid =>
Comment:
I can confirm that this is indeed a Django bug, and I was able to
replicate segfaults on Django's main branch, so it appears the GEOS 3.9
update in https://code.djangoproject.com/ticket/32544 did not fix this.
It's also been replicated in e.g.
https://github.com/libgeos/geos/issues/528.
It appears that in the current GEOS integration, argument signatures were
omitted in favor of automatically casting arguments for a few polygon-
related API calls. While this appears to have worked fine on x86 and x64,
on macOS arm64 (for native M1 processors), it causes seemingly random data
to be provided in place of arguments after the first, reliably causing
segmentation faults whenever a Polygon, MultiPolygon, or MultiLineString
is constructed. Underlying CFFI internals may be less lenient to
unspecified arguments in Apple's implementation than on other platforms.
Moving to explicit signatures works fine, and I see no reason having
explicit signatures should cause problems on other platforms (though we
should ensure CI passes).
I've submitted a PR at https://github.com/django/django/pull/15214.
All GIS tests now pass, and the changes are fully covered by test_geos.
The deeper issue in testing is that CI does not run on Apple hardware, so
it will be difficult to prevent regressions. However, the code changes are
small and well-documented, and I'd suggest that Apple/arm64 CI will be
important but not urgent.
For organizations using older versions of Django, I've also made a monkey-
patch available at
https://gist.github.com/bpartridge/26a11b28415d706bfb9993fc28767d68 which
can be added in any initialization code (or even called from settings.py)
before Polygon objects are created. When applied in test settings against
the main branch, all GIS tests pass as well; functionality is identical to
the PR above.
PR should be ready for review; happy to make changes as requested! Happy
holidays - and hopefully this brings some holiday cheer to developers who
might be seeing segfaults on their newly gifted machines!
--
Ticket URL: <https://code.djangoproject.com/ticket/32600#comment:6>
Django
on macOS arm64 (M1)
-------------------------------------+-------------------------------------
| Partridge
Type: Bug | Status: assigned
Component: GIS | Version: 4.0
Severity: Release blocker | Resolution:
Keywords: Polygon, | Triage Stage: Accepted
MultiPoint, MultiLineString, |
MultiPolygon, Segmentation fault, |
GIS, GeoDjango, GEOS, macOS, M1, |
arm64 |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Brenton Partridge
* status: new => assigned
* version: dev => 4.0
* severity: Normal => Release blocker
* stage: Unreviewed => Accepted
Comment:
Thanks for the investigation! It ends with `Segmentation fault` so IMO it
qualifies for a backport to Django 4.0 as a crashing bug.
--
Ticket URL: <https://code.djangoproject.com/ticket/32600#comment:7>
Django
on macOS arm64 (M1)
-------------------------------------+-------------------------------------
Reporter: Aapo Rista | Owner: Brenton
| Partridge
Type: Bug | Status: assigned
Component: GIS | Version: 4.0
Severity: Release blocker | Resolution:
MultiPoint, MultiLineString, | checkin
MultiPolygon, Segmentation fault, |
GIS, GeoDjango, GEOS, macOS, M1, |
arm64 |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/32600#comment:8>
Django
on macOS arm64 (M1)
-------------------------------------+-------------------------------------
Reporter: Aapo Rista | Owner: Brenton
| Partridge
Component: GIS | Version: 4.0
Severity: Release blocker | Resolution: fixed
Keywords: Polygon, | Triage Stage: Ready for
MultiPoint, MultiLineString, | checkin
MultiPolygon, Segmentation fault, |
GIS, GeoDjango, GEOS, macOS, M1, |
arm64 |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"19fb838803f63eef0726a370050443b693f109be" 19fb8388]:
{{{
#!CommitTicketReference repository=""
revision="19fb838803f63eef0726a370050443b693f109be"
Fixed #32600 -- Fixed Geometry collections and Polygon segmentation fault
on macOS ARM64.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32600#comment:9>
Django
on macOS arm64 (M1)
-------------------------------------+-------------------------------------
Reporter: Aapo Rista | Owner: Brenton
| Partridge
Type: Bug | Status: closed
Component: GIS | Version: 4.0
Severity: Release blocker | Resolution: fixed
Keywords: Polygon, | Triage Stage: Ready for
MultiPoint, MultiLineString, | checkin
MultiPolygon, Segmentation fault, |
GIS, GeoDjango, GEOS, macOS, M1, |
arm64 |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"b85ceaaba6e3de8419a8604cde426914a7915e13" b85ceaab]:
{{{
#!CommitTicketReference repository=""
revision="b85ceaaba6e3de8419a8604cde426914a7915e13"
[4.0.x] Fixed #32600 -- Fixed Geometry collections and Polygon
segmentation fault on macOS ARM64.
Backport of 19fb838803f63eef0726a370050443b693f109be from main
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32600#comment:10>