Segmentation fault on CentOS 6.5 running as WSGI on Apache whilst accessing data URL

[Roald de Wit]

Hi,

I'm experiencing a very obscure segmentation fault. Before submitting a bug report, I'm hoping that someone on this mailing list has either experienced something similar or has an idea where the fault occurs.

Situation:

• Python WSGI script trying to access an OPenDAP data source
• Apache process crashes with a segmentation fault upon trying to access URL (any protocol, even 'file://'). The URL does not need to resolve to experience the segmentation fault.
• No crash when accessing netCDF from file system ('/path/to/netcdffile.nc')
• No crash when running as CGI or from command-line
• No crash on Ubuntu 13.10 running as WSGI (Python 2.7.x)
• No crashes of any of my other WSGI scripts

Environment:

• CentOS 6.5
• Apache 2.2.15
• mod_wsgi 3.2 (running as daemon process)
• Python 2.6.6
• netcdf 4.1.1
  
• python-netCDF 1.0.7 (but any version from 0.6.x till 1.0.7 will crash)
  

Minimal code example:

    from netCDF4 import Dataset
    dset = Dataset('http://....')

Test case:
http://pastebin.com/tgcmq3rJ

Any help finding a solution would be greatly appreciated!

Regards,

Roald

[Jeff Whitaker]

Roald de Wit

January 30, 2014 10:26 PM

---

Roald:  Unfortunately, I have no idea what might be causing this.  Sounds more like a CentOS issue than an issue with netcdf4-python.

-Jeff

[Roald de Wit]

On 01/02/14 02:22, Jeff Whitaker wrote:

Roald:  Unfortunately, I have no idea what might be causing this.  Sounds more like a CentOS issue than an issue with netcdf4-python.

-Jeff

Hi Jeff,

Thanks for your reply. I've dug deeper into the issue and I've managed to narrow down the problem a bit more: it seems to only occur when the script runs as a user without a login shell (apache)!

This is the minimal script needed to trigger the segfault (segfault-cmd-line.py):

#! /bin/env python

from netCDF4 import Dataset
dset = Dataset('http://nomads.ncep.noaa.gov:9090/dods/gfs/gfs20140120/gfs_00z')

When when I run it like this:

sudo -u apache /usr/bin/python segfault-cmd-line.py

I get the following output in /var/log/messages:

Feb  3 11:10:26 localhost kernel: python[657]: segfault at 0 ip 0000003d5f486eb4 sp 00007fff57428840 error 4 in libnetcdf.so.6.0.0[3d5f400000+ea000]
Feb  3 11:10:26 localhost abrtd: Directory 'ccpp-2014-02-03-11:10:26-657' creation detected
Feb  3 11:10:26 localhost abrt[658]: Saved core dump of pid 657 (/usr/bin/python) to /var/spool/abrt/ccpp-2014-02-03-11:10:26-657 (10428416 bytes)
Feb  3 11:10:27 localhost abrtd: Interpreter crashed, but no packaged script detected: '/usr/bin/python segfault-cmd-line.py'
Feb  3 11:10:27 localhost abrtd: 'post-create' on '/var/spool/abrt/ccpp-2014-02-03-11:10:26-657' exited with 1
Feb  3 11:10:27 localhost abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2014-02-03-11:10:26-657'

Below is the gdb backtrace. Can you determine from this whether it may be a netcdf4-python issue or one in libnetcdf?

Thanks, Roald

$ sudo -u apache gdb /usr/bin/python
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/python...Reading symbols from /usr/lib/debug/usr/bin/python.debug...done.
done.
(gdb) run segfault-cmd-line.py
Starting program: /usr/bin/python segfault-cmd-line.py
[Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault.
oc_close (conn=0) at oc.c:102
102        OCVERIFY(OCstate*,state,conn);
Missing separate debuginfos, use: debuginfo-install libffi-3.0.5-3.2.el6.x86_64 libidn-1.18-2.el6.x86_64
(gdb) bt
#0  oc_close (conn=0) at oc.c:102
#1  0x0000003d5f481096 in freeNCDRNO3 (drno=0xa7ac20) at ncdap3.c:399
#2  0x0000003d5f48234e in nc3d_open (path=<value optimized out>, mode=<value optimized out>, ncidp=0xa7abf0) at ncdap3.c:310
#3  0x0000003d5f44c573 in l4nc_open_file (path=0x97c3fc "http://nomads.ncep.noaa.gov:9090/dods/gfs/gfs20140120/gfs_00z", mode=0, basepe=0, chunksizehintp=0x0, use_parallel=<value optimized out>,
    comm=<value optimized out>, info=0, ncidp=0x7fffffffdffc) at nc4file.c:2338
#4  0x0000003d5f499108 in nc4d_open_file (path=0x97c3fc "http://nomads.ncep.noaa.gov:9090/dods/gfs/gfs20140120/gfs_00z", mode=0, basepe=0, chunksizehintp=0x0, use_parallel=0, comm=0, info=0, ncidp=
    0x7fffffffe0fc) at ncdap4.c:122
#5  0x0000003d5f449d7f in nc_open (path=<value optimized out>, mode=<value optimized out>, ncidp=<value optimized out>) at nc4file.c:2400
#6  0x00007ffff1e2f5e7 in __pyx_pf_7netCDF4_7Dataset___init__ (__pyx_v_self=<netCDF4.Dataset at remote 0x971cd0>, __pyx_args=<value optimized out>, __pyx_kwds=<value optimized out>) at netCDF4.c:16164
#7  __pyx_pw_7netCDF4_7Dataset_1__init__ (__pyx_v_self=<netCDF4.Dataset at remote 0x971cd0>, __pyx_args=<value optimized out>, __pyx_kwds=<value optimized out>) at netCDF4.c:15804
#8  0x0000003d62c94298 in type_call (type=<value optimized out>, args=('http://nomads.ncep.noaa.gov:9090/dods/gfs/gfs20140120/gfs_00z',), kwds=0x0) at Objects/typeobject.c:742
#9  0x0000003d62c43c63 in PyObject_Call (func=<type at remote 0x7ffff204e020>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:2492
#10 0x0000003d62cd4f74 in do_call (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:4012
#11 call_function (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:3817
#12 PyEval_EvalFrameEx (f=<value optimized out>, throwflag=<value optimized out>) at Python/ceval.c:2453
#13 0x0000003d62cd7657 in PyEval_EvalCodeEx (co=0x7ffff7f00030, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0,
    defcount=0, closure=0x0) at Python/ceval.c:3044
#14 0x0000003d62cd7732 in PyEval_EvalCode (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>) at Python/ceval.c:545
#15 0x0000003d62cf1bac in run_mod (mod=<value optimized out>, filename=<value optimized out>, globals=
    {'__builtins__': <module at remote 0x7ffff7fb2868>, '__file__': 'segfault-cmd-line.py', 'Dataset': <type at remote 0x7ffff204e020>, '__name__': '__main__', '__package__': None, '__doc__': None}, locals=
    {'__builtins__': <module at remote 0x7ffff7fb2868>, '__file__': 'segfault-cmd-line.py', 'Dataset': <type at remote 0x7ffff204e020>, '__name__': '__main__', '__package__': None, '__doc__': None},
    flags=<value optimized out>, arena=<value optimized out>) at Python/pythonrun.c:1358
#16 0x0000003d62cf1c80 in PyRun_FileExFlags (fp=0x6860a0, filename=0x7fffffffe948 "segfault-cmd-line.py", start=<value optimized out>, globals=
    {'__builtins__': <module at remote 0x7ffff7fb2868>, '__file__': 'segfault-cmd-line.py', 'Dataset': <type at remote 0x7ffff204e020>, '__name__': '__main__', '__package__': None, '__doc__': None}, locals=
    {'__builtins__': <module at remote 0x7ffff7fb2868>, '__file__': 'segfault-cmd-line.py', 'Dataset': <type at remote 0x7ffff204e020>, '__name__': '__main__', '__package__': None, '__doc__': None},
    closeit=1, flags=0x7fffffffe5f0) at Python/pythonrun.c:1344
#17 0x0000003d62cf316c in PyRun_SimpleFileExFlags (fp=0x6860a0, filename=0x7fffffffe948 "segfault-cmd-line.py", closeit=1, flags=0x7fffffffe5f0) at Python/pythonrun.c:948
#18 0x0000003d62cff8a2 in Py_Main (argc=<value optimized out>, argv=<value optimized out>) at Modules/main.c:618
#19 0x0000003d5601ed1d in __libc_start_main (main=0x400710 <main>, argc=2, ubp_av=0x7fffffffe718, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=
    0x7fffffffe708) at libc-start.c:226
#20 0x0000000000400649 in _start ()
(gdb)


 

[Roald de Wit]

Thanks to strace, it has become clear what is happening:

access("/var/www/.dodsrc", R_OK)        = -1 ENOENT (No such file or directory)
open("/var/www/.dodsrc", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EACCES (Permission denied)
open("/var/www/.dodsrc", O_RDONLY)      = -1 ENOENT (No such file or directory)
--- SIGSEGV (Segmentation fault) @ 0 (0) ---

When I create the .dodsrc file (sudo touch /var/www/.dodsrc), I get no segfault. It seems that the code for accessing the .dodsrc is in the libnetcdf.so file.