Problem serving netCDF files

[Derek Broughton]

I have tried to serve up a number of netCDF files with both version 2 and
version 3 of pydap, with almost no success.

My trivial file (<300KB) works beautifully , but I have 6 oceanographic
datasets, each containing a year of daily data for things like temperature,
salinity and current velocity, which run from 1.5GB to 4GB in size and any
attempt to do anything with them returns errors like:

Error {
code = 302;
message = "Traceback (most recent call last):
File \"/usr/local/lib/python2.6/dist-packages/Pydap-3.0.rc.5-
py2.6.egg/pydap/handlers/lib.py\", line 64, in __call__
dataset = self.parse_constraints(environ)
File \"/usr/local/lib/python2.6/dist-packages/pydap.handlers.netcdf-0.4.6-
py2.6.egg/pydap/handlers/netcdf/__init__.py\", line 63, in parse_constraints
raise OpenFileError(message)
OpenFileError: 'Unable to open file
/home/derek/pokm/OceanData/data/spstd_1d_20040101_20050101_grid_U.nc.'
";
}

Are they simply too large to be usable with Pydap?
--
derek

[Roberto De Almeida]

Hi, Derek!

In theory you should be able to serve infinite files with pydap, since netcdf files are read in blocks of configurable size. Which netcdf module are you using -- pupynere, pynetcdf, Scientific.IO.NetCDF or PyNIO? Also, are you sure the files are readable by the user running the server process?

Thanks,
--Rob

--
Dr. Roberto De Almeida
http://dealmeida.net/
http://lattes.cnpq.br/1858859813771449
:wq

[Derek Broughton]

I'm using pupynere (I believe), but figured I'd try pynetcdf too. And if that
doesn't work, I'll try the others.

All of my files have the same permissions, (-rwxr-xr-x), are owned by me, and
I'm running the server from my own account, so permissions shouldn't be an
issue (and the small file works).

--
derek

[Roberto De Almeida]

Pupynere is a pure Python netcdf reader, and it has some problems with large files (specially on Windows). If you install pynetcdf it should be picked up automatically by the netcdf handler, you just need to restart the server.

Cheers,
--Rob

[Derek Broughton]

How can I tell which library is being used to read netCDF?

I think I might just install this on a VM so that I have a little more control
of the environment. Right now, I have three different Pythons (2.4,2.5 and
2.6) and some python packages from pypi and others from Ubuntu.

I'm a little concerned that what I _think_ it's running is not what it's
_actually_ running, but I see nothing to suggest the netCDF library is even
being accessed.

derek

--
derek

[Derek Broughton]

I don't actually seem to be getting the pynetcdf reader, so that's part of my
problem.

I commented out the error handler where it tries to open the file in
pydap/handlers/netcdf/__init__.py and got the following message. I beg you
Roberto, DON'T trap errors like this. If all you're going to do is "raise" a
different error message, it only obscures the problem. In 30 years of program
development, I haven't found an issue to get me excited like this one :-)

Now to get the pynetcdf reader properly installed and see if it works
better...

Error {
code = -1;

message = "Traceback (most recent call last):
File \"/usr/local/lib/python2.6/dist-packages/Pydap-3.0.rc.5-
py2.6.egg/pydap/handlers/lib.py\", line 64, in __call__
dataset = self.parse_constraints(environ)
File \"/usr/local/lib/python2.6/dist-packages/pydap.handlers.netcdf-0.4.6-

py2.6.egg/pydap/handlers/netcdf/__init__.py\", line 59, in parse_constraints
fp = nc(self.filepath)
File \"build/bdist.linux-i686/egg/pupynere.py\", line 159, in __init__
self._read()
File \"build/bdist.linux-i686/egg/pupynere.py\", line 386, in _read
self._read_var_array()
File \"build/bdist.linux-i686/egg/pupynere.py\", line 469, in
_read_var_array
mm = mmap(self.fp.fileno(), begin+self._recs*self._recsize,
access=ACCESS_READ)
OverflowError: memory mapped size must be positive
";
}

derek

[Derek Broughton]

Having managed to get pynetcdf installed properly, I can get some files served,
intermittently - that's a worry.

I get this error, commonly, and it aborts the server:

/usr/local/lib/python2.6/dist-packages/pynetcdf-0.7-py2.6-linux-
i686.egg/pynetcdf/NetCDF.py:143: DeprecationWarning: PyArray_FromDims: use
PyArray_SimpleNew.
/usr/local/lib/python2.6/dist-packages/pynetcdf-0.7-py2.6-linux-
i686.egg/pynetcdf/NetCDF.py:143: DeprecationWarning:
PyArray_FromDimsAndDataAndDescr: use PyArray_NewFromDescr.
*** glibc detected *** /usr/bin/python2.6: free(): invalid pointer: 0x09581098
***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6[0xb76eaff1]
/lib/tls/i686/cmov/libc.so.6[0xb76ec6f2]
/lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0xb76ef79d]
/home/derek/.python-eggs/pynetcdf-0.7-py2.6-linux-i686.egg-
tmp/pynetcdf/_netcdf.so[0xb681226a]
/usr/bin/python2.6[0x816120c]
/usr/bin/python2.6(PyDict_Clear+0xf4)[0x808fe44]
/usr/bin/python2.6[0x808feb1]
/usr/bin/python2.6[0x8106e1e]
/usr/bin/python2.6(_PyObject_GC_Malloc+0xcb)[0x810745b]
/usr/bin/python2.6(_PyObject_GC_New+0x15)[0x81074c5]
/usr/bin/python2.6[0x80a5bec]
/usr/bin/python2.6(PyObject_GetIter+0x6c)[0x8064d5c]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x907)[0x80d8767]
/usr/bin/python2.6(PyEval_EvalCodeEx+0x7d2)[0x80dddf2]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x4354)[0x80dc1b4]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x5524)[0x80dd384]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x5524)[0x80dd384]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x5524)[0x80dd384]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x5524)[0x80dd384]
/usr/bin/python2.6(PyEval_EvalCodeEx+0x7d2)[0x80dddf2]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x4354)[0x80dc1b4]
/usr/bin/python2.6(PyEval_EvalCodeEx+0x7d2)[0x80dddf2]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x4354)[0x80dc1b4]
/usr/bin/python2.6(PyEval_EvalCodeEx+0x7d2)[0x80dddf2]
/usr/bin/python2.6[0x816014c]
/usr/bin/python2.6(PyObject_Call+0x4a)[0x806120a]
/usr/bin/python2.6[0x80684ac]
/usr/bin/python2.6(PyObject_Call+0x4a)[0x806120a]
/usr/bin/python2.6[0x80aea8e]
/usr/bin/python2.6[0x80aa165]
/usr/bin/python2.6(PyObject_Call+0x4a)[0x806120a]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x30b9)[0x80daf19]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x5524)[0x80dd384]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x5524)[0x80dd384]
/usr/bin/python2.6[0x815de2f]
/usr/bin/python2.6[0x80824f2]
/usr/bin/python2.6[0x808296c]
/usr/bin/python2.6[0x80aa165]
/usr/bin/python2.6(PyObject_Call+0x4a)[0x806120a]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x3ddc)[0x80dbc3c]
/usr/bin/python2.6(PyEval_EvalCodeEx+0x7d2)[0x80dddf2]
/usr/bin/python2.6[0x816014c]
/usr/bin/python2.6(PyObject_Call+0x4a)[0x806120a]
/usr/bin/python2.6[0x80684ac]
/usr/bin/python2.6(PyObject_Call+0x4a)[0x806120a]
/usr/bin/python2.6[0x80aea8e]
/usr/bin/python2.6[0x80aa165]
/usr/bin/python2.6(PyObject_Call+0x4a)[0x806120a]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x3ddc)[0x80dbc3c]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x5524)[0x80dd384]
/usr/bin/python2.6(PyEval_EvalCodeEx+0x7d2)[0x80dddf2]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x4354)[0x80dc1b4]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x5524)[0x80dd384]
/usr/bin/python2.6(PyEval_EvalCodeEx+0x7d2)[0x80dddf2]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x4354)[0x80dc1b4]
/usr/bin/python2.6(PyEval_EvalCodeEx+0x7d2)[0x80dddf2]
/usr/bin/python2.6[0x816014c]
/usr/bin/python2.6(PyObject_Call+0x4a)[0x806120a]
/usr/bin/python2.6[0x80684ac]
/usr/bin/python2.6(PyObject_Call+0x4a)[0x806120a]
/usr/bin/python2.6(PyEval_EvalFrameEx+0x3ddc)[0x80dbc3c]
/usr/bin/python2.6(PyEval_EvalCodeEx+0x7d2)[0x80dddf2]
/usr/bin/python2.6[0x816014c]
======= Memory map: ========
08048000-0821b000 r-xp 00000000 08:05 4308 /usr/bin/python2.6
0821b000-0821c000 r--p 001d2000 08:05 4308 /usr/bin/python2.6
0821c000-0826a000 rw-p 001d3000 08:05 4308 /usr/bin/python2.6
0826a000-08273000 rw-p 00000000 00:00 0
08979000-098a4000 rw-p 00000000 00:00 0 [heap]
b1200000-b1221000 rw-p 00000000 00:00 0
b1221000-b1300000 ---p 00000000 00:00 0
b133f000-b14c0000 rw-p 00000000 00:00 0
b14c0000-b14ee000 r-xp 00000000 08:05 172169 /usr/lib/python2.6/lib-
dynload/pyexpat.so
b14ee000-b14f0000 r--p 0002d000 08:05 172169 /usr/lib/python2.6/lib-
dynload/pyexpat.so
b14f0000-b14f1000 rw-p 0002f000 08:05 172169 /usr/lib/python2.6/lib-
dynload/pyexpat.so
b14f1000-b14f2000 rw-p 00000000 00:00 0
b14f2000-b14f3000 ---p 00000000 00:00 0
b14f3000-b1cf3000 rw-p 00000000 00:00 0
b1cf3000-b1cf4000 ---p 00000000 00:00 0
b1cf4000-b24f4000 rw-p 00000000 00:00 0
b24f4000-b24f5000 ---p 00000000 00:00 0
b24f5000-b2cf5000 rw-p 00000000 00:00 0
b2cf5000-b2cf6000 ---p 00000000 00:00 0
b2cf6000-b34f6000 rw-p 00000000 00:00 0
b34f6000-b34f7000 ---p 00000000 00:00 0
b34f7000-b3cf7000 rw-p 00000000 00:00 0
b3cf7000-b3cf8000 ---p 00000000 00:00 0
b3cf8000-b44f8000 rw-p 00000000 00:00 0
b44f8000-b44f9000 ---p 00000000 00:00 0
b44f9000-b4cf9000 rw-p 00000000 00:00 0
b4cf9000-b4cfa000 ---p 00000000 00:00 0
b4cfa000-b54fa000 rw-p 00000000 00:00 0
b54fa000-b54fb000 ---p 00000000 00:00 0
b54fb000-b5cfb000 rw-p 00000000 00:00 0
b5cfb000-b5cfc000 ---p 00000000 00:00 0
b5cfc000-b64fc000 rw-p 00000000 00:00 0
b64fc000-b6506000 r-xp 00000000 08:05 45604
/lib/tls/i686/cmov/libnss_files-2.10.1.so
b6506000-b6507000 r--p 00009000 08:05 45604
/lib/tls/i686/cmov/libnss_files-2.10.1.so
b6507000-b6508000 rw-p 0000a000 08:05 45604
/lib/tls/i686/cmov/libnss_files-2.10.1.so
b6508000-b653a000 r-xp 00000000 08:05 271494 /usr/lib/libnetcdf.so.4.0.0
b653a000-b653b000 r--p 00031000 08:05 271494 /usr/lib/libnetcdf.so.4.0.0
b653b000-b653c000 rw-p 00032000 08:05 271494 /usr/lib/libnetcdf.so.4.0.0
b653c000-b653d000 rw-p 00000000 00:00 0
b653d000-b65a0000 r-xp 00000000 08:05 332771
/usr/lib/pyshared/python2.6/matplotlib/backends/_backend_agg.so
b65a0000-b65a1000 r--p 00063000 08:05 332771
/usr/lib/pyshared/python2.6/matplotlib/backends/_backend_agg.so
b65a1000-b65a2000 rw-p 00064000 08:05 332771
/usr/lib/pyshared/python2.6/matplotlib/backends/_backend_agg.so
b65a2000-b65c8000 r-xp 00000000 08:05 4236 /usr/lib/libpng12.so.0.37.0
b65c8000-b65c9000 r--p 00025000 08:05 4236 /usr/lib/libpng12.so.0.37.0
b65c9000-b65ca000 rw-p 00026000 08:05 4236 /usr/lib/libpng12.so.0.37.0
b65ca000-b65ee000 r-xp 00000000 08:05 332737
/usr/lib/pyshared/python2.6/matplotlib/_png.so
b65ee000-b65ef000 ---p 00024000 08:05 332737
/usr/lib/pyshared/python2.6/matplotlib/_png.so
b65ef000-b65f0000 r--p 00024000 08:05 332737
/usr/lib/pyshared/python2.6/matplotlib/_png.so
b65f0000-b65f1000 rw-p 00025000 08:05 332737
/usr/lib/pyshared/python2.6/matplotlib/_png.so
b65f1000-b6633000 r-xp 00000000 08:05 332739
/usr/lib/pyshared/python2.6/matplotlib/_image.so
b6633000-b6634000 ---p 00042000 08:05 332739
/usr/lib/pyshared/python2.6/matplotlib/_image.so
b6634000-b6635000 r--p 00042000 08:05 332739
/usr/lib/pyshared/python2.6/matplotlib/_image.so
b6635000-b6637000 rw-p 00043000 08:05 332739
/usr/lib/pyshared/python2.6/matplotlib/_image.so
b6637000-b66b1000 r-xp 00000000 08:05 6974
/usr/lib/libfreetype.so.6.3.20
b66b1000-b66b5000 r--p 00079000 08:05 6974
/usr/lib/libfreetype.so.6.3.20
b66b5000-b66b6000 rw-p 0007d000 08:05 6974
/usr/lib/libfreetype.so.6.3.20
b66c0000-b66cc000 r-xp 00000000 08:05 172120 /usr/lib/python2.6/lib-
dynload/parser.so
b66cc000-b66cd000 r--p 0000b000 08:05 172120 /usr/lib/python2.6/lib-
dynload/parser.so
b66cd000-b66ce000 rw-p 0000c000 08:05 172120 /usr/lib/python2.6/lib-
dynload/parser.soAborted

[Roberto De Almeida]

I recommend using virtualenv to create a separate environment for pydap:

  http://pydap.org/server.html#running-pydap-with-apache

--Rob

[Roberto De Almeida]

On Tue, Nov 10, 2009 at 9:02 PM, Derek Broughton <de...@pointerstop.ca> wrote:

I commented out the error handler where it tries to open the file in
pydap/handlers/netcdf/__init__.py  and got the following message.  I beg you
Roberto, DON'T trap errors like this.  If all you're going to do is "raise" a
different error message, it only obscures the problem.  In 30 years of program
development, I haven't found an issue to get me excited like this one :-)

The server traps errors because you don't want it to stop running just because you installed a handler with problems. You can supreme this behavior by setting

  x-wsgiorg.throw_errors = 1

in the server.ini file.

Best,
--Rob

 

[Roberto De Almeida]

By the way, this is not documented anywhere. My fault. 

Sent from my iPhone

[Roberto De Almeida]

And I meant "suppress", not "supreme". :)

Sent from my iPhone

On 11/11/2009, at 12:11, Roberto De Almeida <rob...@dealmeida.net> wrote:

[Roy Mendelssohn]

Hi Derek:

I do not know much python, and I do not know the internals of the various python netcdf readers - so this is just a guess. But given the size of your netdcf files (greater than 2Gb), they had to be created using the 64-bit offset option, and perhaps this is not supported in the python readers. The pointer error you are getting raises that suspicion.

-roy

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Men...@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

[Derek Broughton]

On November 11, 2009 10:11:35 Roberto De Almeida wrote:
> On Tue, Nov 10, 2009 at 9:02 PM, Derek Broughton
<de...@pointerstop.ca>wrote:
> > I commented out the error handler where it tries to open the file in
> > pydap/handlers/netcdf/__init__.py and got the following message. I beg
> > you
> > Roberto, DON'T trap errors like this. If all you're going to do is
> > "raise" a
> > different error message, it only obscures the problem. In 30 years of
> > program
> > development, I haven't found an issue to get me excited like this one :-)
>
> The server traps errors because you don't want it to stop running just
> because you installed a handler with problems.

That's fine when the server is actually going to abort (which is what's
happening with my attempt to use pynetcdf), but in the case where pupynere
couldn't open the file, the server continues, and I see, in the HTML response,
either an unhelpful "Unable to open file" message (as-is), or the actual error
(when I comment out the trap).

I agree, you want to trap errors if they're going to cause the server to
abort, but trapping an error merely to raise a different error - and losing all
the traceback in the process, isn't helpful.
--
derek

[Derek Broughton]

On November 10, 2009 14:09:38 Roberto De Almeida wrote:

> Hi, Derek!
>
> In theory you should be able to serve infinite files with pydap, since
> netcdf files are read in blocks of configurable size. Which netcdf module
> are you using -- pupynere, pynetcdf, Scientific.IO.NetCDF or PyNIO?
>

Rob;

I installed Scientific.IO.NetCDF, and now I don't see where that gets used by
the netcdf handler. The imports, in order, are:

from nio import open_file as nc
from netCDF4 import Dataset as nc
from pynetcdf import NetCDFFile as nc
from pupynere import NetCDFFile as nc

I presume nio is PyNIO, and pynetcdf and pupynere definitely import correctly,
even if neither is working for these datasets, but Scientific.IO.NetCDF doesn't
seem to contain a netCDF4 module. "python-netcdf" does, but I haven't got it
to compile (it needs HDF5 headers, and the ones I have don't seem to be the
ones its looking for).
--
derek

[Derek Broughton]

On November 11, 2009 12:48:53 Derek Broughton wrote:
> On November 10, 2009 14:09:38 Roberto De Almeida wrote:
> > Hi, Derek!
> >
> > In theory you should be able to serve infinite files with pydap, since
> > netcdf files are read in blocks of configurable size. Which netcdf module
> > are you using -- pupynere, pynetcdf, Scientific.IO.NetCDF or PyNIO?
>
> Rob;
>
> I installed Scientific.IO.NetCDF, and now I don't see where that gets used
> by the netcdf handler. The imports, in order, are:
>
> from nio import open_file as nc
> from netCDF4 import Dataset as nc
> from pynetcdf import NetCDFFile as nc
> from pupynere import NetCDFFile as nc
>
> I presume nio is PyNIO, and pynetcdf and pupynere definitely import
> correctly, even if neither is working for these datasets, but
> Scientific.IO.NetCDF doesn't seem to contain a netCDF4 module.
> "python-netcdf" does,

Drat. That was "netcdf4-python".

[Derek Broughton]

On November 11, 2009 12:50:16 Derek Broughton wrote:

> > I installed Scientific.IO.NetCDF, and now I don't see where that gets
> > used by the netcdf handler. The imports, in order, are:
> >
> > from nio import open_file as nc
> > from netCDF4 import Dataset as nc
> > from pynetcdf import NetCDFFile as nc
> > from pupynere import NetCDFFile as nc
> >
> > I presume nio is PyNIO, and pynetcdf and pupynere definitely import
> > correctly, even if neither is working for these datasets, but
> > Scientific.IO.NetCDF doesn't seem to contain a netCDF4 module.
> > "python-netcdf" does,
>
> Drat. That was "netcdf4-python".

I added an import for Scientific.IO.NetCDF, like the ones for pupynere and
pynetcdf, and that seems to be able to handle the files. Now I have to figure
out how to actually get a map out of it :-)
--
derek

[Roberto De Almeida]

You're right, in this case even with the "x-wsgiorg.throw_errors = 1" parameter the original traceback is lost, because the exception is trapped by the handler, not the server. I'll change the handler to remove this behavior, and also to include the Scientific.IO.NetCDF module as you suggested.

Thanks,
--Rob

[Roberto De Almeida]

On Wed, Nov 11, 2009 at 3:01 PM, Derek Broughton <de...@pointerstop.ca> wrote:

I added an import for Scientific.IO.NetCDF, like the ones for pupynere and
pynetcdf, and that seems to be able to handle the files.  Now I have to figure
out how to actually get a map out of it :-)

What do you mean with "get a map"? A map as in a picture from the WMS response?

Version 0.4.7 of the handler is available; it should work with Scientific.IO.NetCDF and it doesn't capture exceptions.

--Rob

[Derek Broughton]

On November 11, 2009 13:36:50 Roberto De Almeida wrote:
> On Wed, Nov 11, 2009 at 3:01 PM, Derek Broughton
<de...@pointerstop.ca>wrote:
> > I added an import for Scientific.IO.NetCDF, like the ones for pupynere
> > and pynetcdf, and that seems to be able to handle the files. Now I have
> > to figure
> > out how to actually get a map out of it :-)
>
> What do you mean with "get a map"? A map as in a picture from the WMS
> response?

Yes. I started another thread for that. Having done a little more digging
with the pydap client in an interactive python session, I rather suspect the
data files are incorrectly defined.
--
derek

[Derek Broughton]

On November 11, 2009 13:27:53 Roberto De Almeida wrote:
> You're right, in this case even with the "x-wsgiorg.throw_errors = 1"
> parameter the original traceback is lost, because the exception is trapped
> by the handler, not the server. I'll change the handler to remove this
> behavior, and also to include the Scientific.IO.NetCDF module as you
> suggested.

Great. Thanks.
--
derek

[Derek Broughton]

On November 11, 2009 10:09:56 Roberto De Almeida wrote:
> I recommend using virtualenv to create a separate environment for pydap:
>
> http://pydap.org/server.html#running-pydap-with-apache
>

FWIW, I ended up creating a "buildout", which if you don't know is a really
nice way to accomplish the same sort of thing, with dependency-checking, and
the ability to update all eggs with single command. If I knew it a little
better, I'd have it creating the wsgi script complete with all the egg paths,
too.
--
derek