Error when extracting large datasets

[Amy]

Hi,

I thought I'd report this as it took me a long time to figure out why
Pydap kept crashing on me. The problem was if you try to extract too
much data at one time, you may exceed the maximum allowed size -- I
think maybe this is server dependent. Anyway, it seems obvious now,
but I would get the following when using Pydap:

In [186]: test = open_dods('http://edac-dap.northerngulfinstitute.org/
thredds/dodsC/ncom/SFC8/ncom_glb_sfc8_2009120500.nc.dods?
water_u.water_u[0:1:12][0:1:1348][0:1:2879]')
---------------------------------------------------------------------------
ValueError Traceback (most recent call
last)

c:\amy\bunk.py in <module>()
----> 1
2
3
4
5

C:\EPD\lib\site-packages\pydap-3.0.rc.8-py2.5.egg\pydap\client.pyc in
open_dods(url, get_metadata)
174 dds, xdrdata = data.split('\nData:\n', 1)
175 dataset = DDSParser(dds).parse()
--> 176 dataset.data = DapUnpacker(xdrdata, dataset).getvalue()
177
178 if get_metadata:

C:\EPD\lib\site-packages\pydap-3.0.rc.8-py2.5.egg\pydap\xdr.pyc in
getvalue(self)
114 var = self.var
115 self.var = child
--> 116 out.append(self.getvalue())
117 self.var = var
118 out = tuple(out)

C:\EPD\lib\site-packages\pydap-3.0.rc.8-py2.5.egg\pydap\xdr.pyc in
getvalue(self)
141
142 if getattr(self.var, 'shape', False):
--> 143 out.shape = tuple(idx for idx in
self.var.shape if idx != 1)
144 else:
145 out = out[0]

ValueError: total size of new array must be unchanged

...so it wasn't obvious to me what was going on. Anyway, its because
in this case what it returned in dataset in this case is an error
message that looks like this:

Dataset {
Int16 water_u[time = 13][lat = 1349][lon = 2880];
} ncom/SFC8/ncom_glb_sfc8_2009120500.nc;

Data:
Error {
code = 500;
message = "Java heap space";
};


Anyway, I know what's going on now, so its not a problem for me any
longer, but thought I should report so that pydap might handle this
case more gracefully in some future version.

Thanks!!!!!

-Amy