"Request too big" error

[Blazetopher]

Hi all,

I'm running into a bit of an issue using pydap as a client for
accessing OPeNDAP data. While I'm able to successfully perform all of
the actions in the 3.0.1 client tutorial (Gridded Data), I keep
hitting the same problem when I try the same thing with other
datasets.

Two datasets in particular that are having trouble are:
http://thredds1.pfeg.noaa.gov/thredds/dodsC/satellite/GR/ssta/1day
and
http://hfrnet.ucsd.edu:8080/thredds/dodsC/HFRNet/USEGC/6km/hourly/RTV

I can open both datasets, interrogate them, and retrieve GridType
objects without any trouble. But when I attempt to get a VERY SMALL
subset of data, I consistently get a "Request too big" error. I'm
familiar with the constraints put on OPeNDAP servers, but the quantity
of data I'm requesting should be FAR below the thresholds. Identical
requests (for a [0,10:14,10:14] subset) work without issue from
ToolsUI as well as the opendap data access form for the datasets.

The steps I'm performing and subsequent error are below. I get the
same result for the second url above. Any insight to what I might be
doing incorrectly would be greatly appreciated.

Thanks,
Chris

*******

DEV014MAC001::~ cmueller: ipython
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
Type "copyright", "credits" or "license" for more information.

IPython 0.11 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.

In [1]: from pydap.client import open_url

In [2]: ds=open_url('http://thredds1.pfeg.noaa.gov/thredds/dodsC/
satellite/GR/ssta/1day')

In [3]: ds.keys()
Out[3]: ['lat', 'lon', 'time', 'analysed_sst', 'analysis_error',
'mask']

In [4]: sst=ds.analysed_sst

In [5]: sst.shape
Out[5]: (2074, 2048, 4096)

In [6]: grid=sst[0,10:14,10:14]
---------------------------------------------------------------------------
ServerError Traceback (most recent call
last)
/Users/cmueller/<ipython-input-6-7ad0df224c2d> in <module>()
----> 1 grid=sst[0,10:14,10:14]

/Library/Python/2.6/site-packages/Pydap-3.0.rc.10-py2.6.egg/pydap/
model.pyc in __getitem__(self, key)
626 out = copy.deepcopy(self)
627 for var, slice_ in zip(out.walk(), [key] +
list(key)):
--> 628 var.data = var.data[slice_]
629 var.shape = var.data.shape
630 return out

/Library/Python/2.6/site-packages/Pydap-3.0.rc.10-py2.6.egg/pydap/
proxy.pyc in __getitem__(self, index)
110 fragment))
111
--> 112 resp, data = request(url)
113 dds, xdrdata = data.split('\nData:\n', 1)
114 dataset = DDSParser(dds).parse()

/Library/Python/2.6/site-packages/Pydap-3.0.rc.10-py2.6.egg/pydap/util/
http.pyc in request(url)
47 data, re.DOTALL | re.MULTILINE)
48 msg = 'Server error %(code)s: "%(msg)s"' %
m.groupdict()
---> 49 raise ServerError(msg)
50
51 return resp, data

ServerError: 'Server error 403: "Request too big=34795.0 Mbytes,
max=750.0"'

*******

[Roberto De Almeida]

Hi, Chris.

There seems to be a problem with that server... this should be valid request:

$ curl -g "http://thredds1.pfeg.noaa.gov/thredds/dodsC/satellite/GR/ssta/1day.asc?analysed_sst.time[0:1:0]"

Error {

    code = 403;

    message = "Request too big=34812.0 Mbytes, max=50.0";

};

It looks like the server is parsing the request for analysed_sst.time[0:1:0] as a request for the complete array, instead of only the first value of the time axis.

My suggestion for you is to request only the data for the array, not for the axes:

grid=sst.array[0,10:14,10:14]

I tested and this works fine. You can then retrieve the axes individually:

time = ds.time[0]

(And then the same for the other axes.)

--Rob

--
You received this message because you are subscribed to the Google Groups "pydap" group.
To post to this group, send email to py...@googlegroups.com.
To unsubscribe from this group, send email to pydap+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pydap?hl=en.

--
Dr. Roberto De Almeida
http://dealmeida.net/
http://lattes.cnpq.br/1858859813771449
:wq

[Roy Mendelssohn]

Hi Roberto:

I just tested that dataset as it is one of ours. It works from the html form. It works from ERDDAP, and it works from toolsUI. So there has to be something else going on.

-Roy

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Men...@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

[Roberto De Almeida]

Hi, Roy!

Sorry, but the fact that it works with some clients doesn't mean it's working as intended. In that dataset there are two different ways to retrieve the time axis:

(1) http://thredds1.pfeg.noaa.gov/thredds/dodsC/satellite/GR/ssta/1day.dds?analysed_sst.time

(2) http://thredds1.pfeg.noaa.gov/thredds/dodsC/satellite/GR/ssta/1day.dds?time

If you check the DDS responses above, you'll see that the two are identical. This is because "time" is a map in the "analysed_sst" grid, but also a variable in the root dataset, as we can see from the unconstrained DDS:

http://thredds1.pfeg.noaa.gov/thredds/dodsC/satellite/GR/ssta/1day.dds

Now, if you replace the ".dds" extension in the two requests above with ".dods", you'll notice that the first one returns a "Request Too Big" error, while the second one is fine. My guess is that there's a bug in the code that estimates the size of the requests.

Why does this work with other clients? As far as I know, pydap is the only opendap client that does requests for the grid map (request #1 above), instead of requests for the root variable (#2). Other clients that I've seen prefer the second approach. I use the first one because... well, guess because I'm a purist who sometimes favors purism over practicality*. :-P

I'm CC'ing James on this email, maybe he knows who we should contact in order to fix this bug.

Best,

--Rob

* Actually that's not true... in pydap, a variable has no knowledge of other variables in the same dataset, and given a grid variable with a "time" axis there's no guarantee in the DAP spec that there'll be a variable "time" on the root dataset, defining the same axis.

[Roy Mendelssohn]

You should send this to the Unidata guys. It is whatever the response that is produced by the thredds data server. Send to the thredds mail list -THREDDS community <thr...@unidata.ucar.edu>

-Roy

[Blazetopher]

Hi Rob,

Many thanks for the reply, the workaround you provided works as
indicated. I agree with Roy that this should be brought to Unidata as
I've had the same problem with datasets from other thredds servers.

Thanks again!
Chris

On Nov 27, 7:53 pm, Roberto De Almeida <robe...@dealmeida.net> wrote:
> Hi, Chris.
>
> There seems to be a problem with that server... this should be valid
> request:
>

> $ curl -g "http://thredds1.pfeg.noaa.gov/thredds/dodsC/satellite/GR/ssta/1day.as...[0:1:0]

> "
> Error {
>     code = 403;
>     message = "Request too big=34812.0 Mbytes, max=50.0";
>
> };
>
> It looks like the server is parsing the request for
> analysed_sst.time[0:1:0]as a request for the complete array, instead
> of only the first value of the
> time axis.
>
> My suggestion for you is to request only the data for the array, not for
> the axes:
>

> grid=sst*.array*[0,10:14,10:14]

> Dr. Roberto De Almeidahttp://dealmeida.net/<http://roberto.dealmeida.net/>http://lattes.cnpq.br/1858859813771449
> :wq

[Roy Mendelssohn]

This has been reported to Unidata, an error has been found, and it has been added to their tracking system.

-Roy

**********************

[Roberto De Almeida]

This is great! Thanks, Roy!

--Rob