Re: using pydap to retrieve grib2 data from NCEP website

[Jingbing Xiao]

Hi,

 

I am trying to use pydap to retrieve data from NCEP website:

 

Here is the script:

________________________

from pydap.client import open_url

dataset = open_url('http://www.ftp.ncep.noaa.gov/data/nccf/com/nam/prod/nam.20111207/nam.t00z.awip1201.tm00.10m.uv.grib2')

print type(dataset)

Then, I got following message:

>>

 

Traceback (most recent call last):

  File "C:/Python27/test_pydap-1.py", line 2, in <module>

    dataset = open_url('http://www.ftp.ncep.noaa.gov/data/nccf/com/nam/prod/nam.20111207/nam.t00z.awip1201.tm00.10m.uv.grib2')

  File "C:\Python27\lib\site-packages\pydap-unknown-py2.7.egg\pydap\client.py", line 60, in open_url

    dataset = response(url)

  File "C:\Python27\lib\site-packages\pydap-unknown-py2.7.egg\pydap\client.py", line 210, in _ddsdas

    dataset = DDSParser(dds).parse()

  File "C:\Python27\lib\site-packages\pydap-unknown-py2.7.egg\pydap\parsers\dds.py", line 26, in _dataset

    self.consume('dataset')

  File "C:\Python27\lib\site-packages\pydap-unknown-py2.7.egg\pydap\parsers\dds.py", line 19, in consume

    token = SimpleParser.consume(self, regexp)

  File "C:\Python27\lib\site-packages\pydap-unknown-py2.7.egg\pydap\parsers\__init__.py", line 25, in consume

    raise Exception("Unable to parse token: %s" % self.buffer[:10])

Exception: Unable to parse token: <!DOCTYPE

_________________________________________________________

However, when I tried to get data from NCDC website, the following script works:

_________________________________

from pydap.client import open_url

dataset = open_url('http://nomads.ncdc.noaa.gov/thredds/dodsC/nam218/201111/20111102/nam_218_20111102_0000_000.grb')

print type(dataset)

 

Can pydap retrieve grib2 data? The only difference I observed is that the NCEP data is in grib2 while NCDC data is in grib. 

 

Thank you for your helps in advance.

 

 

Jingbing Xiao

Meteorologist

Vice President, Technology
Cielo Wind Services, Inc.
823 Congress Suite 500
Austin, Texas 78701 

512.440.0305 phone
512.440.0277 Fax

jx...@cielowind.com

Please view our website at www.cielowind.com 
-----------------------------------------------------------------------------------
The content of this transmission is intended only for the entity, organization or person(s) addressed herein. The subject of this transmission may contain privileged or confidential information and is not to be disseminated or transferred in any manner. In the event an error was made and you are not the intended recipient of this information, please notify the sender and destroy all documentation of this message.

[Roberto De Almeida]

Hi, Jingbing!

The problem with your script is that the first URL is not an OPeNDAP server, while the second is. Pydap can only access datasets on OPeNDAP servers (like Hyrax, THREDDS or pydap itself), and in that case it doesn't matter the format of the data. 

For your first URL you'll need to download the data using something else. If you want to do it in Python I would recommend the requests module (http://pypi.python.org/pypi/requests), or httplib2.

Cheers,

--Rob

--
You received this message because you are subscribed to the Google Groups "pydap" group.
To post to this group, send email to py...@googlegroups.com.
To unsubscribe from this group, send email to pydap+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pydap?hl=en.

--
Dr. Roberto De Almeida
http://dealmeida.net/
http://lattes.cnpq.br/1858859813771449
:wq

[Jingbing Xiao]

 

Hi, Rob,

 

On the NCEP website, it does include an OpenDap option. If I need to get data from following folder:

 

http://nomads.ncep.noaa.gov:9090/dods/nam/nam20111206

 

Would you advise how can I use PyDAP?  I am pretty new to the script and data set.  Your instruction is very much appreciated.

 

Jingbing

 

 

 

Jingbing Xiao

Meteorologist

Vice President, Technology
Cielo Wind Services, Inc.
823 Congress Suite 500
Austin, Texas 78701 

512.440.0305 phone
512.440.0277 Fax

jx...@cielowind.com

Please view our website at www.cielowind.com 
-----------------------------------------------------------------------------------
The content of this transmission is intended only for the entity, organization or person(s) addressed herein. The subject of this transmission may contain privileged or confidential information and is not to be disseminated or transferred in any manner. In the event an error was made and you are not the intended recipient of this information, please notify the sender and destroy all documentation of this message.

From: Jingbing [mailto:jingbi...@yahoo.com]
Sent: Thursday, December 08, 2011 1:22 PM
To: Jingbing Xiao
Subject: Fwd: using pydap to retrieve grib2 data from NCEP website

 

Sent from my iPhone 

Begin forwarded message:

From: xinmin <xin...@gmail.com>
Date: December 7, 2011 20:46:11 CST
To: Jingbing Xiao <jingbi...@yahoo.com>
Subject: Fwd: using pydap to retrieve grib2 data from NCEP website

Sent from my iPhone

Begin forwarded message:

[Roberto De Almeida]

Hi, Jingbing!

Sure, it can be tricky if you don't know anything about Opendap. The website at NOAA assumes you're familiar with the procotol.

Every Opendap dataset has a canonical URL, say:

  http://test.pydap.org/coads.nc

(If you click on that URL you'll download the netcdf file.)

Usually an Opendap server will take you an HTML page that shows information about the data, and gives a form that you can use to download data. That HTML page usually is located at:

  http://test.pydap.org/coads.nc.html

You can also inspect the dataset by changing that extension. Here are a few that you can try:

  http://test.pydap.org/coads.nc.dds

  http://test.pydap.org/coads.nc.das

  http://test.pydap.org/coads.nc.help

If you look at the page at NOAA, each dataset has 3 links: one points to the .info extension, another to the .dds, and a third to .das. The first link, for example, sends you to:

  http://nomads.ncep.noaa.gov:9090/dods/nam/nam20111206/nam1hr_00z.info

This is a page showing some information about the dataset. 

If you want to access this dataset using Pydap (or any other Opendap client), simply open it ommiting the special extension. With Pydap you'd do this:

  from pydap.client import open_url

  dataset = open_url("http://nomads.ncep.noaa.gov:9090/dods/nam/nam20111206/nam1hr_00z")

Note that I removed the .info extension. It should work fine.

Best,

--Rob

[Jingbing Xiao]

Hi, Rob,

It works!  Thank you so much.

Jingbing

From: Roberto De Almeida <rob...@dealmeida.net>
Date: December 8, 2011 3:19:18 PM CST
To: py...@googlegroups.com
Subject: Re: using pydap to retrieve grib2 data from NCEP website
Reply-To: py...@googlegroups.com

Hi, Jingbing!

 

Sure, it can be tricky if you don't know anything about Opendap. The website at NOAA assumes you're familiar with the procotol.

[Cass Kalinski]

Thirteen plus years later, a thanks to Roberto. Your advice is still reaping benefits. You just helped solve a roadblock on accessing NOAA HRRR data. First time using pydap and the Opendap datasets and it was a bit of a blackbox at first.