OPeNDAP question for GHRSST data

Smith, Matthew R. (MSFC-ST11)[UAH]

unread,

Sep 2, 2021, 2:42:38 PM9/2/21

to sup...@opendap.org

Hello,

I’ve been retrieving SST NetCDF files for MODIS, VIIRS, and OSTIA for several months – until a couple of days ago, when my code failed. I was using regex to find the actual NetCDF file I wanted within the lengthy URL – where there are lots of ‘other’ related files – with other extensions. Upon fixing that code – now I retrieve the “.nc” file – only to have it actually be an HTML failure message (below).

The difference in the url that I provided and the “correct” (I think) URL is the ‘time’ of the file. In other words, can you help me determine the proper URL for ‘get’ting the actual NetCDF file?

Thanks,

Matt Smith

-----------------------

<head>

<title>Hyrax: Access Denied</title>

</head>

<body>

<h1>Hyrax : Access Denied (403) </h1>

<div class="large">The requested URL directly references a data source. </div>

<p>You must use the OPeNDAP request interface to get data from the data source.</p>

<p>If you would like to start at the top level of this server, go here:

<a href='https://podaac-opendap.jpl.nasa.gov/opendap/hyrax'>https://podaac-opendap.jpl.nasa.gov/opendap/hyrax</a></p>

<p>If you think that the server is broken (that the URL you

submitted should have worked), then please contact the

OPeNDAP user support coordinator at:

<a href="mailto:sup...@opendap.org">sup...@opendap.org</a></p>

<h1 >Hyrax : Access Denied (403) </h1>

</body>

</html>

--------------------

Matt Smith
NASA/SPoRT Project http://weather.msfc.nasa.gov/sport

The University of Alabama in Huntsville / Information Technology & Systems Center
at the National Space Science and Technology Center
ph 256-961-7809
e-mail matthew...@nasa.gov

James Gallagher

unread,

Sep 12, 2021, 7:22:33 PM9/12/21

to Smith, Matthew R. (MSFC-ST11)[UAH], James Gallagher, sup...@opendap.org

Matthew,

Sorry we did not respond sooner.

How were you logging in previously?

If you send me the URL and your code I can help.

James

James Gallagher
jgall...@opendap.org

Smith, Matthew R. (MSFC-ST11)[UAH]

unread,

Sep 13, 2021, 10:16:53 AM9/13/21

to James Gallagher, sup...@opendap.org

James,

Thanks for your reply. Until about 2 weeks ago I had been successfully retrieving (using the python 'requests' package):
OSTIA files like:
https://podaac-opendap.jpl.nasa.gov/opendap/hyrax/allData/ghrsst/data/L4/GLOB/UKMO/OSTIA/2021/255/20210912-UKMO-L4HRfnd-GLOB-v01-fv02-OSTIA.nc.bz2

VIIRS files like:
https://podaac-opendap.jpl.nasa.gov/opendap/hyrax/allData/ghrsst/data/GDS2/L2P/VIIRS_NPP/OSPO/v2.61/2021/256/20210913050000-OSPO-L2P_GHRSST-SSTsubskin-VIIRS_NPP-ACSPO_V2.61-v02.0-fv01.0.nc

and MODIS files like:
https://podaac-opendap.jpl.nasa.gov/opendap/hyrax/allData/ghrsst/data/GDS2/L2P/MODIS_A/JPL/v2019.0/2021/256/20210913000500-JPL-L2P_GHRSST-SSTskin-MODIS_A-D-v02.0-fv01.0.nc

Then my script began failing. I had to begin appending an extension ('nc4') to the NetCDF4 file 'requests'. That has worked well for MODIS and VIIRS, but I have not been able to find how to get the bz2-compressed OSTIA files (<15MB). If I try the 'nc4' extension, I get the uncompressed version (>250MB), or if I try to add the 'nc4' extension, I get the 'OUCH' failure HTML text.

Is the bz2-compressed version still available?

Thanks,
Matt

-----Original Message-----
From: James Gallagher <jgall...@opendap.org>
Sent: Sunday, September 12, 2021 6:23 PM
To: Smith, Matthew R. (MSFC-ST11)[UAH] <matthew...@nasa.gov>
Cc: James Gallagher <jgall...@opendap.org>; sup...@opendap.org
Subject: [EXTERNAL] Re: [support] OPeNDAP question for GHRSST data

Matthew,

Sorry we did not respond sooner.

How were you logging in previously?

If you send me the URL and your code I can help.

James

> On Sep 2, 2021, at 12:42 PM, 'Smith, Matthew R. (MSFC-ST11)[UAH]' via User Support <sup...@opendap.org> wrote:
>
> Hello,
>

> I've been retrieving SST NetCDF files for MODIS, VIIRS, and OSTIA for several months - until a couple of days ago, when my code failed. I was using regex to find the actual NetCDF file I wanted within the lengthy URL - where there are lots of 'other' related files - with other extensions. Upon fixing that code - now I retrieve the ".nc" file - only to have it actually be an HTML failure message (below).

>
> The difference in the url that I provided and the "correct" (I think) URL is the 'time' of the file. In other words, can you help me determine the proper URL for 'get'ting the actual NetCDF file?
>
> Thanks,
> Matt Smith
> -----------------------
> <html

> xmlns="https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%
> 2Fwww.w3.org%2F1999%2Fxhtml&data=04%7C01%7Cmatthew.r.smith%40nasa.
> gov%7C4daa2075e65144715c5008d976443dac%7C7005d45845be48ae8140d43da96dd
> 17b%7C0%7C0%7C637670857739421660%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wL
> jAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata
> =5QnJQlQeU37kCJao3smxhnSwwU2loX2WNGm8KkJXcU0%3D&reserved=0">

Nathan Potter

unread,

Sep 13, 2021, 11:55:57 AM9/13/21

to Smith, Matthew R. (MSFC-ST11)[UAH], Wang, Jerry (US 173B-Affiliate), Nathan Potter, James Gallagher, OPeNDAP Support

Hi Matt & Jerry,

The server you are interacting with is really meant to be used to subset and process data on the server side before transmission. If all you want/need to do is download the source granule files I think that NASA/JPL have a different service endpoint where you should direct this type of request. I have included Jerry Wang from JPL on this reply and (assuming my previous statement is correct) he will be able to guide you to the correct location.

Thanks,

Nathan

= = =
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317

Smith, Matthew R. (MSFC-ST11)[UAH]

unread,

Sep 13, 2021, 2:06:12 PM9/13/21

to Potter, Nathan (GSFC-4230)[OPENDAP, INC.], Wang, Jerry (JPL-173B)[RAYTHEON], Potter, Nathan (GSFC-4230)[OPENDAP, INC.], James Gallagher, OPeNDAP Support

I should add that I *am* using the OPeNDAP metadata to determine which files I want to grab. (We only want granules containing Northern hemisphere data.)

>> %25
>> 2Fwww.w3.org%2F1999%2Fxhtml&data=04%7C01%7Cmatthew.r.smith%40nasa.
>> gov%7C4daa2075e65144715c5008d976443dac%7C7005d45845be48ae8140d43da96d
>> d
>> 17b%7C0%7C0%7C637670857739421660%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4w
>> L
>> jAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdat

Reply all

Reply to author

Forward