access to MERRA-2 using opendap

[Bernard Legras]

Hi

I am trying to extract MERRA-2 data from gsfc servers using a procedure
based on ncks requests and opendap.
A typical request is like that
ncks -4 -L 9 -v lat,lon,lev,time,DTDTLWRCLR
http://legras:<my_passwd>@goldsmr5.sci.gsfc.nasa.gov/opendap/hyrax/MERRA2/M2T3NVRAD.5.12.4/2010/01/MERRA2_300.tavg3_3d_rad_Nv.20100110.nc4
/dkol/data/MERRA-2/M2T3NVRAD/2010/01/DTDTLWRCLR_Nv.20100110.nc4

It was working perfectly one year ago and does not work anymore. I get a
access denied answer. I can do the extraction through the web interface
but I cannot do that for many files. What should I do to make this
working again?

Best wishes

Bernard Legras

[Nathan Potter]

Hi Bernard,

NASA has instituted authenticated access for data. They use the “Earth Data Login” OAuth2 implementation for this and I think that you can get ncks working in the manner described for the netcdf-c libaray tools here:

http://docs.opendap.org/index.php/DAP_Clients_-_Authentication#Special_Note_To_NASA_Earthdata_Login_.28aka_URS.29_Users

And here:

http://docs.opendap.org/index.php/DAP_Clients_-_Authentication#Earth_Data_Login_.28URS.29

Please let me know if that helps!

Thanks,

Nathan

= = =
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317

[Bernard Legras]

Hi Nathan

Thanks for your answer

I already have a .netrc file and I put all what I found related to earthdata and opendap in my approved list of applications (see below). When I click on the button "approve more application", I do not get the page shown in the link you gave me (that I already found myself) but another page with a long list of applications and nothing like tesy_tesy.

It appears that I can access the opendap server with pydap. This requires to write a module to do what I was doing with ncks but this is quite easy and I am testing that at the moment.

I am, however, still interested to understand how to make ncks working.

Best wishes

Bernard

[Nathan Potter]

Hi Bernard,

A few tings I noticed:

1) Yes, the snapshot for the list of applications is out of date. There’s a ticket for that in our system.

2) The important thing is that one of the “Approved Applications” is associated with goldsmr5.sci.gsfc.nasa.gov/opendap/ ( I think the question mark link on the right will tell you). If there is no “Approved Application” associated with goldsmr5.sci.gsfc.nasa.gov/opendap/ then you’ll have to “Add More Applications” and search for it somehow. My guess is that the Application “NASA GESDIC DATA ARCHIVE” is the one. If that’s the case there must be a problem somewhere else.

One thing people often overlook is that your EDL credentials are not associated with goldsmr5.sci.gsfc.nasa.gov, but rather with either:

uat.urs.earthdata.nasa.gov
or
urs.earthdata.nasa.gov

That has to be made clear in .netrc and it also means that this authentication pattern:

>>> http://legras:<my_passwd>@goldsmr5.sci.gsfc.nasa.gov

Is futile because EDL is a single sign on system - goldsmr5.sci.gsfc.nasa.gov never touches the credentials - it just redirects the client to (uat.)urs.earthdata.nasa.gov where the credentials should be submitted.

Does any of that help with this?

Sincerely,

Nathan

> On Jul 1, 2017, at 1:52 PM, Bernard Legras <bernard...@free.fr> wrote:
>
> Hi Nathan
>
> Thanks for your answer
>
> I already have a .netrc file and I put all what I found related to earthdata and opendap in my approved list of applications (see below). When I click on the button "approve more application", I do not get the page shown in the link you gave me (that I already found myself) but another page with a long list of applications and nothing like tesy_tesy.
>
> It appears that I can access the opendap server with pydap. This requires to write a module to do what I was doing with ncks but this is quite easy and I am testing that at the moment.
>
> I am, however, still interested to understand how to make ncks working.
>
> Best wishes
>
> Bernard
>

> <hpnpkkocllgdjcdi.png>

[Nathan Potter]

Hi Bernard,

Well, that was one thing to try. In general urs.earthdata.nasa.gov is the correct point for EDL authentication the uat.urs.earthdata.nasa.gov is only used for testing.

At this point I think that you might fall back to using the “curl” command line tool (setup is also detailed on that Clients page here: http://docs.opendap.org/index.php/DAP_Clients_-_Authentication#URS ) to debug your problem.

If you can’t get curl to work then something is wrong with either your EDL Approved Application list or with you .netrc file. Putting curl in verbose mode (-v option) should shed some light on what’s wrong.

If you can get curl working and ncdump (or ncks) is still giving you fits then it may be your ~/.dodsrc file that is missing critical content.

If you get to the point where ncdump works and ncks does not then I think there must be a additional requirement that we need to understand and w made need to reach out to their developers.


Sincerely,

Nathan

> On Jul 1, 2017, at 3:12 PM, Bernard Legras <bernard...@free.fr> wrote:
>
> Hi Nathan
>

> My .netrc file was
>
> machine urs.earthdata.nasa.gov login legras password <passwd>
>
> I changed the machine to uat.urs... without result
>
> Best wishes
>
> Bernard

[Bernard Legras]

Hi Nathan

I have not tried curl but I have the same problem with wget. I will give
a try with curl.

I am presently processing data with pydap which gets authentication as
described in the manual and I managed to get the same output as with
ncks. The mechanism bypasses the .netrc file.

I am not familiar with curl but I am unsure it can extract some data on
the server using the opendap service. If I need to retrieve the whole
file, I do not really need to use the opendap service. I can directly
get the data on goldsmr4.gesdisc.eosdis.nasa.gov/data/MERRA2 by wget or
ncks and this works perfectly well.

A last point I am worrying about is: does the hyrax server performs some
compression of the data which is sent? This is not activated by default
as far as I know. If it does not, it might indeed be lighter to download
the whole compressed file in a temp directory than non expanded pieces.

Best wishes

Bernard