Re: [#63481]: Cannot seem to access GES DISC's OPeNDAP (though I have in the past)

[Lynnes, Christopher S. (GSFC-5860)]

I have made some progress here, but not necessarily the good kind.

I have set up my .dodsrc file as recommended, and have set a .netrc to supply username/password to urs.earthdata.nasa.gov. I have also upgraded to netCDF 4.4.1. I can get curl to download a data variable:

curl -k -n -c ursCookies -b ursCookies -L --url http://airsl2.gesdisc.eosdis.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.006/2003/003/AIRS.2003.01.03.003.L2.RetStd.v6.0.7.0.G13186000743.hdf.ascii?satheight
Dataset: AIRS.2003.01.03.003.L2.RetStd.v6.0.7.0.G13186000743.hdf
satheight, 705.263, 705.348, 705.435, 705.525, 705.618, 705.713, 705.811, 705.912, 706.015, 706.12, 706.228, 706.338, 706.451, 706.566, 706.683, 706.803, 706.925, 707.05, 707.177, 707.306, 707.438, 707.571, 707.707, 707.846, 707.986, 708.129, 708.273, 708.42, 708.569, 708.72, 708.873, 709.027, 709.184, 709.343, 709.504, 709.666, 709.83, 709.997, 710.164, 710.334, 710.505, 710.678, 710.853, 711.029, 711.207

And I can get ncdump to dump the header:
ncdump -h http://airsl2.gesdisc.eosdis.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.006/2003/003/AIRS.2003.01.03.003.L2.RetStd.v6.0.7.0.G13186000743.hdf

But I cannot get ncdump to fetch a data variable:

~/bin/ncdump -v satheight http://airsl2.gesdisc.eosdis.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.006/2003/003/AIRS.2003.01.03.003.L2.RetStd.v6.0.7.0.G13186000743.hdf
<lots of header stuff>
satheight = CURL Error: Couldn't connect to server
curl error details:
NetCDF: I/O failure
Location: file vardata.c; line 473

And no way for me to tell if the problem is in the netcdf build I have, the elderly curl library, the URS configuration, the .dodsrc, or what. Can anyone reproduce this with ncdump, or give me some things to try for diagnostics?

> On Sep 19, 2016, at 3:58 PM, Peter Smith <sup...@earthdata.nasa.gov> wrote:
>
> Chris
>
> I've not used ncks, but it is similar to ncdump in that it is built on top of netCDF. We should therefore be able to configure it in the same way. The OPeNDAP group has provided some documentation for this:
>
> http://docs.opendap.org/index.php/DAP_Clients_-_Authentication#ncdump
>
> Let me know if you have any problems making this work.
>
> Embedding your username and password into the URL will not work because even if the client supports the syntax, it will send the credentials only to the data server from which you are downloading data - it will not send them to Earthdata Login when the client is redirected for user authentication.
>
> Thanks,
> Peter Smith
>
>
>
> Ticket History
> Chris Lynnes (Client) Posted On: 19 September 2016 12:17 PM
>
> ~>wget --no-check-certificate http://airsl2.gesdisc.eosdis.nasa.gov:80/opendap/Aqua_AIRS_Level2/AIRX2RET.006/2003/003/AIRS.2003.01.03.003.L2.RetStd.v6.0.7.0.G13186000743.hdf.nc
> --2016-09-19 12:02:53-- http://airsl2.gesdisc.eosdis.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.006/2003/003/AIRS.2003.01.03.003.L2.RetStd.v6.0.7.0.G13186000743.hdf.nc
> Resolving airsl2.gesdisc.eosdis.nasa.gov... 198.118.197.66, 2001:4d0:241a:4041::66
> Connecting to airsl2.gesdisc.eosdis.nasa.gov|198.118.197.66|:80... connected.
> HTTP request sent, awaiting response... 302 Found
> Location: https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fairsl2.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cDovL2FpcnNsMi5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9vcGVuZGFwL0FxdWFfQUlSU19MZXZlbDIvQUlSWDJSRVQuMDA2LzIwMDMvMDAzL0FJUlMuMjAwMy4wMS4wMy4wMDMuTDIuUmV0U3RkLnY2LjAuNy4wLkcxMzE4NjAwMDc0My5oZGYubmM [following]
> --2016-09-19 12:02:53-- https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fairsl2.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cDovL2FpcnNsMi5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9vcGVuZGFwL0FxdWFfQUlSU19MZXZlbDIvQUlSWDJSRVQuMDA2LzIwMDMvMDAzL0FJUlMuMjAwMy4wMS4wMy4wMDMuTDIuUmV0U3RkLnY2LjAuNy4wLkcxMzE4NjAwMDc0My5oZGYubmM
> Resolving urs.earthdata.nasa.gov... 198.118.243.33, 2001:4d0:241a:4081::89
> Connecting to urs.earthdata.nasa.gov|198.118.243.33|:443... connected.
> WARNING: cannot verify urs.earthdata.nasa.gov's certificate, issued by `/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2014 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1M':
> Unable to locally verify the issuer's authority.
> HTTP request sent, awaiting response... 401 Unauthorized
> Authorization failed.
>
> Peter Smith (Staff) Posted On: 19 September 2016 12:32 PM
>
> Hi Chris
>
> When wget exits upon receiving a HTTP 401 status, it usually indicates that it cannot find suitable credentials to authenticate with. Can you please verify that your .netrc file located in your home directory contains a line of the form:
>
> machine urs.earthdata.nasa.gov login <username> password <password>
>
> where <uid> and <password> are your Earthdata Login username and password.
>
> If your .netrc file is correct, can you please run a test by adding in the following options to your wget command line:
>
> --http-user=<username> --ask-password
>
> where <username> is your Earthdata Login username.
>
> Thank you,
> Peter Smith
> Earthdata Login Suport
>
>
>
>
> Lynnes Christopher S. (GSFC-6102) (Recipient (CC)) Posted On: 19 September 2016 02:37 PM
>
>
> On Sep 19, 2016, at 12:32 PM, Peter Smith <sup...@earthdata.nasa.gov<mailto:sup...@earthdata.nasa.gov>> wrote:
>
> Hi Chris
>
> When wget exits upon receiving a HTTP 401 status, it usually indicates that it cannot find suitable credentials to authenticate with. Can you please verify that your .netrc file located in your home directory contains a line of the form:
>
> machine urs.earthdata.nasa.gov<http://urs.earthdata.nasa.gov> login <username> password <password>
>
> where <uid> and <password> are your Earthdata Login username and password.
>
> If your .netrc file is correct, can you please run a test by adding in the following options to your wget command line:
>
> --http-user=<username> --ask-password
>
> where <username> is your Earthdata Login username.
>
> Thank you,
> Peter Smith
> Earthdata Login Suport
>
> Ah, I see my mistake; I had a GES DISC address in .netrc instead of URS. Must have been left over from URS3.
>
> Unfortunately, other command line tools, like ncks, do not appear to know how to login with .netrc...
>
>
>
>
>
> Ticket History
> Chris Lynnes (Client) Posted On: 19 September 2016 12:17 PM
>
>
> Lynnes Christopher S. (GSFC-6102) (Recipient (CC)) Posted On: 19 September 2016 02:54 PM
>
>
> On Sep 19, 2016, at 2:37 PM, Lynnes, Christopher S. (GSFC-5860) <christophe...@nasa.gov<mailto:christophe...@nasa.gov>> wrote:
>
>
> On Sep 19, 2016, at 12:32 PM, Peter Smith <sup...@earthdata.nasa.gov<mailto:sup...@earthdata.nasa.gov>> wrote:
>
> Hi Chris
>
> When wget exits upon receiving a HTTP 401 status, it usually indicates that it cannot find suitable credentials to authenticate with. Can you please verify that your .netrc file located in your home directory contains a line of the form:
>
> machine urs.earthdata.nasa.gov<http://urs.earthdata.nasa.gov/> login <username> password <password>
>
> where <uid> and <password> are your Earthdata Login username and password.
>
> If your .netrc file is correct, can you please run a test by adding in the following options to your wget command line:
>
> --http-user=<username> --ask-password
>
> where <username> is your Earthdata Login username.
>
> Thank you,
> Peter Smith
> Earthdata Login Suport
>
> Ah, I see my mistake; I had a GES DISC address in .netrc instead of URS. Must have been left over from URS3.
>
> Unfortunately, other command line tools, like ncks, do not appear to know how to login with .netrc...
>
> Should embedding username and password in the URL work? http://username:password@host/etc. I can't seem to make it work...
>
>
>
>
>
>
> Ticket History
> Chris Lynnes (Client) Posted On: 19 September 2016 12:17 PM
>
>
> Ticket Details
> Ticket ID: 63481
> Department: URS
> Type: Data/Science
> Status: Closed
> Priority: Normal
>
> Helpdesk: https://support.earthdata.nasa.gov/index.php?

--
Christopher Lynnes NASA/GSFC 301-614-5185
“Don't loaf and invite inspiration; light out after it with a club..." - J. London

[Peter Smith]

Hi Chris

I managed to reproduce this problem. I believe this is related to an issue that Mike Theobald flagged on Friday, where the Earthdata Login server is mistakenly changing the protocol from http to https on one of the redirect responses. I'm working with development to try and isolate the problem.

If you add:

HTTP.VERBOSE=1

to your .dodsrc file, ncdump (and possibly any tool built on netCDF) will enable verbose logging from libcurl, and that provides a lot of useful information about the http request/responses.

I will let you know as soon as we have diagnosed the problem.

Thanks,
Peter Smith

Ticket History

Chris Lynnes (Client) Posted On: 19 September 2016 12:17 PM

~>wget --no-check-certificate http://airsl2.gesdisc.eosdis.nasa.gov:80/opendap/Aqua_AIRS_Level2/AIRX2RET.006/2003/003/AIRS.2003.01.03.003.L2.RetStd.v6.0.7.0.G13186000743.hdf.nc
--2016-09-19 12:02:53-- http://airsl2.gesdisc.eosdis.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.006/2003/003/AIRS.2003.01.03.003.L2.RetStd.v6.0.7.0.G13186000743.hdf.nc
Resolving airsl2.gesdisc.eosdis.nasa.gov... 198.118.197.66, 2001:4d0:241a:4041::66
Connecting to airsl2.gesdisc.eosdis.nasa.gov|198.118.197.66|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fairsl2.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cDovL2FpcnNsMi5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9vcGVuZGFwL0FxdWFfQUlSU19MZXZlbDIvQUlSWDJSRVQuMDA2LzIwMDMvMDAzL0FJUlMuMjAwMy4wMS4wMy4wMDMuTDIuUmV0U3RkLnY2LjAuNy4wLkcxMzE4NjAwMDc0My5oZGYubmM [following]
--2016-09-19 12:02:53-- https://urs.earthdata.nasa.gov/oauth/authorize/?scope=uid&app_type=401&client_id=e2WVk8Pw6weeLUKZYOxvTQ&response_type=code&redirect_uri=http%3A%2F%2Fairsl2.gesdisc.eosdis.nasa.gov%2Fdata-redirect&state=aHR0cDovL2FpcnNsMi5nZXNkaXNjLmVvc2Rpcy5uYXNhLmdvdi9vcGVuZGFwL0FxdWFfQUlSU19MZXZlbDIvQUlSWDJSRVQuMDA2LzIwMDMvMDAzL0FJUlMuMjAwMy4wMS4wMy4wMDMuTDIuUmV0U3RkLnY2LjAuNy4wLkcxMzE4NjAwMDc0My5oZGYubmM
Resolving urs.earthdata.nasa.gov... 198.118.243.33, 2001:4d0:241a:4081::89
Connecting to urs.earthdata.nasa.gov|198.118.243.33|:443... connected.
WARNING: cannot verify urs.earthdata.nasa.gov's certificate, issued by `/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2014 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1M':
Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 401 Unauthorized
Authorization failed.

Peter Smith (Staff) Posted On: 19 September 2016 12:32 PM

Hi Chris

When wget exits upon receiving a HTTP 401 status, it usually indicates that it cannot find suitable credentials to authenticate with. Can you please verify that your .netrc file located in your home directory contains a line of the form:

machine urs.earthdata.nasa.gov login <username> password <password>

where <uid> and <password> are your Earthdata Login username and password.

If your .netrc file is correct, can you please run a test by adding in the following options to your wget command line:

--http-user=<username> --ask-password

where <username> is your Earthdata Login username.

Thank you,
Peter Smith
Earthdata Login Suport




Lynnes Christopher S. (GSFC-6102) (Recipient (CC)) Posted On: 19 September 2016 02:37 PM

On Sep 19, 2016, at 12:32 PM, Peter Smith <sup...@earthdata.nasa.gov<mailto:sup...@earthdata.nasa.gov>> wrote:

Hi Chris

When wget exits upon receiving a HTTP 401 status, it usually indicates that it cannot find suitable credentials to authenticate with. Can you please verify that your .netrc file located in your home directory contains a line of the form:

machine urs.earthdata.nasa.gov<http://urs.earthdata.nasa.gov> login <username> password <password>

where <uid> and <password> are your Earthdata Login username and password.

If your .netrc file is correct, can you please run a test by adding in the following options to your wget command line:

--http-user=<username> --ask-password

where <username> is your Earthdata Login username.

Thank you,
Peter Smith
Earthdata Login Suport

Ah, I see my mistake; I had a GES DISC address in .netrc instead of URS. Must have been left over from URS3.

Unfortunately, other command line tools, like ncks, do not appear to know how to login with .netrc...





Ticket History
Chris Lynnes (Client) Posted On: 19 September 2016 12:17 PM


Lynnes Christopher S. (GSFC-6102) (Recipient (CC)) Posted On: 19 September 2016 02:54 PM

On Sep 19, 2016, at 2:37 PM, Lynnes, Christopher S. (GSFC-5860) <christophe...@nasa.gov<mailto:christophe...@nasa.gov>> wrote:


On Sep 19, 2016, at 12:32 PM, Peter Smith <sup...@earthdata.nasa.gov<mailto:sup...@earthdata.nasa.gov>> wrote:

Hi Chris

When wget exits upon receiving a HTTP 401 status, it usually indicates that it cannot find suitable credentials to authenticate with. Can you please verify that your .netrc file located in your home directory contains a line of the form:

machine urs.earthdata.nasa.gov<http://urs.earthdata.nasa.gov/> login <username> password <password>

where <uid> and <password> are your Earthdata Login username and password.

If your .netrc file is correct, can you please run a test by adding in the following options to your wget command line:

--http-user=<username> --ask-password

where <username> is your Earthdata Login username.

Thank you,
Peter Smith
Earthdata Login Suport

Ah, I see my mistake; I had a GES DISC address in .netrc instead of URS. Must have been left over from URS3.

Unfortunately, other command line tools, like ncks, do not appear to know how to login with .netrc...

Should embedding username and password in the URL work? http://username:password@host/etc. I can't seem to make it work...






Ticket History
Chris Lynnes (Client) Posted On: 19 September 2016 12:17 PM

Peter Smith (Staff) Posted On: 19 September 2016 03:58 PM

---

Chris

I've not used ncks, but it is similar to ncdump in that it is built on top of netCDF. We should therefore be able to configure it in the same way. The OPeNDAP group has provided some documentation for this:

http://docs.opendap.org/index.php/DAP_Clients_-_Authentication#ncdump

Let me know if you have any problems making this work.

Embedding your username and password into the URL will not work because even if the client supports the syntax, it will send the credentials only to the data server from which you are downloading data - it will not send them to Earthdata Login when the client is redirected for user authentication.

Thanks,
Peter Smith

Lynnes Christopher S. (GSFC-6102) (Recipient (CC)) Posted On: 25 September 2016 06:43 PM

---

Ticket Details

---

[thanh...@gmail.com]

Dear All,

I have the same problems, but I use wget on window via Git Bash. The problems like this.

Connecting to urs.earthdata.nasa.gov (urs.earthdata.nasa.gov)|2001:4d0:241a:4081::89|:443... connected.

ERROR: cannot verify urs.earthdata.nasa.gov's certificate, issued by `/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2014 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1M':

  Unable to locally verify the issuer's authority.

To connect to urs.earthdata.nasa.gov insecurely, use `--no-check-certificate'.

--2016-11-13 16:16:48--  http://disc2.gesdisc.eosdis.nasa.gov/daac-bin/OTF/HTTP_services.cgi?FILENAME=%2Fdata%2FTRMM_L3%2FTRMM_3B42.7%2F2006%2F017%2F3B42.20060117.03.7A.HDF&FORMAT=bmV0Q0RGLw&BBOX=4.22%2C97.03%2C29.18%2C120.76&LABEL=3B42.20060117.03.7A.SUB.nc&SHORTNAME=TRMM_3B42&SERVICE=HDF_TO_NetCDF&VERSION=1.02&DATASET_VERSION=7

Please help me to give some advice!

Thanks a lot,

Nguyen TT

===========

[Nathan Potter]

Greetings Nguyen et al.,

I just examined the TLS (SSL) certificate being offered by urs.earthdata.nasa.gov and it is, in fact, expired:

Not Valid Before: Tuesday, October 20, 2015 at 10:38:28 AM Pacific Daylight Time

Not Valid After:  Friday, October 20, 2017 at 11:08:26 AM Pacific Daylight Time

So it’s not surprising that curl/wget/NetCDF are balking. What’s actually more curious to me is that the three browsers I tested (Safari, Firefox, & Chrome) all reported this certificate as valid, despite the fact that it appears to be expired. Very odd…

Regardless I think that until NASA updates the certificate on that system you will have to do as the software agent suggests and forego checking the certificates validity by utilizing the "--no-check-certificate” option. This option does contain some risk due to the fact that in order to perform the URS authentication your user agent must be configured to follow 302 redirect responses, and this (in conjunction with --no-check-certificate) does promote the possibility that a malicious primary site could get you into a MITM attack. :( Not my favorite path, but I’ve done it in the past and will likely do it again in the future.

Ultimately the admins at urs.earthdata.gov need to update their cert and then I think it should all work for you.

Sincerely,

Nathan 

= = =
Nathan Potter                        ndp at opendap.org
OPeNDAP, Inc.                        +1.541.231.3317

[Nathan Potter]

Hi Nguyen,



First - I misread the urs.earthdata.nasa.gov certificate dates - they’re valid. Sorry about the confusion.

I tried some things on my system and here is what I learned, I am on os-x so my results may not reflect your experience in windows.


Here’s what I found.


Curl works fine for me:

curl -v -k -n -c ursCookies -b ursCookies -L --url "http://airsl2.gesdisc.eosdis.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.006/2003/003/AIRS.2003.01.03.003.L2.RetStd.v6.0.7.0.G13186000743.hdf.nc”

And it worked dropping the -k switch:

curl -v -n -c ursCookies -b ursCookies -L --url "http://airsl2.gesdisc.eosdis.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.006/2003/003/AIRS.2003.01.03.003.L2.RetStd.v6.0.7.0.G13186000743.hdf.nc”


In my .netrc file I have:

machine uat.urs.earthdata.nasa.gov
login ndp_opendap
password ############################


On my system wget doesn’t work for this because it was built without SSL (oy, the hassles of OS X) and so the https protocol is unsupported and wget fails (on my system) to follow the HTTPS redirect.


Sincerely,

Nathan




> On Nov 13, 2016, at 5:47 AM, Nguyen Tien Thanh <thanh...@gmail.com> wrote:
>
> Dear Nathan,
>
> Thank you for your advice. I tried, but not successful with the line command:
>
> wget --no-check-certificate --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition -i
>
> More importantly :
>
> wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --auth-no-challenge=on --keep-session-cookies --content-disposition --http-user=<username> --ask-password -i
>
> All of them is not successful.
>
> Thanks for your help!
>
> Sincerely,
>
> Nguyen

[Nathan Potter]

Hi Nguyen,


I got a full capable version of wget (SSL etc.) for OS-X and I was able to get this to work:

wget --load-cookies cookies --save-cookies cookies --keep-session-cookie "http://airsl2.gesdisc.eosdis.nasa.gov/opendap/Aqua_AIRS_Level2/AIRX2RET.006/2003/003/AIRS.2003.01.03.003.L2.RetStd.v6.0.7.0.G13186000743.hdf.nc”

With the same .netrc file that I used for the curl example.


SO at this point I’m not sure what to recommend to you.


Sorry I can’t be more helpful.


Sincerely,

Nathan

[Nathan Potter]

> On Nov 13, 2016, at 7:42 AM, Nathan Potter <n...@opendap.org> wrote:
>
>

> In my .netrc file I have:
>
> machine uat.urs.earthdata.nasa.gov
> login ndp_opendap
> password ############################

I should have added that in addition to the test server (sat) I have this the production URS server in my .netrc file too:


machine urs.earthdata.nasa.gov
login ndp_opendap
password #######################