broken download
Jaap Onderwaater
Hi,
I made a search and was supplied a list of links and a download script. Both are not working for me. The link gives me the error below.
Best regards,
Jaap Onderwaater
Username: jacobuso
You do not have permission to access the requested resource.
The specific error message associated with your request was:
ERROR - The HTTP GET request for the source URL: https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/IASI_SST_METOP_B-OSISAF-L2P-v1.0/20181230235956-OSISAF-L2P_GHRSST-SSTskin-IASI_SST_METOP_B-sstiasi_metop01_20181230_235956_32609-v02.0-fv01.0.nc.dmrpp FAILED. CURLINFO_EFFECTIVE_URL: https://urs.earthdata.nasa.gov/oauth/authorize?client_id=HrBaq4rVeFgp2aOo6PoUxA&response_type=code&redirect_uri=https://archive.podaac.earthdata.nasa.gov/login&state=%2Fpodaac-ops-cumulus-protected%2FIASI_SST_METOP_B-OSISAF-L2P-v1.0%2F20181230235956-OSISAF-L2P_GHRSST-SSTskin-IASI_SST_METOP_B-sstiasi_metop01_20181230_235956_32609-v02.0-fv01.0.nc.dmrpp&app_type=401 The response had an HTTP status of 401 which means 'Unauthorized: Contact the server administrator.'
If you think that the server is broken (that the URL you submitted should have worked), then please contact the OPeNDAP user support coordinator at: sup...@opendap.org
Any email message from EUMETSAT is sent in good faith but shall neither be binding nor construed as constituting a commitment by EUMETSAT, except where provided for in a written agreement or contract or if explicitly stated in the email. Please note that any views or opinions presented in this email are solely those of the sender and do not necessarily represent those of EUMETSAT. This message and any attachments are intended for the sole use of the addressee(s) and may contain confidential and privileged information. Any unauthorised use, disclosure, dissemination or distribution (in whole or in part) of its contents is not permitted. If you received this message in error, please notify the sender and delete it from your system.
Jaap Onderwaater
Update:
I can now download from the browser, but the download script still has an issue as shown below (the link that is offered to ensure I have authorisation works).
If I disable curl and force the script to use wget, it actually downloads. I’m using Fedora 34. Considering this I would file this as a bug report, and no longer ask for support.
Best regards,
Jaap
jacobuso:# chmod 777 download.sh
jacobuso:# ./download.sh
Enter your Earthdata Login or other provider supplied credentials
Username (jacobuso):
Password:
Unable to Retrieve Data
Please ensure that you have authorized the remote application by visiting the link below
Nathan Potter
We use command line cURL everyday with these servers and it is very reliable when invited correctly, so I suspect your issue is in how you are using cURL.
If you have not done so yet I think you might want to check out our Hyrax documentation pages. The Earth Data Login section and the authenticating with the cURL client application sections should help you get this issue sorted on your system.
Here is the section about client authentication with EDL:
https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_earthdata_login_users
For the server "https://opendap.earthdata.nasa.gov” you will want to be sure that you have authorized the "Hyrax in the cloud” application for your user.
Here is a link to the authenticating with the cURL client discussion:
https://opendap.github.io/hyrax_guide/Master_Hyrax_Guide.html#_curl_a_k_a_lib_curl
I think if you follow these examples you should have better success with your cURL requests.
Please let us know if that helps, or if you have additional questions.
Sincerely,
Nathan
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317
Jaap Onderwaater
Thank you for your feedback.
I'm still not entirely sure what is the issue, but detect_app_approval in the download script for me gets approved=301. If I simply modify the following if-statement to accept this outcome, the download script also works.
Best regards,
Jaap
-----Original Message-----
From: Nathan Potter <n...@opendap.org>
Sent: Monday 17 January 2022 15:35
To: Jaap Onderwaater <Jaap.Ond...@eumetsat.int>
Cc: Nathan Potter <n...@opendap.org>; OPeNDAP Support <sup...@opendap.org>
Subject: Re: [support] broken download
WARNING: This email originated outside the organisation. Treat links and attachments with caution.
Nathan Potter
I don't have the download script to which you are referring and I was not involved with the production of it. I can't really help you with it unless you provide the complete script to me.
Having said that, an http status of 301 indicates that the resource has been moved, in other words it's a redirect.
If this is coming back from a cURL request then it's likely that adding the "-L" command line switch to the cURL invocation will help. (The "-L" switch tells cURL to follow redirects)
curl -L <AllTheOtherStuff>
Sincerely,
Nathan
Nathan Potter
I tried one of the request URLs in the scripts by hand:
# curl --netrc-file ~/.netrc -c ~/ursCookies -b ~/ursCookies -L -o foo.nc4 "https://opendap.earthdata.nasa.gov/collections/C2036877829-POCLOUD/granules/20181230235956-OSISAF-L2P_GHRSST-SSTskin-IASI_SST_METOP_B-sstiasi_metop01_20181230_235956_32609-v02.0-fv01.0.nc4"
And that worked fine for me. The output is placed in a file called foo.nc4
It's about a 1.5MB file:
# ls -l foo.nc4
-rw-r--r-- 1 ndp staff 152062 Jan 17 11:12 foo.nc4
I checked it with ncdump like this:
# ncdump foo.nc4
And it's valid file.
Can you please try the same exact thing?
You may have to modify the location of the netrc file to match your local system.
If the returned file is a very different size have a look at it with an editor and see if it contains error content.
Thanks,
Nathan
Jaap Onderwaater
That curl works for me too. Looks like just detect_app_approval is getting a different code than expected.