Not able to access files from MONARCH thredds server (earth.bsc.es)

[Bas Reijmerink]

Dear sir/madam,

 

I’m currently trying to access data from the dataset below through OpenDAP using either Matlab or Python.

 

https://earth.bsc.es/thredds_dustclim/catalog/monarch-dustclim/3hourly/concdubin8-av_an/catalog.html?dataset=monarch-dustclim/3hourly/concdubin8-av_an

 

However, I fail to access the data, also after setting the .netrc and .dodsrc files in my C:\Users\reijmeri\ folder. I get the following errors:
Matlab: The NetCDF library encountered an error during execution of 'open' function - 'Access failure (-77)'.
Python: [Errno -77] NetCDF: Access failure: 'https://earth.bsc.es/thredds_dustclim/dodsC/monarch-dustclim/3hourly/concdubin8-av_an/concdubin8_2007010103_av_an.nc'

 

The .netrc and .dodsrc files have the following content

 

.netrc content:

 

machine earth.bsc.es

login USERNAME*

password PASSWORD*

* these are replaced by my credentials.

 

.dodsrc content:

 

HTTP.NETRC=C:\Users\reijmeri\.netrc  

HTTP.COOKIEJAR=C:\Users\reijmeri\.dap_cookies  # writable file for login cookies

 

Both Matlab and Python are able to find the .netrc and .dodsrc files, but no luck when accessing the data.

 

Do you by chance have an idea what might go wrong here?

 

p.s. I can access the data via HTTPServer in Matlab (using the credentials), but I rather only collect a subset of the data rather than downloading each complete file.

 

Many thanks in advance for your support.

 

With kind regards,

 

	Bas Reijmerink ? Senior consultant Offshore and Port Hydrodynamics +31 6 46 91 11 52 ? bas.rei...@deltares.nl   Deltares – Enabling Delta Life
Be green, keep it on the screen

 

 

DISCLAIMER: This message is intended exclusively for the addressee(s) and may contain confidential and privileged information. If you are not the intended recipient please notify the sender immediately and destroy this message. Unauthorized use, disclosure or copying of this message is strictly prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, The Netherlands, Commercial Registration Number 41146461, is not liable in any way whatsoever for consequences and/or damages resulting from the improper, incomplete and untimely dispatch, receipt and/or content of this e-mail.

[Miguel Jimenez]

Hello Bas Reijmnerink -- I just noticed your email, sorry about the delayed response. 

I tried to access the data myself also using Python, but I could not find a way to create an account in their system to check if I also experience issues accessing the data. There could be a few things going on on the server side, but without access to the data it may be difficult to verify.

With that being said, I noticed from your description of the netrc file above that you are using a Windows machine. I am not a Windows user myself, but I believe that in Windows systems the authentication file should be named _netrc instead of .netrc. You mention your preferred API can find this file, but I would double check that based on the name. For example, what happens if you try to access data from NASA' Earthdata behind authentication (see example here in pure python: https://pydap.github.io/pydap/en/notebooks/SWOT.html)

I recommend as a simple test, using python, to try to access the data with PyDAP (https://pydap.github.io/pydap/en/intro.html), since it does not require a .dodsrc file. Since PyDAP uses Python's Requests library to access data, it can automatically discover your credentials as long as the _netrc file is located in the default location. Also, since PyDAP is pure python, it may be easier to diagnose any following issue with opendap access.

This may not resolve all your issues. If it does not, feel free to reach out here and/or to the earth.bsc.es system.

Kind regards, 

Miguel Jimenez-Urias