Proxy Server Configuration with NetCDF4/PyDAP

260 views
Skip to first unread message

Nathan Potter

unread,
Feb 21, 2017, 6:57:53 PM2/21/17
to Donato Moscara, Support Unidata, Nathan Potter, support@opendap.org support
On February 13, 2017 at 07:42:01, Donato Moscara (mos...@istat.it) wrote:

> Hi, in order to evaluate the best suitable tool for us for getting a dataset from an openadap server I built a client using either C++ and Java. They both worked fine. I specified http proxy server value putting it inside .dodsrc file (C++ source program) or setting command line variables -Dhttp.proxyHost=x.x.x -Dhttp.proxyPort=xxxx (Java source program). So I could get data from an opendap server from the Net, outside our Company local area network.

Nice work!

>
> I then tried to build an opendap python client using either PyDap library, installed with PIP, and NetCDF4 python library. Both PyDap and NetCDF4 client worked fine and I got opendap data accessing an opendap server but only inside our local area network. Unfortunately I wasn't able to set http proxy for getting data from an remote opendap server. I couldn't find on line suitable documentation.

I am not entirely sure, but I think that if you can get the nctools (such as ncdump) working with your proxy server then that configuration will be utilized by PyDAP and by the NetCDF4 library. The ncdump is smart about OPeNDAP - you can use DAP URLs with them as well as file names:

ncdump -h http://test.opendap.org/opendap/data/nc/fnoc1.nc

I too was unable to locate a useful online discussion of how to configure the NetCDF4 stack for a Proxy server.

Interestingly I think it actually appears, as comments, in a page I wrote about authentication:
http://docs.opendap.org/index.php/DAP_Clients_-_Authentication

Specifically, read the section about ncdump:
http://docs.opendap.org/index.php/DAP_Clients_-_Authentication#ncdump

To make it work for authentication you have to edit two files, “.dodsrc” and “.netrc”

In the “.dodsrc” example you can see where the lines that control the proxy server are commented out.

The libcurl library is what utilizes the .netrc file in the authentication example, but I don’t think it’s likely to play a roll in the proxy-configuration.

IF what little I have offered here gets things going for you then great!

But I suspect we won’t be so lucky and so I am cc’ing UNIDATA support to see if they have any thoughts about how to make this work.

Thanks for your patience.


Sincerely,


Nathan


> Also PyDap.util package, used in many examples, seems not be included in my pydap library installation. I use Linux RHEL6 server.



>
>
> Please, could you suggest me how resolve this issues? I would like to build a client, behind an http proxy, accessing to a remote opendap data server using python. Is ther any other python library that I could use other than PyDap and NetCDF4 libraries?
>
> Best regards.
>
> Donato Moscara
>
> ------
> Istituto Nazionale di Statistica
> Ing. Donato Moscara
> DIRM - DCIT - ITA
> Via Balbo 16, 00184 Roma
> Tel +39.06.4673.2173
> ------






= = =
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317

Donato Moscara

unread,
Feb 23, 2017, 3:28:15 AM2/23/17
to Nathan Potter, Support Unidata, support
        Hi Nathan, thank you for your support. I apologize for the very long email content: I hope this could be useful to some other user. 

the latest rpm netcdf package installed on my linux server is:  netcdf-4.1.1-3.el6.5.x86_64
...The following works with the ncdump (and oc client) code bundled with NetCDF-4.3.3.1 Previous versions including 4.3.2 and 4.3.1 will not work...
and supposed that ...The ncdump is smart about OPeNDAP - you can use DAP URLs with them as well as file names:

In order to build latest netcdf package I downloaded hdf5-1.10.0-patch1.tar under /dapuser directory
then untar
./configure --prefix=$HOME  --enable-java  --enable-cxx
then make and install
it worked!

then I downloaded  netcdf-4.4.1.1.tar
then untar
export CPPFLAGS=-I/dapuser/include
export LDFLAGS="-L/dapuser/netcdf-4.4.1.1/liblib/.libs/ -L/dapuser/lib/"
./configure  --disable-hdf4  --prefix=/dapuser/netcdf-4.4.1.1
then make, install
it worked too!

Then I tried using the new built ncdump tool
note: hyrax-test is a server inside my local area network (don't need any proxy). 
It worked!

Then I tried
It didn't work!

This is content of my .dodsrc file
-bash-4.1$ cat /dapuser/.dodsrc
# No proxy authentication needed

I verified using the command:
 strace -o /dapuser/strace.txt  /dapuser/netcdf-4.4.1.1/ncdump/ncdump  http://test.opendap.org/opendap/data/nc/coads_climatology.nc?COADSX[0:1:10],COADSY[0:1:5]

strace.txt file content shows that ncdump tool opens and reads the .dodsrc file.

open("./.daprc", O_RDONLY)              = -1 ENOENT (No such file or directory)
open("/dapuser/.daprc", O_RDONLY)       = -1 ENOENT (No such file or directory)
open("./.dodsrc", O_RDONLY)             = -1 ENOENT (No such file or directory)
open("/dapuser/.dodsrc", O_RDONLY)      = 3
close(3)                                = 0
open("/dapuser/.dodsrc", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=51, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f582e4000
read(3, "PROXY_SERVER=http,http://myproxy"..., 4096) = 51
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f6f582e4000, 4096)

I also used .netrc file

-bash-4.1$ cat /dapuser/.dodsrc
USE_CACHE=0
MAX_CACHE_SIZE=20
MAX_CACHED_OBJ=5
IGNORE_EXPIRES=0
CACHE_ROOT=/dapuser/.dods_cache/
DEFAULT_EXPIRES=1
ALWAYS_VALIDATE=1
DEFLATE=0
HTTP.COOKIEJAR=/dapuser/.cookies
HTTP.NETRC=/dapuser/.netrc

-bash-4.1$ cat  /dapuser/.netrc
machine http://myproxy.istat.it:3128
login
password

strace.txt file content shows that ncdump tool opens and reads .dodsrc and .netrc files
...
open("/dapuser/.netrc", O_RDONLY)       = 3
fstat(3, {st_mode=S_IFREG|0600, st_size=57, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa497113000
read(3, "machine http://myproxy.istat."..., 4096) = 57
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7fa497113000, 4096)            = 0

But it didn't work. Also setting proxy inside /dapuser/.curlrc file didn't work.

But the solution was easier than I could have imagined.

I just set and exported an environment variable:


and finally it worked fine!

Now I am going to verify if a PyDap client works too.

Best regards.

Donato.


------
Istituto Nazionale di Statistica
Ing. Donato Moscara
DIRM - DCIT - ITA
Via Balbo 16, 00184 Roma
Tel +39.06.4673.2173
------

Da: "Nathan Potter" <n...@opendap.org>
A: "Donato Moscara" <mos...@istat.it>, "Support Unidata" <sup...@unidata.ucar.edu>
Cc: "Nathan Potter" <n...@opendap.org>, "support" <sup...@opendap.org>
Inviato: Mercoledì, 22 febbraio 2017 0:57:50
Oggetto: Proxy Server Configuration with NetCDF4/PyDAP
On February 13, 2017 at 07:42:01, Donato Moscara (mos...@istat.it) wrote:

>           Hi, in order to evaluate the best suitable tool for us for getting a dataset from an openadap server I built a client using either C++ and Java. They both worked fine. I specified http proxy server value putting it inside .dodsrc file (C++ source program) or setting command line variables   -Dhttp.proxyHost=x.x.x   -Dhttp.proxyPort=xxxx (Java source program). So I could get data from an opendap server from the Net, outside our Company local area network.

Nice work!

>
>            I then tried to build an opendap python client using either PyDap library, installed with PIP,  and NetCDF4 python library. Both PyDap and NetCDF4 client worked fine and I got opendap data accessing an opendap server but only inside our local area network. Unfortunately I wasn't able to set http proxy for getting data from an remote opendap server. I couldn't find on line suitable documentation.

I am not entirely sure, but I think that if you can get the nctools (such as ncdump) working with your proxy server then that configuration will be utilized by PyDAP and by the NetCDF4 library. The ncdump is smart about OPeNDAP - you can use DAP URLs with them as well as file names:



I too was unable to locate a useful online discussion of how to configure the NetCDF4 stack for a Proxy server.

Interestingly I think it actually appears, as comments, in a page I wrote about authentication:


Specifically, read the section about ncdump:


To make it work for authentication you have to edit two files, “.dodsrc” and “.netrc”

In the “.dodsrc” example you can see where the lines that control the proxy server are commented out.

The libcurl library is what utilizes the .netrc file in the authentication example, but I don’t think it’s likely to play a roll in the proxy-configuration.

IF what little I have offered here gets things going for you then great!

But I suspect we won’t be so lucky and so I am cc’ing UNIDATA support to see if they have any thoughts about how to make this work.

Thanks for your patience.


Sincerely,


Nathan


> Also PyDap.util package, used in many examples, seems not be included in my pydap library installation. I use Linux RHEL6 server.      



>  
>
>           Please, could you suggest me how resolve this issues? I would like to build a client, behind an http proxy, accessing to a remote opendap data server using python. Is ther any other python library that I could use other than PyDap and NetCDF4 libraries?
>
>           Best regards.
>
>           Donato Moscara
>
> ------
> Istituto Nazionale di Statistica
> Ing. Donato Moscara
> DIRM - DCIT - ITA
> Via Balbo 16, 00184 Roma
> Tel +39.06.4673.2173
> ------






= = =
Nathan Potter                        ndp at https://urlsand.esvalabs.com/?u=http%3A%2F%2Fopendap.org&e=e7a274c6&h=3a8a8afa&f=n
OPeNDAP, Inc.                        +1.541.231.3317

Nathan Potter

unread,
Feb 23, 2017, 9:26:02 AM2/23/17
to Donato Moscara, Nathan Potter, Support Unidata, support
Hi Donato,

That is great news, I didn’t know this thing about the netcdf library where the environment variable is used to control the proxy.

Thanks for the tip!


Sincerely,

Nathan

> export http_proxy=http://myproxy.istat.it:3128
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317

Donato Moscara

unread,
Feb 24, 2017, 4:00:28 AM2/24/17
to Nathan Potter, Support Unidata, support
         Hi Nathan, I also tried to run the script tagged Example1, reported at  http://polar.ncep.noaa.gov/waves/examples/usingpython.shtml
that use python and netcdf4-python package, this time running the script on my Microsoft Windows 7 PC (until now I used a Linux Red Hat 6 x64 server).
    
         First I run it from command line inside a DOS command window and I got a CURL package error reported as "...couldn't contact server...".

         Then at DOS command prompt I set http_proxy environment variable as below

         SET http_proxy=http://myproxy.istat.it:3128

         before running the script  and then it worked fine
         
         Best regards.

         Donato.

------
Istituto Nazionale di Statistica
Ing. Donato Moscara
DIRM - DCIT - ITA
Via Balbo 16, 00184 Roma
Tel +39.06.4673.2173
------

Da: "Nathan Potter" <n...@opendap.org>
A: "Donato Moscara" <mos...@istat.it>
Cc: "Nathan Potter" <n...@opendap.org>, "Support Unidata" <sup...@unidata.ucar.edu>, "support" <sup...@opendap.org>
Inviato: Giovedì, 23 febbraio 2017 15:26:03
Oggetto: Re: [support] Re: Proxy Server Configuration with NetCDF4/PyDAP
Hi Donato,

That is great news, I didn’t know this thing about the netcdf library where the environment variable is used to control the proxy.

Thanks for the tip!


Sincerely,

Nathan




> On Feb 23, 2017, at 12:28 AM, Donato Moscara <mos...@istat.it> wrote:
>
>         Hi Nathan, thank you for your support. I apologize for the very long email content: I hope this could be useful to some other user.
>
> the latest rpm netcdf package installed on my linux server is:  netcdf-4.1.1-3.el6.5.x86_64

> ...The following works with the ncdump (and oc client) code bundled with NetCDF-4.3.3.1 Previous versions including 4.3.2 and 4.3.1 will not work...
> and supposed that ...The ncdump is smart about OPeNDAP - you can use DAP URLs with them as well as file names:
>
> In order to build latest netcdf package I downloaded hdf5-1.10.0-patch1.tar under /dapuser directory
> then untar
> ./configure --prefix=$HOME  --enable-java  --enable-cxx
> then make and install
> it worked!
>
> then I downloaded  netcdf-4.4.1.1.tar
> then untar
> export CPPFLAGS=-I/dapuser/include
> export LDFLAGS="-L/dapuser/netcdf-4.4.1.1/liblib/.libs/ -L/dapuser/lib/"
> ./configure  --disable-hdf4  --prefix=/dapuser/netcdf-4.4.1.1
> then make, install
> it worked too!
>
> Then I tried using the new built ncdump tool

> note: hyrax-test is a server inside my local area network (don't need any proxy).
> It worked!
>
> Then I tried

> It didn't work!
>
> This is content of my .dodsrc file
> -bash-4.1$ cat /dapuser/.dodsrc

> # No proxy authentication needed
>
> I verified using the command:

>
> strace.txt file content shows that ncdump tool opens and reads the .dodsrc file.
>
> open("./.daprc", O_RDONLY)              = -1 ENOENT (No such file or directory)
> open("/dapuser/.daprc", O_RDONLY)       = -1 ENOENT (No such file or directory)
> open("./.dodsrc", O_RDONLY)             = -1 ENOENT (No such file or directory)
> open("/dapuser/.dodsrc", O_RDONLY)      = 3
> close(3)                                = 0
> open("/dapuser/.dodsrc", O_RDONLY)      = 3
> fstat(3, {st_mode=S_IFREG|0664, st_size=51, ...}) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6f582e4000

> read(3, "", 4096)                       = 0
> close(3)                                = 0
> munmap(0x7f6f582e4000, 4096)
>
> I also used .netrc file
>
> -bash-4.1$ cat /dapuser/.dodsrc
> USE_CACHE=0
> MAX_CACHE_SIZE=20
> MAX_CACHED_OBJ=5
> IGNORE_EXPIRES=0
> CACHE_ROOT=/dapuser/.dods_cache/
> DEFAULT_EXPIRES=1
> ALWAYS_VALIDATE=1
> DEFLATE=0
> HTTP.COOKIEJAR=/dapuser/.cookies
> HTTP.NETRC=/dapuser/.netrc
>
> -bash-4.1$ cat  /dapuser/.netrc

> login
> password
>
> strace.txt file content shows that ncdump tool opens and reads .dodsrc and .netrc files
> ...
> open("/dapuser/.netrc", O_RDONLY)       = 3
> fstat(3, {st_mode=S_IFREG|0600, st_size=57, ...}) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa497113000

> read(3, "", 4096)                       = 0
> close(3)                                = 0
> munmap(0x7fa497113000, 4096)            = 0
>
> But it didn't work. Also setting proxy inside /dapuser/.curlrc file didn't work.
>
> But the solution was easier than I could have imagined.
>
> I just set and exported an environment variable:
>

>
> and finally it worked fine!
>
> Now I am going to verify if a PyDap client works too.
>
> Best regards.
>
> Donato.
>
>
> ------
> Istituto Nazionale di Statistica
> Ing. Donato Moscara
> DIRM - DCIT - ITA
> Via Balbo 16, 00184 Roma
> Tel +39.06.4673.2173
> ------
>
> Da: "Nathan Potter" <n...@opendap.org>
> A: "Donato Moscara" <mos...@istat.it>, "Support Unidata" <sup...@unidata.ucar.edu>
> Cc: "Nathan Potter" <n...@opendap.org>, "support" <sup...@opendap.org>
> Inviato: Mercoledì, 22 febbraio 2017 0:57:50
> Oggetto: Proxy Server Configuration with NetCDF4/PyDAP
> On February 13, 2017 at 07:42:01, Donato Moscara (mos...@istat.it) wrote:
>
> >           Hi, in order to evaluate the best suitable tool for us for getting a dataset from an openadap server I built a client using either C++ and Java. They both worked fine. I specified http proxy server value putting it inside .dodsrc file (C++ source program) or setting command line variables   -Dhttp.proxyHost=x.x.x   -Dhttp.proxyPort=xxxx (Java source program). So I could get data from an opendap server from the Net, outside our Company local area network.
>
> Nice work!
>
> >
> >            I then tried to build an opendap python client using either PyDap library, installed with PIP,  and NetCDF4 python library. Both PyDap and NetCDF4 client worked fine and I got opendap data accessing an opendap server but only inside our local area network. Unfortunately I wasn't able to set http proxy for getting data from an remote opendap server. I couldn't find on line suitable documentation.
>
> I am not entirely sure, but I think that if you can get the nctools (such as ncdump) working with your proxy server then that configuration will be utilized by PyDAP and by the NetCDF4 library. The ncdump is smart about OPeNDAP - you can use DAP URLs with them as well as file names:
>

>
> I too was unable to locate a useful online discussion of how to configure the NetCDF4 stack for a Proxy server.
>
> Interestingly I think it actually appears, as comments, in a page I wrote about authentication:

>
> Specifically, read the section about ncdump:

>
> To make it work for authentication you have to edit two files, “.dodsrc” and “.netrc”
>
> In the “.dodsrc” example you can see where the lines that control the proxy server are commented out.
>
> The libcurl library is what utilizes the .netrc file in the authentication example, but I don’t think it’s likely to play a roll in the proxy-configuration.
>
> IF what little I have offered here gets things going for you then great!
>
> But I suspect we won’t be so lucky and so I am cc’ing UNIDATA support to see if they have any thoughts about how to make this work.
>
> Thanks for your patience.
>
>
> Sincerely,
>
>
> Nathan
>
>
> > Also PyDap.util package, used in many examples, seems not be included in my pydap library installation. I use Linux RHEL6 server.      
>
>
>
> >  
> >
> >           Please, could you suggest me how resolve this issues? I would like to build a client, behind an http proxy, accessing to a remote opendap data server using python. Is ther any other python library that I could use other than PyDap and NetCDF4 libraries?
> >
> >           Best regards.
> >
> >           Donato Moscara
> >
> > ------
> > Istituto Nazionale di Statistica
> > Ing. Donato Moscara
> > DIRM - DCIT - ITA
> > Via Balbo 16, 00184 Roma
> > Tel +39.06.4673.2173
> > ------
>
>
>
>
>
>
> = = =
Reply all
Reply to author
Forward
0 new messages