Is it possible for wget to wildcard this file from this URL?

MWBradburne

unread,

Jan 20, 2016, 6:35:16 AM1/20/16

to

Is it possible for wget to wildcard this file from this URL?

Here is the site: http://www.vpngate.net/en/

There is a table there, with the 7th column being "OpenVPN".

In each row, there is a link to an "ovpn" UDP text file whose
filename includes the IP address & port and is of the format:
vpngate_54.201.110.154_udp_1194.ovpn

Manually, I can get wget to download that file using the URL:
$ wget http://www.vpngate.net/common/openvpn_download.aspx?sid=1453289244324&udp=1&host=54.201.110.154&port=1194&hid=3604167&/vpngate_54.201.110.154_udp_1194.ovpn

But, my question is how to *wildcard* that process so that we
(any one of us here) can download *all* the similar ovpn files
that are UDP (for speed) and that have the IP address (so as to
eliminate dns-derived censorship)?

If we had that wildcard syntax, any of us could more easily obtain
dozens of freely available ovpn files for personal use.

MWBradburne

unread,

Jan 20, 2016, 6:51:23 AM1/20/16

to

On Wed, 20 Jan 2016 11:35:15 +0000, MWBradburne wrote:

> Manually, I can get wget to download that file using the URL:
> $ wget http://www.vpngate.net/common/openvpn_download.aspx?sid=1453289244324&udp=1&host=54.201.110.154&port=1194&hid=3604167&/vpngate_54.201.110.154_udp_1194.ovpn

MWBradburne

unread,

Jan 20, 2016, 7:02:18 AM1/20/16

to

The URL changed between sessions from the 1st to the 2nd below:
http://www.vpngate.net/common/openvpn_download.aspx?sid=1453289934632&udp=1&host=49.170.126.127&port=1195&hid=794415&/vpngate_49.170.126.127_udp_1195.ovpn
http://www.vpngate.net/common/openvpn_download.aspx?sid=1453291036464&udp=1&host=49.170.126.127&port=1195&hid=794415&/vpngate_49.170.126.127_udp_1195.ovpn

So, the session ID (sid) seems to be slightly different each time
even though I used the same browser, same public IP address, same
computer, etc., only a few minutes apart to download the same file.

Eef Hartman

unread,

Jan 20, 2016, 7:08:06 AM1/20/16

to

MWBradburne <MWBra...@cisco.com> wrote:
> Is it possible for wget to wildcard this file from this URL?
>
> Here is the site: http://www.vpngate.net/en/

The http protocol doesn't allow wildcards so you'll have to download
the list itself to a local file and then script it to create the
additional wget commands (and then call that script).
That's always a bit of trial and error, especially when you're not
really familiar with tools like sed etc

MWBradburne

unread,

Jan 20, 2016, 7:13:21 AM1/20/16

to

Changing my IP address (via VPN) but keeping everything else the same
resulted in the following two URLs for the same file, the first with
me using one VPN IP address & the second with me changing to a new
VPN IP address:

http://www.vpngate.net/common/openvpn_download.aspx?sid=1453289934616&udp=1&host=1.64.77.253&port=1194&hid=2634121&/vpngate_1.64.77.253_udp_1194.ovpn
http://www.vpngate.net/common/openvpn_download.aspx?sid=1453291391580&udp=1&host=1.64.77.253&port=1194&hid=2634121&/vpngate_1.64.77.253_udp_1194.ovpn

http://www.vpngate.net/common/openvpn_download.aspx?sid=1453289934632&udp=1&host=148.251.18.120&port=1194&hid=3631737&/vpngate_148.251.18.120_udp_1194.ovpn
http://www.vpngate.net/common/openvpn_download.aspx?sid=1453291391580&udp=1&host=148.251.18.120&port=1194&hid=3631737&/vpngate_148.251.18.120_udp_1194.ovpn

Do the only thing that seems to change between sessions is the
"sid", but, the sum total of available files changes also.

J.O. Aho

unread,

Jan 20, 2016, 12:00:37 PM1/20/16

to

You don't have to download the list and create a script of it, wget can
crawl and download all the pages (files) which one page has links to
when using the --recursive ( -r ) option, you may want to limit how many
levels the recursive fetch is done on with --level=X ( -l X ) where X is
the depth you want to go.

wget -r -l 1 http://www.vpngate.net/en/

and I'm assuming the list is on the page
http://www.vpngate.net/en/

--

//Aho

MWBradburne

unread,

Jan 20, 2016, 1:50:19 PM1/20/16

to

On Wed, 20 Jan 2016 18:00:35 +0100, J.O. Aho wrote:

> wget -r -l 1 http://www.vpngate.net/en/
>
> and I'm assuming the list is on the page
> http://www.vpngate.net/en/

$ wget -r -l 1 http://www.vpngate.net/en/
FINISHED --2016-01-20 10:25:18--
Total wall clock time: 2m 55s
Downloaded: 165 files, 4.9M in 1m 34s (53.2 KB/s)

$ ls -ltr
total 4
drwxrwxr-x 7 mwb mwb 4096 Jan 20 10:25 www.vpngate.net

$ ls -ltr ./www.vpngate.net/
total 64
-rw-rw-r-- 1 mwb mwb 40 Mar 6 2013 style_en.css
-rw-rw-r-- 1 mwb mwb 19595 Feb 12 2014 common_style.css
drwxrwxr-x 2 mwb mwb 4096 Jan 20 10:23 ja
drwxrwxr-x 2 mwb mwb 4096 Jan 20 10:23 cn
drwxrwxr-x 3 mwb mwb 4096 Jan 20 10:23 images
drwxrwxr-x 3 mwb mwb 20480 Jan 20 10:25 en
drwxrwxr-x 4 mwb mwb 4096 Jan 20 10:25 api

There's an "index.html" in the "en" (english?) and "cn" (china?)
and "ja" (japan?) directories.

I am looking through all the files to see if I can find
the actual links to the ovpn files that are on the second page.

Once I find that, I can hack out the awk/sed/greps and post back
the script so that everyone benefits from the work in progress.

MWBradburne

unread,

Jan 25, 2016, 7:22:05 PM1/25/16

to

On Wed, 20 Jan 2016 18:00:35 +0100, J.O. Aho wrote:

> You don't have to download the list and create a script of it, wget can
> crawl and download all the pages (files) which one page has links to
> when using the --recursive ( -r ) option, you may want to limit how many
> levels the recursive fetch is done on with --level=X ( -l X ) where X is
> the depth you want to go.
>
>
> wget -r -l 1 http://www.vpngate.net/en/
>
> and I'm assuming the list is on the page
> http://www.vpngate.net/en/

Thanks to your wget suggestion, I found a way to (manually) reconstruct the
URL to all the vpngate ovpn files to feed to wget so that we can download
all the vpngate.net ovpn files in one fell swoop.

1. Use wget at level 1 to obtain HTML files pointing to the ovpn files
2. Use grep to cull out the relative path to the ovpn file on the server
3. Use sed to create the full path to the ovpn file on the server
4. Run wget with the full path to all the ovpn files

Here are the (manual) results of the 4-step process stated above:

Download all the HTML files at level 1:
$ mkdir /tmp/vpnget/
$ cd /tmp/vpnget/
$ wget -r -l 1 http://www.vpngate.net/en/
FINISHED --2016-01-24 23:40:14--
Total wall clock time: 6m 6s
Downloaded: 164 files, 5.3M in 4m 25s (20.4 KB/s)
$ ls ./www.vpngate.net/en/do_openvpn.aspx?fqdn=vpn*
The file names are of the format:
"do_openvpn.aspx?fqdn=vpn717004313.opengw.net&ip=93.48.253.169&tcp=0&udp=50220&sid=1453753857341&hid=2792973"
"do_openvpn.aspx?fqdn=vpn523667150.opengw.net&ip=219.79.35.88&tcp=443&udp=1194&sid=1453753857341&hid=2634121"

From those long file names we can manually reconstruct the URL to the
desired ovpn file but even easier, *inside* of that HTML file is the
full URL to the ovpn file:

Here we see two different OVPN files, one using DNS and one using
the IP address directly, skipping the DNS lookup:
$ grep common\/ *93.48.253.169*
<a href="/common/openvpn_download.aspx?sid=1453753857341&udp=1&host=vpn717004313.opengw.net&port=50220&hid=2792973&/vpngate_vpn717004313.opengw.net_udp_50220.ovpn">
<a href="/common/openvpn_download.aspx?sid=1453753857341&udp=1&host=93.48.253.169&port=50220&hid=2792973&/vpngate_93.48.253.169_udp_50220.ovpn">

Here's another example, but with UDP and TCP variants:
$ grep common\/ *219.79.35.88*
<a href="/common/openvpn_download.aspx?sid=1453753857341&udp=1&host=vpn523667150.opengw.net&port=1194&hid=2634121&/vpngate_vpn523667150.opengw.net_udp_1194.ovpn">
<a href="/common/openvpn_download.aspx?sid=1453753857341&tcp=1&host=vpn523667150.opengw.net&port=443&hid=2634121&/vpngate_vpn523667150.opengw.net_tcp_443.ovpn">
<a href="/common/openvpn_download.aspx?sid=1453753857341&udp=1&host=219.79.35.88&port=1194&hid=2634121&/vpngate_219.79.35.88_udp_1194.ovpn">
<a href="/common/openvpn_download.aspx?sid=1453753857341&tcp=1&host=219.79.35.88&port=443&hid=2634121&/vpngate_219.79.35.88_tcp_443.ovpn">

Pretty much that URL will work just fine for wget:

$ wget "http://www.vpngate.net/common/openvpn_download.aspx?sid=1453755615303&udp=1&host=219.79.35.88&port=1194&hid=2634121&/vpngate_219.79.35.88_udp_1194.ovpn"
That results in the desired *.ovpn file.

So, all I'd need to do is create a sed/awk/grep/whatever script
to do all that stuff above that I did manually.

But, after having said all that above, I think it's easier to
just tell wget to go to level 2 as I will explain next, which
will download that ovpn file directly.

MWBradburne

unread,

Jan 25, 2016, 7:36:59 PM1/25/16

to

On Tue, 26 Jan 2016 00:22:04 +0000, MWBradburne wrote:

> But, after having said all that above, I think it's easier to
> just tell wget to go to level 2 as I will explain next, which
> will download that ovpn file directly.

If I use your suggestion to just get everything up to level 2,
that downloads all the ovpn files automatically!

A. Run wget to download everything up to level 2:

$ mkdir /tmp/vpnget/
$ cd /tmp/vpnget/

$ wget -r -l 2 http://www.vpngate.net/en/

B. Notice you now have *all* the ovpn files!
$ cd /tmp/vpnget/www.vpngate.net/common/
$ ls *93.48.253.169*
openvpn_download.aspx?sid=1453756386281&udp=1&host=93.48.253.169&port=50220&hid=2792973&%2Fvpngate_93.48.253.169_udp_50220.ovpn
$ ls *219.79.35.88*
openvpn_download.aspx?sid=1453756386281&tcp=1&host=219.79.35.88&port=443&hid=2634121&%2Fvpngate_219.79.35.88_tcp_443.ovpn
openvpn_download.aspx?sid=1453756386281&udp=1&host=219.79.35.88&port=1194&hid=2634121&%2Fvpngate_219.79.35.88_udp_1194.ovpn

Those are horribly named but perfectly valid *.ovpn files.

C. Add the fix for dns leaks:
#!/bin/bash
# fixovpn.sh adds three lines to all ovpn files in the directory to stop dns leaks
# https://forum.vpn.ac/discussion/13/running-openvpn-in-linux-terminal-with-no-dns-leaks
# script-security 2
# up /etc/openvpn/update-resolv-conf
# down /etc/openvpn/update-resolv-conf
for filename in *.ovpn; do echo -e "script-security 2\nup /etc/openvpn/update-resolv-conf\ndown /etc/openvpn/update-resolv-conf" >> $filename; done
# WIP. Need to add a check not to do this twice if it's run by mistake a second time!

D. Make the file name a LOT shorter!
$ mv "openvpn_download.aspx?sid=1453756386281&udp=1&host=93.48.253.169&port=50220&hid=2792973&%2Fvpngate_93.48.253.169_udp_50220.ovpn" \
vpngate_93.48.253.169_udp_50220.ovpn

NOTE: I'm doing this step MANUALLY (yuck).
If you have a way of shortening the file names automatically, that would be
a nice improvement. All you need is the IP address because we can then run
the geolocation renaming step below.

E. Optionally, run a geolocation on that file based on the IP address.
NOTE: For consistency, this geolocation-renaming program assumes the
file name is the same as it would be had we manually right clicked on
the link at vpngate.net to download the UDP IP-address ovpn file manually.

#!/bin/bash
# vpngeo.sh renames openvpn *.ovpn files based on their geolocation
lsfile=/tmp/lsovpn.txt
ipfile=/tmp/ipovpn.txt
opfile=/tmp/ipgeo.csv
shfile=/tmp/renameovpn.sh
ls *.ovpn > $lsfile
vi $lsfile
#####
# "lsfile" content is of the format:
# $ touch vpngate_76.64.65.32_udp_1519.ovpn
# $ touch vpngate_76.64.65.32_tcp_1519.ovpn
# $ touch vpngate_88.187.124.69_udp_1195.ovpn
# $ touch vpngate_104.238.116.222_udp_1194.ovpn
# $ touch vpngate_108.224.9.179_udp_1195.ovpn
#####
# strip out just the IP address from the *.ovpn file name
sed -e 's/^[^0-9,.]*//g' -e 's/_.*//g' $lsfile | sort -u -o $ipfile
vi $ipfile
#####
# "ipfile" content is of the format:
# 76.64.65.32
# 88.187.124.69
# 104.238.116.222
# 108.224.9.179
#####
# "opfile" content is of the format:
rm $opfile
for i in $( cat $ipfile ); do echo "$i,\"$( geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $i | cut -d' ' -f4-99 )\""|sed -e 's/, /,/g' -e 's/ \+/-/g' >> $opfile; done
vi $opfile
rm $shfile
#####
# "opfile" content is of the format:
# 76.64.65.32,"Rev-1:-CA,N/A,N/A,N/A,N/A,43.642502,-79.387199,0,0"
# 88.187.124.69,"Rev-1:-FR,B8,Provence-Alpes-Cote-d'Azur,Menton,06500,43.803501,7.496100,0,0"
# 104.238.116.222,"Rev-1:-US,AZ,Arizona,Scottsdale,85260,33.611900,-111.890602,753,480"
# 108.224.9.179,"Rev-1:-US,CA,California,Garden-Grove,92843,33.754501,-117.946404,803,714"
#####
sed -e 's/\"Rev-1:-//g' -e 's/_ /_/g' -e 's/\///g' $opfile|tr -d "'"|awk -F, '{print "mv *"$1"* vpn_"$2"_"$4"_"$5"_"$1".ovpn"}' >> $shfile
vi $shfile
#####
# "shfile" content is of the format:
# mv *76.64.65.32* vpn_CA_NA_NA_76.64.65.32.ovpn
# mv *88.187.124.69* vpn_FR_Provence-Alpes-Cote-dAzur_Menton_88.187.124.69.ovpn
# mv *104.238.116.222* vpn_US_Arizona_Scottsdale_104.238.116.222.ovpn
# mv *108.224.9.179* vpn_US_California_Garden-Grove_108.224.9.179.ovpn
#####
chmod u+x $shfile
sh $shfile
exit 0

F. Now that the files have a short name, just run Marek's vpnrun on the directory!
$ vpnrun.sh
(which runs the following command, but gives you choices what do to if it fails):
$ sudo openvpn --config vpn_US_California_Garden-Grove_108.224.9.179.ovpn

I realize this process is ripe for improvement, but, since it would give
all of us easier access to freeware VPN servers, I post this for others
to benefit from.

I'm having trouble figuring out how to shorten the huge original file names
so that the geolocation program can rename them automatically, so, I'm doing
that ugly filename shortening step manually (yuck).