Free openvpn config using efficient scripts (please improve so all benefit)

[MWBradburne]

How to obtain free openvpn config using efficient scripts
The result in a few minutes is about 300 current VPN configuration files!
(please improve so all benefit)

1. Download all the current available free *.ovpn config files:
(solution courtesy of J.O. Aho)
$ wget -r -l 2 http://www.vpngate.net/en/
...
FINISHED --2016-01-26 22:49:48--
Total wall clock time: 15m 10s
Downloaded: 1306 files, 27M in 7m 2s (66.6 KB/s)

Notice that the ovpn files have a horrid very long file name:
$ ls ./www.vpngate.net/common/*
openvpn_download.aspx?sid=1453756386281&tcp=1&host=104.36.16.13&port=443&hid=3682177&%2Fvpngate_104.36.16.13_tcp_443.ovpn
openvpn_download.aspx?sid=1453756386281&tcp=1&host=106.0.176.61&port=443&hid=460215&%2Fvpngate_106.0.176.61_tcp_443.ovpn
openvpn_download.aspx?sid=1453756386281&tcp=1&host=119.76.36.35&port=1597&hid=2976869&%2Fvpngate_119.76.36.35_tcp_1597.ovpn

2. Delete unnecessary directories & unnecessary files:
$ rm -rf ./www.vpngate.net/{api,images,ja,cn,common_images,en}
$ rm -rf ./www.vpngate.net/{common_style.css,style_en.css,index.html,style_cn.css,style_ja.css}
$ rm -rf ./www.vpngate.net/common/mail{1,2}.jpg

3. Add dns leak fix for openvpn when run from the command line:
(solution courtesy of JG Miller and many others)
$ fixovpn.sh (see script below)
NOTE: This appends 3 lines to each ovpn file:
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

WIP: This script needs to be fixed to never add the lines twice!

4. Convert long names to the *same* short names that a manual download would get:
(solution courtesy of Marek Novotny & William Unruh)
$ for i in ./www.vpngate.net/common/*.ovpn;do mv $i $(echo $i|awk -F'F' '{print $2}');done
vpngate_104.36.16.13_tcp_443.ovpn
vpngate_106.0.176.61_tcp_443.ovpn
vpngate_119.76.36.35_tcp_1597.ovpn

5. Rename those short names to geotagged file names (country, state, city):
(solution courtesy of Eef Hartman)
$ vpngeo.sh (see script below)
vpn_TH_Krung-Thep_Bangkok_106.0.176.61.ovpn
vpn_TH_Krung-Thep_Bangkok_119.76.36.35.ovpn
vpn_US_California_Palo-Alto_104.36.16.13.ovpn

WIP: This script needs to be fixed to ignore ovpn files that don't
have an IP address in the "remote" line (or in the file name).
(Currently I sort & manually delete them from the "mv" list.)

6. Run openvpn using Marek's vpntest.sh (which allows ditching of bad config files):
(solution courtesy of Marek Novotny)
$ vpnrun.sh (I renamed vpntest to vpnrun because I use it to run openvpn)

The result in a few minutes is about 300 good VPN configuration files!

PLEASE IMPROVE SO WE ALL BENEFIT!

Separately I will post the following files:
< fixdnsleak.sh for fixing the dns leak issue with openvpn >
< fixgeotag.sh for converting file names to geotagged file names >
< runopenvpn.sh although you're better off getting vpntest.sh from Marek Novotny >

[MWBradburne]

On Wed, 27 Jan 2016 07:18:25 +0000, MWBradburne wrote:

> Separately I will post the following files:
> < fixdnsleak.sh for fixing the dns leak issue with openvpn >
> < fixgeotag.sh for converting file names to geotagged file names >
> < runopenvpn.sh although you're better off getting vpntest.sh from Marek Novotny >

Here is the script to fix the ovpn files in step 3.

#!/bin/bash
# fixdnsleak.sh adds three lines to all ovpn files in the directory to stop dns leaks
# https://forum.vpn.ac/discussion/13/running-openvpn-in-linux-terminal-with-no-dns-leaks
# WIP: Needs some way to prevent it from doing it twice if run accidentally again.
# script-security 2
# up /etc/openvpn/update-resolv-conf
# down /etc/openvpn/update-resolv-conf
for filename in *.ovpn; do echo -e "script-security 2\nup /etc/openvpn/update-resolv-conf\ndown /etc/openvpn/update-resolv-conf" >> $filename; done
# end

[MWBradburne]

On Wed, 27 Jan 2016 07:18:25 +0000, MWBradburne wrote:

> Separately I will post the following files:
> < fixdnsleak.sh for fixing the dns leak issue with openvpn >
> < fixgeotag.sh for converting file names to geotagged file names >
> < runopenvpn.sh although you're better off getting vpntest.sh from Marek Novotny >

Here is the fixgeotag.sh hack which takes files that have IP
addresses & converts the file names to country, city, & state.

#!/bin/bash
# fixgeotag.sh renames openvpn *.ovpn files based on their geolocation
# WIP: It really should get the IP address from inside the file.
# WIP: It really should ignore files that use DNS instead (no IP address).
# WIP: Or, it could run a whois to figure out the IP address for geotagging.
lsfile=/tmp/lsovpn.txt
ipfile=/tmp/ipovpn.txt
opfile=/tmp/ipgeo.csv
shfile=/tmp/renameovpn.sh
ls *.ovpn > $lsfile
vi $lsfile
#####
# "lsfile" content is of the format:
# $ touch vpngate_76.64.65.32_udp_1519.ovpn
# $ touch vpngate_76.64.65.32_tcp_1519.ovpn
# $ touch vpngate_88.187.124.69_udp_1195.ovpn
# $ touch vpngate_104.238.116.222_udp_1194.ovpn
# $ touch vpngate_108.224.9.179_udp_1195.ovpn
#####
# strip out just the IP address from the *.ovpn file name
sed -e 's/^[^0-9,.]*//g' -e 's/_.*//g' $lsfile | sort -u -o $ipfile
vi $ipfile
#####
# "ipfile" content is of the format:
# 76.64.65.32
# 88.187.124.69
# 104.238.116.222
# 108.224.9.179
#####
# "opfile" content is of the format:
rm $opfile
for i in $( cat $ipfile ); do echo "$i,\"$( geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $i | cut -d' ' -f4-99 )\""|sed -e 's/, /,/g' -e 's/ \+/-/g' >> $opfile; done
vi $opfile
rm $shfile
#####
# "opfile" content is of the format:
# 76.64.65.32,"Rev-1:-CA,N/A,N/A,N/A,N/A,43.642502,-79.387199,0,0"
# 88.187.124.69,"Rev-1:-FR,B8,Provence-Alpes-Cote-d'Azur,Menton,06500,43.803501,7.496100,0,0"
# 104.238.116.222,"Rev-1:-US,AZ,Arizona,Scottsdale,85260,33.611900,-111.890602,753,480"
# 108.224.9.179,"Rev-1:-US,CA,California,Garden-Grove,92843,33.754501,-117.946404,803,714"
#####
sed -e 's/\"Rev-1:-//g' -e 's/_ /_/g' -e 's/\///g' $opfile|tr -d "'"|awk -F, '{print "mv *"$1"* vpn_"$2"_"$4"_"$5"_"$1".ovpn"}' >> $shfile
vi $shfile
#####
# "shfile" content is of the format:
# mv *76.64.65.32* vpn_CA_NA_NA_76.64.65.32.ovpn
# mv *88.187.124.69* vpn_FR_Provence-Alpes-Cote-dAzur_Menton_88.187.124.69.ovpn
# mv *104.238.116.222* vpn_US_Arizona_Scottsdale_104.238.116.222.ovpn
# mv *108.224.9.179* vpn_US_California_Garden-Grove_108.224.9.179.ovpn
#####
chmod u+x $shfile
sh $shfile
exit 0
# end of fixgeotag.sh

[MWBradburne]

> Separately I will post the following files:
> < fixdnsleak.sh for fixing the dns leak issue with openvpn >
> < fixgeotag.sh for converting file names to geotagged file names >
> < runopenvpn.sh although you're better off getting vpntest.sh from Marek Novotny >

You are better off getting this script from Marek Novotny himself!

Here is my hack of Marek's vpntest.sh which I've renamed to runopenvpn.sh
because I use it to run openvpn in a directory that is filled to the brim
with freeware openvpn files.

This script will kick out bad files, so that, over time, only
good files remain in the current directory.

This script *also* acts as your openvpn run script.

#!/bin/bash

#########################################################
#
# script: vpntest.sh
# version: .01
# date: 2015-11-24
# purpose: allow testing of a directory of
# ovpn files, and offer to keep those
# which are valid in separate directory
#
#########################################################
# Note that a config that is bad today may be good tomorrow.
BAD_DIR=../bad
mkdir -p $BAD_DIR

kickOrKeep () {
echo ""
read -p "Was that successful? " answer
case $answer in
[Nn]* )
mv -i "$x" $BAD_DIR
;;
[Yy]* )
;;
* )
kickOrKeep
;;
esac
}


for x in $(ls *.ovpn) ; do
sudo openvpn --config "$x"
kickOrKeep
done

[MWBradburne]

On Wed, 27 Jan 2016 07:18:25 +0000, MWBradburne wrote:

> Separately I will post the following files:
> < fixdnsleak.sh for fixing the dns leak issue with openvpn >
> < fixgeotag.sh for converting file names to geotagged file names >
> < runopenvpn.sh although you're better off getting vpntest.sh from Marek Novotny >

In addition, it's a good idea, after you are on VPN, to run a script
which will keep watch on the tunnel, and which will kill sensitive
apps immediately if the tunnel fails (which it does from time to time).

Again, this is Marek Novotny's script, which he named "vpnstatus.sh"
but which I have renamed "vpnwatch.sh" for my use and which is best
obtained from him.

#!/bin/bash

#############################################################
#
# script: vpnstatus.sh (which I've renamed vpnwatch.sh)
# written by: Marek Novotny
# version: 2.8
# date: Mon Dec 28 05:41:00 PST 2015
# purpose: test status of live vpn connection
# : kill torrent if vpn disconnects
# licence: GPL v2 (only)
# github: https://github.com/marek-novotny
# readme:
#
#############################################################

condition=""

sendMessage()
{
echo "$1"
}

# apps that should be terminated if VPN fails
# processList=("transmission" "firefox" "pan")
processList=("transmission" "firefox" "pan" "slrn")
# NOTE this also kills the tor browser bundle (which is fine).

# apps that should not be running under vpn
restrictedApps=("thunderbird")

# check of a process stored in the variable task is running or not

checkProcess()
{
unset procID
procID="$(ps -e | grep $task | grep -v panel | awk '{print $1}')"
if [ ! -z $procID ] ; then
return 0
else
return 1
fi
}

# terminate the given process stored in the variable task

terminateProcess()
{
kill -9 $procID
killall firefox
killall transmission
killall pan

}

# routine to test for processes, report their status and kill them if running

processTerminator()
{
checkProcess
if (($? == 0)) ; then
sendMessage "$task is running..."
sendMessage "Terminating $task..."
terminateProcess
if (($? == 0)) ; then
sendMessage "$task terminated..."
else
sendMessage "$task is still running..."
fi
fi
}

# generate a random IP to test ip route against

randomizer()
{
IFS=$' '
ary=()
for x in {1..4} ; do
ary+=($(($RANDOM % 221 + 1)))
done

if [[ ${ary[0]} -eq 10 || ${ary[0]} -eq 100 ]] ; then
randomizer
elif [[ ${ary[0]} -eq 169 ]] && [[ ${ary[1]} -eq 254 ]] ; then
randomizer
elif [[ ${ary[0]} -eq 172 ]] && [[ ${ary[1]} -eq 16 ]] ; then
randomizer
elif [[ ${ary[0]} -eq 192 ]] && [[ ${ary[1]} -eq 168 ]] ; then
randomizer
elif [[ ${ary[0]} -eq 198 ]] && [[ ${ary[1]} -eq 18 ]] ; then
randomizer
else
addr=$(echo "${ary[@]}" | awk '{print $1"."$2"."$3"."$4}')
fi
}

# kill apps that should not be running if VPN is connected.
# kills these apps once, if the script is running and the VPN
# tunnel becomes active

vpnOn()
{
if [[ $condition != "on" ]] ; then
condition="on"
echo "VPN status: $condition - ${devType[0]}: ${devType[1]}"

for x in ${restrictedApps[@]} ; do
task=$x
processTerminator
done
fi
}

# drop apps that should not be running if vpn tunnel fails

vpnOff()
{
if [[ $condition != "off" ]] ; then
condition="off"
echo "VPN status: $condition - ${devType[0]}: ${devType[1]}"
echo "Terminating apps..."

for x in ${processList[@]} ; do
task=$x
processTerminator
done
fi
}

randomizer
while true ; do
devType=($(ip route get $addr | awk 'NR==1 {print $(NF-2),$(NF-0)}'))
if [[ ${devType[0]} == tun? || ${devType[0]} == ppp? ]] ; then
vpnOn
else
vpnOff
fi
done

#END

[MWBradburne]

On Wed, 27 Jan 2016 07:18:25 +0000, MWBradburne wrote:

> PLEASE IMPROVE SO WE ALL BENEFIT!

This script prevents certain apps from running while you're on VPN.

You should get this next file from Marek Novotny himself.

Google Gmail, in particular, hates when you log in from a different
IP address upon every invocation. Eventually Gmail will LOCK YOU OUT
FOREVER (ask me how I know) if you don't ALSO give it a phone number.

Since I never give out my phone number, I can't afford to be locked
out of Gmail so Marek's mlaunch.sh is critical for me to use instead
of thunderbird.sh.

Also, using some apps while on VPN could give away too much information
to any program you accidentally launch when you're on VPN.

But I only have my Ubuntu Thunderbird icon set to invoke Marek's
mlaunch.sh script below.

#!/bin/bash

###############################################################
#
# script: mlaunch.sh
# version: .10 beta
# purpose: launch approved apps if using approved IP
# date: Fri Aug 07 2015 06:12PM
# by: marek novotny
#
# revisions: added ping test
# : added results test for obtaining IP info
# : using only public IP addressing
# : added xmessage features
# : changed name to mlaunch (multi launch)
# : added array for multi-app support
# : various code improvements.
#
# notes: creates $HOME/.mlaunch with approved
# : public IP for launching apps.
# : more IPs can be added if needed.
# : first approved ip is stored in
# : $HOME/.mlaunch and used to compare
# : to existing public IP for launch
#
# requirements: wget or curl, xmessage and the apps
# : apps you wish to launch.
#
################################################################

# apps=("firefox" "thunderbird")
apps=("thunderbird")
myApps="${apps[@]}"

sendMessage()
{
xmessage -display $DISPLAY -fg white -bg purple \
-title "${title}" -geom +60+30 -timeout 4 \
-buttons Okay:1 -default Okay "$1"
}

sendError()
{
xmessage -display $DISPLAY -fg black -bg orange \
-title "${title}" -geom +60+30 \
-buttons Okay:1 -default Okay "$2"
exit $1
}

sendPrompt()
{
xmessage -display $DISPLAY -fg black -bg green \
-title "${title}" -geom +60+30 \
-buttons "$buttons" -default "$default" "$1"
}

setupDisplay()
{
if [[ -z "$DISPLAY" ]]
then
DISPLAY=':0'
fi
}

checkCommands()
{
if [[ ! $(which xmessage) ]]
then
echo " This script requires xmessage to be installed..."
exit 1
fi

if [[ ! $(which wget) ]] && [[ ! $(which curl) ]]
then
title="${0##*/} Error: Required App Not Found"
sendError 1 " wget or curl required to execute this script. "
fi

for x in "${apps[@]}" ; do
if [[ ! $(which $x) ]]
then
title="${0##*/} Error: Required App Not Found"
sendError 1 " $x is not installed. "
fi
done

for x in "${apps[@]}" ; do
if [[ $(pgrep -x $x) ]]
then
title="${0##*/} Alert:"
sendMessage " $x is already running. "
exit 1
fi
done
}

checkNetworkStatus()
{
ip addr | grep "state UP" > /dev/null 2>&1
if (($? != 0))
then
title="${0##*/} Error: Network Down"
sendError 1 " Your network has been detected as down."
fi

requiredHosts=("icanhazip.com")
for xi in "${requiredHosts[@]}"
do
ping -q -c2 "$xi" > /dev/null 2>&1
if (($? != 0))
then
title="${0##*/} Error: Ping"
sendError 1 " A required ping test of site: ${xi} has failed."
fi
done
}

obtainIP()
{
which wget > /dev/null

if (($? == 0))
then

cmd='wget'
args=('-4' '-qO-')
else
cmd='curl'
args=('-s' '-4')
fi

publicIP=$(${cmd} ${args[@]} icanhazip.com)
if (($? != 0))
then
title="${0##*/} Error: Outside IP Address"
sendError 1 " Unable to obtain your public IP address. "
fi
}

testAndExecuteApp()
{
let approved=0
for x in "${approvedIPs[@]}"
do
if [[ "${publicIP}" == "$x" ]]
then
((approved++))
fi
done

if ((approved >= 1))
then
title="${0##*/} Alert:"
sendMessage " Launching ${myApps}..."
for x in "${apps[@]}" ; do
$($x > /dev/null 2>&1 &)
done
exit 0
else
title="${0##*/} Request Denied:"
sendError 1 " ${myApps} are not approved to launch from ${publicIP}. "
fi
}

readIPTable()
{
IFS=$'\n'
approvedIPs=($(cat $HOME/.mlaunch))
testAndExecuteApp
}

writeIPTable()
{
echo "${publicIP}" >> $HOME/.mlaunch
if (($? != 0))
then
title="${0##*/} Write Error:"
sendError 1 " Configuration file could not be written."
else
readIPTable
fi
}

getUserInput()
{
while true
do
title="${0##*/} Authorization Request:"
buttons='Yes:2,No:1'
default="No"

sendPrompt \
"
There are no currently approved IP addresses
set to execute: ${myApps}...
Will you approve the current address: $publicIP

"
case $? in
[2] )
writeIPTable
;;
[1] )
exit 1
;;
esac
done
}

testFile()
{
if [[ -f "$HOME/.mlaunch" ]] && [[ -r "$HOME/.mlaunch" ]]
then
readIPTable
else
getUserInput
fi
}

setupDisplay
checkCommands
checkNetworkStatus
obtainIP
testFile

## end of mlaunch.sh ##

[MWBradburne]

> PLEASE IMPROVE SO WE ALL BENEFIT!

This script allows certain apps to only run while you're on VPN.

This program is adapted from Marek Novotny's "tbird.sh" script,
which, like all of Marek's scripts that I've posted, you should
get from him.

Some apps, like your bittorrent and newsreader clients, you might
only want to run when you're ON VPN (since they habitually give away
your IP address).

So you don't want to accidentally run these apps UNLESS you're
on VPN.

Also, you might want to ensure that any inadvertent configuration
changes made to the program while in use are destroyed when you
kill the program (so I modified Marek's script accordingly).

So Marek's tbird.sh script was adapted to ensure that pan started
every single time with the *same* setup, no matter what happens
to Pan settings in the interim.

(A similar script was modified for Firefox.)

#!/bin/bash
# Run this first when you are NOT on VPN so that it will ask you to
# create a $HOME/.pan file and it will ask to add the current IP
# address to that file. After that, the script will not start pan, &
# the script will only start pan when you are NOT on that IP address.
###############################################################
#
# script: pan.sh (adapted from Marek's tbird.sh)
# version: .07.03 beta
# purpose: launch pan only if NOT using your home IP
# date: Mon Jun 22 2015 02:56PM

# by: marek novotny
#
# revisions: added ping test
# : added results test for obtaining IP info
# : using only public IP addressing
# : added xmessage features
#

# notes: creates $HOME/.pan with approved
# : public IP for NOT launching pan.

# : more IPs can be added if needed.
# : first approved ip is stored in

# : $HOME/.pan and used to compare

# : to existing public IP for launch
#

# requirements: wget or curl, xmessage and pan
#
################################################################

sendMessage()
{
xmessage -display $DISPLAY -fg white -bg purple -title "${title}" -geom +60+30 -timeout 4 -buttons Okay:1 -default Okay "$1"
}

sendError()
{

xmessage -display $DISPLAY -fg black -bg orange -title "${title}" -geom +60+30 -buttons Okay:1 -default Okay "$1"

}

sendPrompt()
{
xmessage -display $DISPLAY -fg black -bg green -title "${title}" -geom +60+30 -buttons "$buttons" -default "$default" "$1"
}

setupDisplay()
{
if [[ -z "$DISPLAY" ]]
then
DISPLAY=':0'
fi
}

checkCommands()
{

if [[ ! $(type -p xmessage) ]]

then
echo " This script requires xmessage to be installed..."
exit 1
fi

if [[ ! $(type -p wget) ]] && [[ ! $(type -p curl) ]]

then
title="${0##*/} Error: Required App Not Found"

sendError " This script requires wget or curl which are not installed. "
exit 1
fi

if [[ ! $(type -p pan) ]]

then
title="${0##*/} Error: Required App Not Found"

sendError " Pan is not installed. "
exit 1
fi

# This ifthenelse requires the script NOT to have "pan" in the name (Pan is ok):
# if [[ $(pgrep pan) ]]
# This allows the letters "pan" to be in the file name (e.g., pan.sh):
if [[ $(pgrep -x pan) ]]

then
title="${0##*/} Alert:"

sendMessage " Pan is already running. "
exit 1
fi
}

checkNetworkStatus()
{
ip addr | grep "state UP" > /dev/null 2>&1
if (($? != 0))
then
title="${0##*/} Error: Network Down"

sendError " Your network has been detected as down."
exit 1

fi

requiredHosts=("icanhazip.com")
for xi in "${requiredHosts[@]}"
do
ping -q -c2 "$xi" > /dev/null 2>&1
if (($? != 0))
then
title="${0##*/} Error: Ping"

sendError " A required ping test of site: ${xi} has failed."
exit 1
fi
done
}

obtainIP()
{
type -p wget > /dev/null

if (($? == 0))
then

cmd='wget -4 -qO-'
else
cmd='curl -s -4'
fi

publicIP=$($cmd icanhazip.com)

if (($? != 0))
then
title="${0##*/} Error: Outside IP Address"

sendError " Unable to obtain your public IP address. "
exit 1

fi
}

testAndExecuteApp()
{
let approved=0
for x in "${approvedIPs[@]}"
do
if [[ "${publicIP}" == "$x" ]]
then
((approved++))
fi
done

# The copy commands below ensure pan always starts up the same!
# if ((approved >= 1))
if ((approved < 1))

then
title="${0##*/} Alert:"

sendMessage " Launching Pan"
cp $HOME/.pan2_golden/accels.txt $HOME/.pan2/.
cp $HOME/.pan2_golden/servers.xml $HOME/.pan2/.
cp $HOME/.pan2_golden/posting.xml $HOME/.pan2/.
cp $HOME/.pan2_golden/group-preferences.xml $HOME/.pan2/.
pan > /dev/null 2>&1 &

exit 0
else
title="${0##*/} Request Denied:"

sendError " Pan is not approved to launch from ${publicIP}. "
exit 1
fi
}

readIPTable()
{
IFS=$'\n'
approvedIPs=($(cat $HOME/.pan))
testAndExecuteApp
}

writeIPTable()
{
echo "${publicIP}" >> $HOME/.pan

if (($? != 0))
then
title="${0##*/} Write Error:"

sendError " Configuration file could not be written."
exit 1

else
readIPTable
fi
}

getUserInput()
{
while true
do
title="${0##*/} Authorization Request:"
buttons='Yes:2,No:1'
default="No"

sendPrompt "There are no currently approved IP addresses set to execute Pan

Will you approve the current address: $publicIP"

case $? in
[2] )
writeIPTable
;;
[1] )
exit 1
;;
esac
done
}

testFile()
{

if [[ -f "$HOME/.pan" ]] && [[ -r "$HOME/.pan" ]]

then
readIPTable
else
getUserInput
fi
}

setupDisplay
checkCommands
checkNetworkStatus
obtainIP
testFile

## end of pan.sh ##

[MWBradburne]

On Wed, 27 Jan 2016 07:18:25 +0000, MWBradburne wrote:

> PLEASE IMPROVE SO WE ALL BENEFIT!

This is a script that isn't really related to VPN but it
ensures that Firefox *always* starts up golden.

Any inadvertent configuration changes to Firefox are *always*
destroyed upon a new invocation of firefox.

For speed, and privacy, the firefox profile is placed into RAM
memory, but that's not really necessary. It's just nice since
I can't think of any drawbacks to doing it that way.

Also note that the duckduckgo.xml file can leak your operating
system, so you need to ensure only the modified XML file exists
in your golden profile.

#!/bin/bash
# runfirefox.sh will always start firefox in RAM memory using a golden profile
# To modify the golden profile, point Firefox to the pristine original:
# $ firefox -new-instance -profile $HOME/.golden/default_firefox_profile/
# Test which profile using:
# Help»Troubleshooting Information»Application Basics»Profile Directory»[Open Directory]
# Remember to create a "searchplugins/" dir in the golden profile & put the modified duckduckgo.xml there
rm -rf /TMP_RAM/firefox
cp -pr $HOME/.golden/default_firefox_profile/ /TMP_RAM/
# Open a new instance (not a new window!) in an existing firefox process
/usr/lib/firefox/firefox -new-instance -profile /TMP_RAM/default_firefox_profile
exit 0
## end of runfirefox.sh

[Andy Burns]

MWBradburne wrote:

> Notice that the ovpn files have a horrid very long file name:
> $ ls ./www.vpngate.net/common/*
> openvpn_download.aspx?sid=1453756386281&tcp=1&host=104.36.16.13&port=443&hid=3682177&%2Fvpngate_104.36.16.13_tcp_443.ovpn
> openvpn_download.aspx?sid=1453756386281&tcp=1&host=106.0.176.61&port=443&hid=460215&%2Fvpngate_106.0.176.61_tcp_443.ovpn
> openvpn_download.aspx?sid=1453756386281&tcp=1&host=119.76.36.35&port=1597&hid=2976869&%2Fvpngate_119.76.36.35_tcp_1597.ovpn
>
> 2. Delete unnecessary directories & unnecessary files:
> $ rm -rf ./www.vpngate.net/{api,images,ja,cn,common_images,en}
> $ rm -rf ./www.vpngate.net/{common_style.css,style_en.css,index.html,style_cn.css,style_ja.css}
> $ rm -rf ./www.vpngate.net/common/mail{1,2}.jpg

This is stupid!

There must be a way to download just the files you want
without needing to delete 100 megabytes of junk right
after downloading it!

[MWBradburne]

On Wed, 27 Jan 2016 19:53:15 +0000, Andy Burns wrote:

> There must be a way to download just the files you want
> without needing to delete 100 megabytes of junk right
> after downloading it!

On Wed, 27 Jan 2016 19:53:15 +0000, Andy Burns wrote:

> There must be a way to download just the files you want
> without needing to delete 100 megabytes of junk right
> after downloading it!

I tried, and failed to get *just* the ovpn files:
$ wget -A.ovpn -r -l 2 http://www.vpngate.net/en/
FINISHED --2016-01-27 12:01:52--
Total wall clock time: 17s
Downloaded: 4 files, 1.6M in 13s (127 KB/s)

That only got a few directories, but no *ovpn files in the
./www.vpngate.net/common/* directory.

I couldn't get the "common" files directly either:
$ wget -A.ovpn -r -l 2 http://www.vpngate.net/common
--2016-01-27 12:09:48-- http://www.vpngate.net/common
Resolving www.vpngate.net (www.vpngate.net)... 130.158.6.80, 130.158.6.97, 130.158.6.87, ...
Connecting to www.vpngate.net (www.vpngate.net)|130.158.6.80|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://www.vpngate.net/common/ [following]
--2016-01-27 12:09:49-- http://www.vpngate.net/common/
Reusing existing connection to www.vpngate.net:80.
HTTP request sent, awaiting response... 403 Forbidden
2016-01-27 12:09:50 ERROR 403: Forbidden.

I also tried to get both html,ovpn to see if that would work:
$ wget -A ovpn,html -r -l 2 http://www.vpngate.net/en/
FINISHED --2016-01-27 12:11:17--
Total wall clock time: 17s
Downloaded: 4 files, 1.6M in 12s (134 KB/s)

But that didn't download the "common" directory either, which is where
the *ovpn* files are stored.

So if you or anyone else knows *how* to get *just* the ovpn files in
the "common" directory, I'm all ears!

Alternatively, I can get the html file in the "en" directory and then
use awk/sed/grep/etc to create just the wget commands for the 300 ovpn
files in the common directory which are called by Anchored Hypertext
References:

[gamo]

El 27/01/16 a las 21:16, MWBradburne escribió:

Just a look of the wget man page says:

Recursive Accept/Reject Options
-A acclist --accept acclist
-R rejlist --reject rejlist
Specify comma-separated lists of file name suffixes or
patterns to
accept or reject. Note that if any of the wildcard
characters, *,
?, [ or ], appear in an element of acclist or rejlist, it
will be
treated as a pattern, rather than a suffix. In this case,
you have
to enclose the pattern into quotes to prevent your shell from
expanding it, like in -A "*.mp3" or -A '*.mp3'.


--
http://gamo.eu.pn/
The generation of random numbers is too important to be left to chance

[aitch]

In alt.os.linux, MWBradburne wrote:

[...]

> But that didn't download the "common" directory either, which is where
> the *ovpn* files are stored.

The *.ovpn files aren't necessarily stored there.

> So if you or anyone else knows *how* to get *just* the ovpn files in
> the "common" directory, I'm all ears!

The query strings complicate things slightly, but it should be possible.
I haven't tested this fully, but you may have some success with it:

wget -r -I /en,/common -A do_openvpn.aspx,openvpn_download.aspx\* \
-nc -nd http://www.vpngate.net/en/

You should be left with just the *.ovpn files in the current directory.
The filenames will still need trimming though:

for f in *.ovpn ; do mv "$f" "${f##*%2F}" ; done

You could also try wget's --content-disposition option, although the
documentation for my (fairly old) version describes it as a buggy,
experimental feature. You'd probably have to change the "-A ..." above
to "-A do_openvpn.aspx,openvpn_download.aspx,.ovpn" if you do try it.

--
aitch

[Cybe R. Wizard]

On Thu, 28 Jan 2016 19:55:31 -0000 (UTC)
aitch <f...@bar.invalid> wrote:

> aitch

I've seen that sig before. Did you once sign:
-=H=-?

Cybe R. Wizard
--
Registered GNU/Linux user # 126326
Registered Ubuntu User (deprecated) # 2136

[MWBradburne]

Wow. That's quite an improvement!

I'm going to have to study your wget options because it did EXACTLY what
we needed, which was to just download the ovpn files and nothing else!

Here's what I used just now:
$ wget -b -r -I /en,/common -A do_openvpn.aspx,openvpn_download.aspx\* -nc -nd http://www.vpngate.net/en/ --user-agent "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0"
FINISHED --2016-01-28 15:03:35--
Total wall clock time: 20m 19s
Downloaded: 424 files, 3.9M in 12m 20s (5.35 KB/s)

I added the "-b" to put the command into the background, and, I added
the "--user-agent" in an attempt to fool the vpngate sysadmin that it's
not wget getting the files, but a browser (just in case they don't want
us getting their config files en masse).

This resulted in 424 file files who names were of the format:
openvpn_download.aspx?sid=1454017403774&udp=1&host=201.68.99.108&port=1775&hid=3497502&%2Fvpngate_201.68.99.108_udp_1775.ovpn
openvpn_download.aspx?sid=1454017403774&udp=1&host=203.205.106.14&port=1265&hid=2571073&%2Fvpngate_203.205.106.14_udp_1265.ovpn
openvpn_download.aspx?sid=1454017403774&udp=1&host=210.121.175.192&port=1195&hid=1085449&%2Fvpngate_210.121.175.192_udp_1195.ovpn

Shortening the long file names was easy with your second suggestion:
$ for f in *.ovpn ; do mv "$f" "${f##*%2F}" ; done

Which resulted in file names of the same format had we downloaded them manually:
vpngate_201.68.99.108_udp_1775.ovpn
vpngate_203.205.106.14_udp_1265.ovpn
vpngate_210.121.175.192_udp_1195.ovpn

Adding the three lines to eliminate DNS leaks was also easy:
$ for filename in *.ovpn; do echo -e "script-security 2\nup /etc/openvpn/update-resolv-conf\ndown /etc/openvpn/update-resolv-conf" >> $filename; done

I also geotagged the files based on the IP address, but that's optional.

Using that purposefully free VPN was as easy running "sudo openvpn --config filename.ovpn":

for x in $(ls *.ovpn) ; do
sudo openvpn --config "$x"

done

[MWBradburne]

On Wed, 27 Jan 2016 07:18:25 +0000, MWBradburne wrote:

Shorter summary [Thanks to many on alt.os.linux!]
(Improvements to help everyone use free VPN are always welcome!)

1. Obtain ovpn files: (solution mainly courtesy of J.O. Aho & aitch)

$ wget -b -r -I /en,/common -A do_openvpn.aspx,openvpn_download.aspx\* -nc -nd http://www.vpngate.net/en/ --user-agent "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0"

2. Optional: Add DNS-leak fix: (solution courtesy of JG Miller, et. al.)

$ for filename in *.ovpn; do echo -e "script-security 2\nup /etc/openvpn/update-resolv-conf\ndown /etc/openvpn/update-resolv-conf" >> $filename; done

3. Optional: Convert long names to shorter names: (solution mainly courtesy of marrgol, Marek Novotny, & William Unruh)
$ for i in *.ovpn;do mv $i $(echo $i|cut -d'F' -f2);done
Results in two naming styles, one using DNS and one without DNS:
vpngate_vpn961834324.opengw_udp_1986.ovpn
vpngate_94.78.83.40_udp_1986.ovpn

4. Optional: Geotag based on the IP address: (solution courtesy of Eef Hartman, et. al.)
$ ping -c1 vpn961834324.opengw.net
94.78.83.40
$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 94.78.83.40
GeoIP City Edition, Rev 1: US, CO, Colorado, Boulder, 80302, 40.048100, -105.384201, 751, 303

5. Start VPN: (solution courtesy of Marek Novotny)
$ for i in $(ls *.ovpn) ; do
sudo openvpn --config "$i"
kickOrKeep
done

Where kickOrKeep is defined as:

kickOrKeep () {
echo ""
read -p "Was that successful? " answer
case $answer in
[Nn]* )
mv -i "$x" $BAD_DIR
;;
[Yy]* )
;;
* )
kickOrKeep
;;
esac
}

5. Optional: Start VPN and add the DNS-leak fix at that time:
$ for i in $(ls *.ovpn) ; do
sudo openvpn --config "$i" --script-security 2 --up /etc/openvpn/update-resolv-conf --down /etc/openvpn/update-resolv-conf
kickOrKeep
done

[aitch]

Cybe R. Wizard wrote:

> I've seen that sig before. Did you once sign:
> -=H=-?

No, that's not me. There must be another aitch around.

--
aitch

[MWBradburne]

On Fri, 29 Jan 2016 07:04:32 +0000, MWBradburne wrote:

> 1. Obtain ovpn files: (solution mainly courtesy of J.O. Aho & aitch)
> $ wget -b -r -I /en,/common -A do_openvpn.aspx,openvpn_download.aspx\* -nc -nd http://www.vpngate.net/en/ --user-agent "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0"

I've slightly improved the script using a trick from Marek's changemac.sh
script to randomize the user-agent string so that the web site doesn't
get mad at us for using wget to download its openvpn config files.

I've also added a random wait time of 1 minute * (1/2 that to 1-1/2 that)
for each download, again, so as to not appear to be an automated wget script.

And I removed the -b (background) command because we need the script to
wait for the wget to finish before renaming the ovpn files from long names
to shorter names.

Here's the resultant simple script (improvements always welcome) for all to use.

#!/bin/bash
# getovpn.sh (downloads and renames about 300 openvpn config files per run)
# Set up a list of user agent strings (truncated here for simplicity)
# http://useragentstring.com/pages/Firefox/
UserAgentArray=(
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" # Firefox 43.0
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" # Firefox 40.1
"Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0" # Firefox 36.0
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10; rv:33.0) Gecko/20100101 Firefox/33.0" # Firefox 33.0
"Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0" # Firefox 31.0
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20130401 Firefox/31.0" # Firefox 31.0
"Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0" # Firefox 31.0
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20120101 Firefox/29.0" # Firefox 29.0
"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/29.0" # Firefox 29.0
"Mozilla/5.0 (X11; OpenBSD amd64; rv:28.0) Gecko/20100101 Firefox/28.0" # Firefox 28.0
)
RANGE=$((${#UserAgentArray[@]} + 1))
i=$RANDOM
let "i %= $RANGE"
USERAGENT=${UserAgentArray[$i]}
wget --wait=1m --random-wait -r -I /en,/common -A do_openvpn.aspx,openvpn_download.aspx\* -nc -nd http://www.vpngate.net/en/ --user-agent "$USERAGENT"

for i in *.ovpn;do mv $i $(echo $i|cut -d'F' -f2);done

## END ##

[William Unruh]

On 2016-02-01, MWBradburne <MWBra...@cisco.com> wrote:
> On Fri, 29 Jan 2016 07:04:32 +0000, MWBradburne wrote:
>
>> 1. Obtain ovpn files: (solution mainly courtesy of J.O. Aho & aitch)
>> $ wget -b -r -I /en,/common -A do_openvpn.aspx,openvpn_download.aspx\* -nc -nd http://www.vpngate.net/en/ --user-agent "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0"
>
> I've slightly improved the script using a trick from Marek's changemac.sh
> script to randomize the user-agent string so that the web site doesn't
> get mad at us for using wget to download its openvpn config files.

?? Why would the web site get mad at you for using wget to download its
openvpn config files? Why would you have to do it more than once?

[MWBradburne]

On Mon, 01 Feb 2016 08:10:11 +0000, William Unruh wrote:

> ?? Why would the web site get mad at you for using wget to download its
> openvpn config files? Why would you have to do it more than once?

In my experience with *manual* downloads, the free openvpn config files
get stale, in that they might work today, but not tomorrow.

So, with *manual* downloads, you have to pretty much download once a
day or once every few days, although the vpn-checking script that
Marek wrote helps because it has a kickorkeep section which tosses
the bad files.

My brand-new experience with automatic downloads was different in one
respect in that 300 files were available, instead of about 25 files.

So, with 300 files to chomp through with Marek's kickOrKeep script,
there should be far less of a need for daily downloads.