Script inspired by Marek for changing the name of vpn files

[VPN user]

Here is script, while inspired by Marek, isn't elegant as his are.

But, it works and has been tested against over 100 openvpn config
files from vpngate.

I post the script so that you might make use of it (or improve it).

The script just changes the file name based on IP address, geolocations,
server name, protocol, port, and date (but not speed).

FROM something like:
vpngate_vpn195786088.opengw.net_udp_1195.ovpn
vpngate_vpn805768177.opengw.net_tcp_1841.ovpn
vpngate_114.159.54.159_udp_1942.ovpn
vpngate_78.188.128.102_tcp_16671.ovpn

TO something like:
vpngate_JP_40_Tokyo_Hachioji_119.242.216.189-vpn195786088.opengw.net_udp1195_20160302.ovpn
vpngate_PL_77_Malopolskie_Kraków_83.27.207.198-vpn805768177.opengw.net_tcp1841_20160302.ovpn
vpngate_JP_na_na_na_114.159.54.159-114.159.54.159_udp1942_20160302.ovpn
vpngate_TR_34_Istanbul_Istanbul_78.188.128.102-78.188.128.102_tcp16671_20160302.ovpn

Based on a variety of geolocation lookups of the form:
GeoIP City Edition, Rev 1: can't resolve hostname ( None )
GeoIP City Edition, Rev 1: DE, 15, Thuringen, Tabarz, 99891, 50.883301, 10.516700, 0, 0
GeoIP City Edition, Rev 1: GB, N/A, N/A, N/A, N/A, 51.500000, -0.130000, 0, 0
GeoIP City Edition, Rev 1: HK, 00, N/A, Central District, N/A, 22.283300, 114.150002, 0, 0
GeoIP City Edition, Rev 1: JP, 34, Saitama, Niiza, 352-0006, 35.797901, 139.525406, 0, 0
GeoIP City Edition, Rev 1: JP, 40, Tokyo, Hachioji, 193-0931, 35.655800, 139.323898, 0, 0
GeoIP City Edition, Rev 1: JP, 40, Tokyo, Tokyo, 100-0001, 35.685001, 139.751404, 0, 0
GeoIP City Edition, Rev 1: JP, N/A, N/A, N/A, N/A, 35.689999, 139.690002, 0, 0
GeoIP City Edition, Rev 1: KR, 11, Seoul-t'ukpyolsi, Seoul, N/A, 37.598499, 126.978302, 0, 0
GeoIP City Edition, Rev 1: KR, 13, Kyonggi-do, Suwon, N/A, 37.291100, 127.008904, 0, 0
GeoIP City Edition, Rev 1: KR, N/A, N/A, N/A, N/A, 37.570000, 126.980003, 0, 0
GeoIP City Edition, Rev 1: TR, 34, Istanbul, Istanbul, 34437, 41.037102, 28.986601, 0, 0
GeoIP City Edition, Rev 1: TR, 71, Konya, Konya, 42100, 37.927299, 32.419899, 0, 0
GeoIP City Edition, Rev 1: US, CO, Colorado, Colorado Springs, 80918, 38.914200, -104.774902, 752, 719
GeoIP City Edition, Rev 1: US, MA, Massachusetts, Haverhill, 01835, 42.752998, -71.086098, 506, 978
GeoIP City Edition, Rev 1: US, NH, New Hampshire, Canterbury, 03224, 43.349201, -71.548103, 506, 603
GeoIP City Edition, Rev 1: US, NY, New York, Buffalo, 14221, 42.986401, -78.727898, 514, 716
GeoIP City Edition, Rev 1: US, WI, Wisconsin, Milwaukee, 53202, 43.043400, -87.894501, 617, 414
GeoIP City Edition, Rev 1: VE, N/A, N/A, N/A, N/A, 8.000000, -66.000000, 0, 0

Work in progress will be to add the speedtest, but I am having great
troubles with Marek's too-elegant-for-me-to-understand "sed" method
(mostly my problem is changing Marek's syntax to suit "my" needs)
as shown below:

THIS IS MAREK'S SYNTAX BUT I AM HAVING TROUBLE CHANGING IT:
set -- $(geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $IP)
country="${2/*: /}"
state_abv="${3/' N/A'/na}"
state_long="${4/' N/A'/na}"
city="${5/' N/A'/na}"
zip="${6/' N/A'/na}"

Given I don't "understand" how to change the above syntax, I'll stick
with the "sed,awk,cut" that I've been using to add the speedtest results.

But, until I add the speedtest results, here's the renaming script using
just the country, state, city, IP, servername, protocol, port, & date:

Improvements are always welcome (but I have to be able to *understand*
them sufficient to leverage those improvements further!).

#!/bin/bash
# vpnrename.sh (renames current VPN ovpn file mostly based on geolocation)
# The log below is a 1-line file created only by running "vpntest" where
# vpntest is a modified Marek script that tests a config file by running openvpn.
# When vpntest starts VPN, it puts the full fspec of the ovpn file into the log.
# ---
echo "Getting current ovpn file & date..."
VPN_DATE=$(date +%Y%m%d)
VPN_LOG=/tmp/openvpn.log
VPN_FSPEC=$(cat $VPN_LOG)
VPN_DIR=$(echo $VPN_FSPEC|sed 's%/[^/]*$%/%')
VPN_FILE=$(echo $VPN_FSPEC|sed -e 's/.ovpn$//')
VPN_FNAME=$(echo $VPN_FSPEC|sed -e 's!.*/!!')
VPN_PREFIX=$(echo $VPN_FILE|sed -e 's!.*/!!'|awk -F_ '{print $1}')
# ---
echo "Getting server, protocol, & port..."
SERVER=$(grep ^remote `cat $VPN_LOG`|sed -e 's/
//g'|awk '{print $2}'|sed -e 's/ /-/g')
PROTOCOL=$(grep ^proto `cat $VPN_LOG`|sed -e 's/
//g'|awk -F" " '{print $2}'|sed -e 's/ /-/g')
PORT=$(grep ^remote `cat $VPN_LOG`|sed -e 's/
//g'|awk '{print $3}'|sed -e 's/ /-/g')
# ---
echo "Getting current IP address..."
IP=$(inxi -i |grep IP|awk '{print $5}')
# ---
echo "Getting geolocation information..."
GEO_STRING=$(geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $IP)
GEO_COUNTRY=$(echo $GEO_STRING|awk -F, '{print $2}'|sed -e 's/Rev 1: //' -e 's/ //g'|cut -c 1-2)
GEO_STATE=$(echo $GEO_STRING|awk -F, '{print $3}'|sed -e 's/ //g'|sed -e 's/N\/A/na/')
GEO_LONGSTATE=$(echo $GEO_STRING|awk -F, '{print $4}'|sed -e 's/ //g'|sed -e 's/N\/A/na/'|tr "'" "-")
GEO_CITY=$(echo $GEO_STRING|awk -F, '{print $5}'|sed -e 's/ //g'|sed -e 's/N\/A/na/')
# --
echo "Renaming ovpn file..."
mkdir -p ${VPN_DIR}/renamed
mv $VPN_FSPEC ${VPN_DIR}renamed/${VPN_PREFIX}_${GEO_COUNTRY}_${GEO_STATE}_${GEO_LONGSTATE}_${GEO_CITY}_${IP}-${SERVER}_${PROTOCOL}${PORT}_${VPN_DATE}.ovpn
echo "mv $VPN_FSPEC ${VPN_DIR}renamed/${VPN_PREFIX}_${GEO_COUNTRY}_${GEO_STATE}_${GEO_LONGSTATE}_${GEO_CITY}_${IP}-${SERVER}_${PROTOCOL}${PORT}_${VPN_DATE}.ovpn"
exit 0
## End ##

--- news://freenews.netfront.net/ - complaints: ne...@netfront.net ---

[VPN user]

On Wed, 02 Mar 2016 23:17:56 +0000, VPN user wrote:

> echo "Getting current IP address..."
> IP=$(inxi -i |grep IP|awk '{print $5}')

Sometimes, for some reason, the IP address can't be obtained
this way, yet, a "route -n" shows the IP address.

So, in those cases, I obtain the IP address simply from:

#!/bin/bash
# vpnroute.sh outputs the current VPN IP address without using curl, wget, or inxi
VPN_ROUTE=$(route -n|grep -v ^0.0.0.0|grep -v ^10.211|grep -v ^192.168.1.0|grep -v ^128.0.0.0|grep -v ^Kernel|grep -v ^Destination|awk '{print $1}')
echo $VPN_ROUTE
exit 0
## end ##

[VPN user]

On Thu, 03 Mar 2016 12:11:38 +0000, VPN user wrote:

> So, in those cases, I obtain the IP address simply from:

> VPN_ROUTE=$(route -n|grep... blah blah blah

Here is Marek's "kickorkeep" shell script modified only in 5 ways:
1. His vpn_winners & losers directories are variables.
2. The IP address is obtained from the "route -n" command.
3. The country, state, & city are geolocated from that IP address.
4. The server name, protocol, & port are taken from the ovpn file contents.
5. All this information is used to *rename* the file as it's being moved.

The only bug I know of is that I can't get rid of the error messages
when the geolocate command runs the five times necessary in the loop
to test whether the VPN connection has been established.

I tried the ">&/dev/null" trick but it didn't work for me because
it suppressed the output ok but it also suppressed setting the variable.

If you know how to suppress the multiple errors (as the geoip lookup occurs
about 5 times inside one of the loops) while still being able to set the
variable, that would be a great improvement.

Here's the modified Marek file:

#!/bin/bash
# Modified to get the server, protocol, & port from the ovpn file contents
# And to get the IP address from the "oute -n" command.
# And then to get the country, state, and city from that IP address.
# And then to modify the vpn_winners file name with the new information.
############################################################
#
# script: kickorkeep
# version: .01
# date: 2016-02-22
# written by: marek novotny
# git: https://github.com/marek-novotny/vpntools
# license: GPL v2 (only)
# dependencies: wget, openvpn, ip
# purpose: test ovpn file and keep it or
# : kick it or keep it
# usage: execute script from within
# : a directory full of ovpn files
#
############################################################
DATE=$(date +%Y%m%d)
VPN_WINNERS=$HOME/doc/cert/vpn_winners/
VPN_LOSERS=$HOME/doc/cert/vpn_losers/
VPN_PREFIX=vpngate

ovpnTest () {

let count=6
echo
echo "$(basename $0) message: testing $x"
echo
echo "$(basename $0) message: obtaining device id... "
devID=$(ip route get 8.8.8.8 | awk '{print $5}')
echo "$(basename $0) message: device id set: ${devID}..."
vpnID=$devID
printf "%s" "$(basename $0) message: obtaining vpn connection..."
$priv openvpn --config $x &> /dev/null &
while [[ $vpnID == $devID ]] ; do
sleep 5
((count--))
printf "%s" "${count}."
vpnID=$(ip route get 8.8.8.8 | awk '{print $5}')
######## ######## ######## ######## ######## ######## ########
# Begin modifications:
# Get the server, protocol, & port from the ovpn file contents
# Be careful when copying because the caret-M has to be newline!
SERVER=$(grep ^remote $x|sed -e 's/ //g'|awk '{print $2}'|sed -e 's/ /-/g')
PROTOCOL=$(grep ^proto $x|sed -e 's/ //g'|awk -F" " '{print $2}'|sed -e 's/ /-/g')
PORT=$(grep ^remote $x|sed -e 's/ //g'|awk '{print $3}'|sed -e 's/ /-/g')
# Get the IP address from the route command:
IP=$(route -n|grep -v ^"0.0.0.0"|grep -v ^"10.211.1."|grep -v ^"192.168.1.0"|grep -v ^"128.0.0.0"|grep -v ^Kernel|grep -v ^Destination|awk '{print $1}')
# Get the geolocation iso-3166, country, state, city from the IP address
# WIP I need to suppress the error that prints in the loop when the IP address isn't set yet.
# GEO_STRING=$(geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $IP)>&/dev/null

GEO_STRING=$(geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $IP)
GEO_COUNTRY=$(echo $GEO_STRING|awk -F, '{print $2}'|sed -e 's/Rev 1: //' -e 's/ //g'|cut -c 1-2)
GEO_STATE=$(echo $GEO_STRING|awk -F, '{print $3}'|sed -e 's/ //g'|sed -e 's/N\/A/na/')
GEO_LONGSTATE=$(echo $GEO_STRING|awk -F, '{print $4}'|sed -e 's/ //g'|sed -e 's/N\/A/na/'|tr "'" "-")
GEO_CITY=$(echo $GEO_STRING|awk -F, '{print $5}'|sed -e 's/ //g'|sed -e 's/N\/A/na/')

# End modifications:
######## ######## ######## ######## ######## ######## ######## ########
if [ $count -le 1 ] ; then
$priv pkill openvpn
printf "\n%s\n" "$(basename $0) error: $(basename $x) hung..."
echo "$(basename $0) message: putting this file in the loser bin $VPN_LOSERS..."
return 1
fi
done
printf "\n%s\n" "$(basename $0) message: obtained vpn: ${vpnID}..."
echo "$(basename $0) message: successful connect..."
echo "$(basename $0) message: putting this file in the winner bin... $VPN_WINNERS"
$priv pkill openvpn
sleep 3
return 0
}

if [ $# -ne 0 ] ; then
echo "$(basename $0) error: just run this without arguments)"
echo "from within a directory full of .ovpn config files..."
exit 1
fi

if [ $(id -u) -ne 0 ] ; then
priv=sudo
$priv -v
else
priv=""
fi

if [ ! -d $VPN_WINNERS -o ! -d $VPN_LOSERS ] ; then
mkdir -p $VPN_WINNERS/
mkdir -p $VPN_LOSERS/
fi

IFS=$'\n'
array=( $(find . -maxdepth 1 -type f -name "*.ovpn") )
if [ "${#array[@]}" -ge 1 ] ; then
for x in "${array[@]}" ; do
ovpnTest
if [ $? -eq 0 ] ; then
# mv $x $VPN_WINNERS/.
######## ######## ######## ######## ######## ######## ######## ########
# Begin modifications:
# Move the vpn config file to a renamed name in the winners directory
echo "mv $x ${VPN_WINNERS}${VPN_PREFIX}_${GEO_COUNTRY}_${GEO_STATE}_${GEO_LONGSTATE}_${GEO_CITY}_${IP}-${SERVER}_${PROTOCOL}${PORT}_${DATE}.ovpn"
mv $x ${VPN_WINNERS}${VPN_PREFIX}_${GEO_COUNTRY}_${GEO_STATE}_${GEO_LONGSTATE}_${GEO_CITY}_${IP}-${SERVER}_${PROTOCOL}${PORT}_${DATE}.ovpn
# End modifications:
######## ######## ######## ######## ######## ######## ######## ########
else
mv $x $VPN_LOSERS/.
fi
done
else
echo "$(basename $0) message: no ovpn files found here..."
echo "$(basename $0) message: exiting..."
fi

## END ##
#
# --
# Marek Novotny
# https://github.com/marek-novotny
#
# I wrote another kickorkeep script that should be more fully automated.
# https://github.com/marek-novotny/vpntools
#
# The way this works is you run the script while sitting in a directory
# full of .ovpn files you've downloaded. It will automatically test them
# all and separate the good from the bad.
#
# The winners will go in $HOME/kickorkeep/winners
# The losers will go in $HOME/kickorkeep/losers
#
# You can modify that easily enough.
# Now, once you have a directory full of known good, you can run vpnit on
# that whole directory with a simple loop...
#
# Go into the winners directory and execute a simple for x loop
#
# for x in $(ls) ; do
# vpnit "$x"
# done
#
# You can type that right in the command line. It will then test every
# good ovpn file and rename them all accordingly. Just let it run...
#
# Here is the kickorkeep

[VPN user]

On Fri, 04 Mar 2016 01:21:34 +0000, VPN user wrote:

> Here is Marek's "kickorkeep" shell script modified only in 5 ways:
> 1. His vpn_winners & losers directories are variables.
> 2. The IP address is obtained from the "route -n" command.
> 3. The country, state, & city are geolocated from that IP address.
> 4. The server name, protocol, & port are taken from the ovpn file contents.
> 5. All this information is used to *rename* the file as it's being moved.

Here are five files that this script renamed as it moved the good files
to the vpn winners directory:

ORIGINAL FILE NAMES:
a. vpngate_118.7.210.56_udp_1719.ovpn
b. vpngate_vpn126951274.opengw.net_tcp_1745.ovpn
c. vpngate_71.123.228.241_udp_1751.ovpn
d. vpngate_vpn921944171.opengw.net_tcp_1195.ovpn
e. vpngate_vpn172836300.opengw.net_tcp_1765.ovpn

VPN WINNERS FILE NAMES:
a. vpngate_JP_40_Tokyo_Tokyo_118.7.210.56-118.7.210.56_udp1719_20160303.ovpn
b. vpngate_KR_na_na_na_125.178.127.203-vpn126951274.opengw.net_tcp1745_20160303.ovpn
c. vpngate_US_TX_Texas_Rowlett_71.123.228.241-71.123.228.241_udp1751_20160303.ovpn
d. vpngate_VE_na_na_na_190.207.32.173-vpn921944171.opengw.net_tcp1195_20160303.ovpn
e. vpngate_VN_44_HaNoi_Hanoi_1.54.140.200-vpn172836300.opengw.net_tcp1765_20160303.ovpn

Where the final file name is the combination of:
prefix,country,short state,long state,city,ip,servername,protocol,port,date,extension

I could put the speedtests in there, but that would take too long
since this is a batch job that just moves files that it simply
tests for a moment to see if they're good (vpn winners) or bad
(vpn losers).

So these are all the relevant "quick" tests I can make which turn
a file name into something useful.

Thanks, as always, go to Marek who is so wonderful with his generous code
offerings that I just felt I had to give back to the team in any way
that I could.

[Marek Novotny]

On 2016-03-04, VPN user <vpn...@example.com> wrote:
> On Thu, 03 Mar 2016 12:11:38 +0000, VPN user wrote:
>
>> So, in those cases, I obtain the IP address simply from:
>> VPN_ROUTE=$(route -n|grep... blah blah blah
>
> Here is Marek's "kickorkeep" shell script modified only in 5 ways:
> 1. His vpn_winners & losers directories are variables.
> 2. The IP address is obtained from the "route -n" command.
> 3. The country, state, & city are geolocated from that IP address.
> 4. The server name, protocol, & port are taken from the ovpn file contents.
> 5. All this information is used to *rename* the file as it's being moved.
>
> The only bug I know of is that I can't get rid of the error messages
> when the geolocate command runs the five times necessary in the loop
> to test whether the VPN connection has been established.
>
> I tried the ">&/dev/null" trick but it didn't work for me because
> it suppressed the output ok but it also suppressed setting the variable.
>
> If you know how to suppress the multiple errors (as the geoip lookup occurs
> about 5 times inside one of the loops) while still being able to set the
> variable, that would be a great improvement.
>
> Here's the modified Marek file:

// snip

Please change the written by from my name to your own name. You can
credit me with a link to the original or simply my name.

thanks,

--
Marek Novotny
https://github.com/marek-novotny

[VPN user]

On Thu, 03 Mar 2016 17:43:37 -0800, Marek Novotny wrote:

> Please change the written by from my name to your own name. You can
> credit me with a link to the original or simply my name.

I knew you'd say that.
But, in "my" directories, I don't use the same conventions *you*
use, so I keep all your original comments verbatim.

But I do agree with you that I need to do *something* to indicate
that it's a "bastardization" of "your" file, so you shouldn't be
penalized for my lousing coding lack of capabilities!

Thanks for the script.

The only 'observation' I have is that the way you originally obtained
the geolocation (and speedtest), while seemingly simple, used a "sed"
mechanism that I just couldn't manage to get around since I needed
pipes to awk and cut.

The only question I have is how to make the script NOT output
the geolocate errors constantly when it loops before connecting
to VPN.

I tried the ">&/dev/null" but that also suppressed setting the
variable.

# WIP I need to suppress the error that prints in the loop when the IP address isn't set yet.
# GEO_STRING=$(geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $IP)>&/dev/null
GEO_STRING=$(geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $IP)

Is there a way to suppress the error?

[Marek Novotny]

On 2016-03-04, VPN user <vpn...@example.com> wrote:

> On Thu, 03 Mar 2016 17:43:37 -0800, Marek Novotny wrote:
>
>> Please change the written by from my name to your own name. You can
>> credit me with a link to the original or simply my name.
>
> I knew you'd say that.

If I write it, I sign it. If anyone else writes it, modifies, et al,
then I'm no longer the author. Someone else is.

> But, in "my" directories, I don't use the same conventions *you*
> use, so I keep all your original comments verbatim.
>
> But I do agree with you that I need to do *something* to indicate
> that it's a "bastardization" of "your" file, so you shouldn't be
> penalized for my lousing coding lack of capabilities!
>
> Thanks for the script.

Welcome.

> The only 'observation' I have is that the way you originally obtained
> the geolocation (and speedtest), while seemingly simple, used a "sed"
> mechanism that I just couldn't manage to get around since I needed
> pipes to awk and cut.

I never used sed in either geolocation or speed tests. You might mean I
used set to set positional parameters and parameter expansion to modify
the stdout of the parameter expansion.

> The only question I have is how to make the script NOT output
> the geolocate errors constantly when it loops before connecting
> to VPN.

1> and > redirect the stdout
2> redirects the stderr
&> is both stdout and stderr

But as you observed, you want stdout but want to redirect stderr, so
you'd use 2> /dev/null

> I tried the ">&/dev/null" but that also suppressed setting the
> variable.
>
> # WIP I need to suppress the error that prints in the loop when the IP address isn't set yet.
> # GEO_STRING=$(geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $IP)>&/dev/null
> GEO_STRING=$(geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $IP)
>
> Is there a way to suppress the error?

[VPN user]

On Thu, 03 Mar 2016 18:45:22 -0800, Marek Novotny wrote:

> If I write it, I sign it. If anyone else writes it, modifies,
> et al, then I'm no longer the author. Someone else is.

I understand.
I'm just trying to give back to the team in a way that I can.

Here's a followup set of tiny scripts that save time once
the modified kickorkeep script has filled a vpn_winners
directory full of hundreds of winners who now have the
country code as part of the file name:

FROM:
vpn_winners/vpngate_VE_23_Zulia_Cabimas_190.142.249.20-vpn794876020.opengw.net_udp1195_20160303.ovpn
vpn_winners/vpngate_US_NY_NewYork_Buffalo_192.3.226.12-vpn175001405.opengw.net_udp1194_20160303.ovpn
vpn_winners/vpngate_TH_na_na_na_122.155.168.29-122.155.168.29_udp1194_20160303.ovpn
etc.

TO:
vpn_winners/VE/(that file)
vpn_winners/US/(that file)
vpn_winners/TH/(that file)
etc.

#!/bin/bash
# vpnmove.sh moves kickorkeep-renamed VPN files into iso3166 folders
# AE BR CA CN DE DO FR HK IP JP KR NL PL QA RO RU SA SE TH TR US VE VN
# Note this automatically creates "IP" for the unknown countries
for FNAME in $(ls *.ovpn) ; do
COUNTRY=$(echo $FNAME|awk -F_ '{print $2}')
mkdir -p $COUNTRY
echo "Moving $FNAME into $COUNTRY subfolder"
mv $FNAME $COUNTRY
done
exit 0

This script will simply look up the country codes based on a text file:

#!/bin/bash
# 3166.sh will grep a previously saved iso3166 text file from:
# Use: 3166.sh AE
# https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2
COUNTRY_FILE=/usr/local/bin/iso3166.txt
egrep -i ^${1} ${COUNTRY_FILE}

Thanks again to Marek and others here for doing the hard lifting in downloading
and testing and renaming vpngate openvpn configuration files, so that it's
now much easier and more efficient for an average user (like me) to use VPN.

[VPN user]

On Thu, 03 Mar 2016 18:45:22 -0800, Marek Novotny wrote:

> I never used sed in either geolocation or speed tests. You might mean I
> used set to set positional parameters and parameter expansion to modify
> the stdout of the parameter expansion.

I called it "sed" in quotes but I should have said regular-expressions
or "sed-like" or something like that to indicate what you used.

Here is exactly what you used in your wonderful "vpnit" script:
speedData () {

echo "$(basename $0) message: collecting speed test data..."
IFS=$'\n'
set -- $(speedtest-cli --simple --secure --timeout 5)
pingResult="${1/*: /}"
downResult="${2/*: /}"
upResult="${3/*: /}"
IFS=$'\t\n '
}

The problem "I" had was those "sed-like" regular expression braces (or
whatever they're called) worked fine the way you wrote them, but every
time I tried to add more regular expressions or a pipe to awk or cut,
I couldn't figure out the syntax.

So, I changed what you did above, to the following:

GEO_STRING=$(geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $IP)
GEO_COUNTRY=$(echo $GEO_STRING|awk -F, '{print $2}'|sed -e 's/Rev 1: //' -e 's/ //g'|cut -c 1-2)
GEO_STATE=$(echo $GEO_STRING|awk -F, '{print $3}'|sed -e 's/ //g'|sed -e 's/N\/A/na/')
GEO_LONGSTATE=$(echo $GEO_STRING|awk -F, '{print $4}'|sed -e 's/ //g'|sed -e 's/N\/A/na/'|tr "'" "-")
GEO_CITY=$(echo $GEO_STRING|awk -F, '{print $5}'|sed -e 's/ //g'|sed -e 's/N\/A/na/')

Given all the vagaries of the geoip string such as those shown below:
a. Handling errors:
GeoIP City Edition, Rev 1: IP Address not found

GeoIP City Edition, Rev 1: can't resolve hostname ( None )

b. Handling slashes:

GeoIP City Edition, Rev 1: GB, N/A, N/A, N/A, N/A, 51.500000, -0.130000, 0, 0
GeoIP City Edition, Rev 1: HK, 00, N/A, Central District, N/A, 22.283300, 114.150002, 0, 0

GeoIP City Edition, Rev 1: JP, N/A, N/A, N/A, N/A, 35.689999, 139.690002, 0, 0

c. Handling dashes:

GeoIP City Edition, Rev 1: KR, 13, Kyonggi-do, Suwon, N/A, 37.291100, 127.008904, 0, 0

d. Handling quotes:

GeoIP City Edition, Rev 1: KR, 11, Seoul-t'ukpyolsi, Seoul, N/A, 37.598499, 126.978302, 0, 0

e. Handling spaces:

GeoIP City Edition, Rev 1: US, CO, Colorado, Colorado Springs, 80918, 38.914200, -104.774902, 752, 719

GeoIP City Edition, Rev 1: US, NH, New Hampshire, Canterbury, 03224, 43.349201, -71.548103, 506, 603
GeoIP City Edition, Rev 1: US, NY, New York, Buffalo, 14221, 42.986401, -78.727898, 514, 716

f. And handling the easy ones:

GeoIP City Edition, Rev 1: DE, 15, Thuringen, Tabarz, 99891, 50.883301, 10.516700, 0, 0

GeoIP City Edition, Rev 1: JP, 34, Saitama, Niiza, 352-0006, 35.797901, 139.525406, 0, 0
GeoIP City Edition, Rev 1: JP, 40, Tokyo, Hachioji, 193-0931, 35.655800, 139.323898, 0, 0

GeoIP City Edition, Rev 1: TR, 34, Istanbul, Istanbul, 34437, 41.037102, 28.986601, 0, 0
GeoIP City Edition, Rev 1: TR, 71, Konya, Konya, 42100, 37.927299, 32.419899, 0, 0

GeoIP City Edition, Rev 1: US, MA, Massachusetts, Haverhill, 01835, 42.752998, -71.086098, 506, 978

[VPN user]

Oooooooops. Scratch that last post because I included the wrong sed-like
section from your vpninit file.

Here is the CORRECT sed-like regular expresssion "thingey" you used:
.... corrected post below ....

I called it "sed" in quotes but I should have said regular-expressions
or "sed-like" or something like that to indicate what you used.

Here is *exactly* what you used in your wonderful "vpnit" script:
geoData () {

echo "$(basename $0) message: collecting geo-data..."
IFS=$','
set -- $(geoiplookup -f "$geoDat" $ip_address)

country="${2/*: /}"
state_abv="${3/' N/A'/na}"
state_long="${4/' N/A'/na}"
city="${5/' N/A'/na}"
zip="${6/' N/A'/na}"

[Bit Twister]

On Fri, 4 Mar 2016 13:54:52 -0000 (UTC), VPN user wrote:

> I called it "sed" in quotes but I should have said regular-expressions
> or "sed-like" or something like that to indicate what you used.
>
> Here is *exactly* what you used in your wonderful "vpnit" script:
> geoData () {
>
> echo "$(basename $0) message: collecting geo-data..."
> IFS=$','
> set -- $(geoiplookup -f "$geoDat" $ip_address)
> country="${2/*: /}"
> state_abv="${3/' N/A'/na}"
> state_long="${4/' N/A'/na}"
> city="${5/' N/A'/na}"
> zip="${6/' N/A'/na}"
> IFS=$'\t\n '
> }
>
> The problem "I" had was those "sed-like" regular expression braces (or
> whatever they're called)

You can call them bash string operations if you like.
I used substring as key words for searching for them in my unix.help file.

> worked fine the way you wrote them, but every
> time I tried to add more regular expressions or a pipe to awk or cut,
> I couldn't figure out the syntax.

Maybe this will help.
http://tldp.org/LDP/abs/html/refcards.html#AEN22828

[VPN user]

On Fri, 04 Mar 2016 14:02:06 +0000, Bit Twister wrote:

> Maybe this will help.
> http://tldp.org/LDP/abs/html/refcards.html#AEN22828

Nice. Very nice. Thank you for that reference.

I love that everyone helps everyone else, even though we're all at
vastly different levels.

My stuff is very basic, so I will use that as a reference for
understanding bash string operations that Marek put in his
nice vpnit.sh script.

[VPN user]

On Thu, 03 Mar 2016 18:45:22 -0800, Marek Novotny wrote:

> 1> and > redirect the stdout
> 2> redirects the stderr
> &> is both stdout and stderr

Not having ever used these, the syntax is diabolically pernicious.

This is a working command:
$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 8.8.8.8
GeoIP City Edition, Rev 1: US, CA, California, Mountain View, 94040, 37.384499, -122.088097, 807, 650

This generates an error:
$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8
GeoIP City Edition, Rev 1: IP Address not found

This redirects errors away just fine but you lose the data:
$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8 >&/dev/null

But it's diabolically impossible to get just the error redirected!
$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8 >2/dev/null
bash: 2/dev/null: No such file or directory

$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8 2>/dev/null
GeoIP City Edition, Rev 1: IP Address not found

$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8 >2
bash: 2: Permission denied

$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8 >2 /dev/null
bash: 2: Permission denied

[VPN user]

On Fri, 04 Mar 2016 14:33:04 +0000, VPN user wrote:

>> 1> and > redirect the stdout
>> 2> redirects the stderr
>> &> is both stdout and stderr
>
> Not having ever used these, the syntax is diabolically pernicious.

The reason I say it's diabolical is seen when you look here at
the "very simple" example of using "2>":

http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-3.html
$ grep da * 2> grep-errors.txt

OK. That's nice.

So why doesn't *this* work (using the *same* syntax!)?

$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8 2> /dev/null
GeoIP City Edition, Rev 1: IP Address not found

That's why I say it's diabolical.
Clearly I'm missing something obvious, but what?

[Eef Hartman]

VPN user <vpn...@example.com> wrote:
> That's why I say it's diabolical.
> Clearly I'm missing something obvious, but what?

Maybe your default shell isn't bash ??
For instance (t)csh uses a different one for redirections (btw: the
form withOUT digits and/or & is the same in all shells, so >, >> and <
still do work).
The (t)csh can NOT redirect standard error separately, it only knows
the forms > (redirect standard output) and >& (standard out- AND error
BOTH going to the same place).
(I oversimplify, but don't want to make it too complicated).

Other shells still have other redirection rules. As the "original"
Bourne shell only had >, >> and < all others are extensions BY that
specific shell.

The book you got a reference too is specific for the bash shell.

[Marek Novotny]

On 2016-03-04, VPN user <vpn...@example.com> wrote:

> On Thu, 03 Mar 2016 18:45:22 -0800, Marek Novotny wrote:
>
>> 1> and > redirect the stdout
>> 2> redirects the stderr
>> &> is both stdout and stderr
>
> Not having ever used these, the syntax is diabolically pernicious.
>
> This is a working command:
> $ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 8.8.8.8
> GeoIP City Edition, Rev 1: US, CA, California, Mountain View, 94040, 37.384499, -122.088097, 807, 650
>
> This generates an error:
> $ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8
> GeoIP City Edition, Rev 1: IP Address not found
>
> This redirects errors away just fine but you lose the data:
> $ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8 >&/dev/null
>
> But it's diabolically impossible to get just the error redirected!
> $ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8 >2/dev/null
> bash: 2/dev/null: No such file or directory
>
> $ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8 2>/dev/null
> GeoIP City Edition, Rev 1: IP Address not found

This is the correct syntax. I can see I've been very stupid. The
geoiplookup might be making the same stupid mistake I'm making. When I
write an error to the user I just echo it to stdout. I'll have to go
back and change all of that from now on.

So what is likely going on here is that the author isn't writing errors
to stderr. He's writing them to stdout, like I'm doing. Just a stupid
mistake that will drive you nuts...

Here is an example of what I mean...

#!/bin/bash

echo "This is going to stderr" >&2
exit 1

## END

Now I named this stupid script sample. So here is the test to see what I
mean. Save this script and run it. You'll see the following:

$ ./sample
this is going to stderr

Now run it again, like this...

$ ./sample > /dev/null
this is going to stderr

And now run it like this...

$ ./sample 2> /dev/null

And you see nothing... And that's because you will have redirected
stderr and the echo statement went to stderr. The exit code I assigned
is exit 1, for an error, and I wrote it to stderr. So that's much
better.

I'll do this from now on and I can't believe what a freaking stupid
oversight that was. It's likely the same stupid mistake I'm making. It's
not being written properly. Another duh dumbass moment. :(

> $ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8 >2
> bash: 2: Permission denied
>
> $ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8 >2 /dev/null
> bash: 2: Permission denied
>

[VPN user]

On Fri, 04 Mar 2016 15:01:54 +0000, Eef Hartman wrote:

> Maybe your default shell isn't bash ??
> For instance (t)csh uses a different one for redirections (btw: the
> form withOUT digits and/or & is the same in all shells, so >, >> and <
> still do work).
> The (t)csh can NOT redirect standard error separately, it only knows
> the forms > (redirect standard output) and >& (standard out- AND error
> BOTH going to the same place).
> (I oversimplify, but don't want to make it too complicated).
>
> Other shells still have other redirection rules. As the "original"
> Bourne shell only had >, >> and < all others are extensions BY that
> specific shell.
>
> The book you got a reference too is specific for the bash shell.

I'm pretty sure the shell is bash only because it says so in the
top right corner and because it's Ubuntu (with KDE as the desktop).

Plus my ~/.bashrc & ~/.bash_aliases work.

It's got to be bash because I'm too much a novice to change it
otherwise, but is there a way to doubly confirm what the shell is?

[Bit Twister]

env | grep -i shell :)

Your shell does not matter when you declare was you wish the script to execute
with the first line with the magic word "#!" followed with /some/bin/here

[VPN user]

On Fri, 04 Mar 2016 07:26:52 -0800, Marek Novotny wrote:

Marek Novotny wrote just now:

> And now run it like this...
> $ ./sample 2> /dev/null
> And you see nothing... And that's because you will have redirected
> stderr and the echo statement went to stderr. The exit code I assigned
> is exit 1, for an error, and I wrote it to stderr. So that's much
> better.

Thank you for the example script, as a working example is *always*
the best thing (I tweak existing scripts better than I write them).

It worked exactly was you noted, so, I guess, as you said, the geoiplookup
isn't sending errors to standard error but to standard out, which, for me,
means I have to handle the errors as if they're the same as the output
even though all the fields and field separators are different.

Luckily I can easily do that and I already have done that, but at
least you *explained* why I was so frustrated trying to get rid of
standard error but not standard out in the geoiplookup command.

I will write up a summary, separately, of the steps that I use to
obtain, organize, and use the vpn files from vpngate so that others
can benefit from all the efforts of everyone here.

[Eef Hartman]

VPN user <vpn...@example.com> wrote:
> I'm pretty sure the shell is bash only because it says so in the
> top right corner and because it's Ubuntu (with KDE as the desktop).

There are two different things: the interactive shell (for the
specific user, set in /etc/passwd) and the default shell, used by
scripts that do not request an explicit one with
#!/path/to/shell

The latter is determined by what /bin/sh is pointing TO and i.e. for
Debian (and maybe Ubunto too, I don't know) that is dash: the Debian
Almquist SHell, less powerfull but MUCH faster in scripts then bash

> It's got to be bash because I'm too much a novice to change it
> otherwise, but is there a way to doubly confirm what the shell is?

Interactive: grep "your username" /etc/passwd
Default: ls -l /bin/sh

You _can_ always let a script use bash by putting
#!/bin/bash
as the FIRST line of that script.
The _default_ though is /bin/sh and that will be using the above
mentioned link.

[VPN user]

On Fri, 04 Mar 2016 15:36:37 +0000, Bit Twister wrote:

> env | grep -i shell

$ env | grep -i shell
SHELL=/bin/bash
KONSOLE_PROFILE_NAME=Shell
SHELL_SESSION_ID=1962156318f2400000c122bfa7f7dca4

> Your shell does not matter when you declare was you wish the script to execute
> with the first line with the magic word "#!" followed with /some/bin/here

That's what I would have thought also.

Anyway, Marek explained why I was having diabolical thoughts when trying
to redirect the "geoiplookup" command errors.

Apparently both the errors and the actual output of geoiplookup go to standard
out, even though the syntax and field separators are vastly different between
an output and an error of that command.

Typical output:

$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 8.8.8.8
GeoIP City Edition, Rev 1: US, CA, California, Mountain View, 94040, 37.384499, -122.088097, 807, 650

Typical error:

$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.8.8.8

GeoIP City Edition, Rev 1: IP Address not found

$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 0.0.0.0
GeoIP City Edition, Rev 1: can't resolve hostname ( 0.0.0.0 )

What I wanted in the script was the typical output but not the error!
Anyway, I solved it with a bunch of seds, greps, awks, and cut.

Marek's method of using what Bit Twister called "bash string operations"
was elegant, but I couldn't extend them to handle the errors and
funky characters that are in place names, so, I resorted to a bunch
of dumb echos, seds, awks, cuts, and greps.

The problem I had was only that the errors went to standard out. :(

In the snippet below, $x is the original ovpn file filespec and
the caret-M below is the special character ^M (newline character)
that is at the end of every line in the given ovpn files!

# Get server, protocol, & port:
SERVER=$(grep ^remote $x|sed -e 's/^M//g'|awk '{print $2}'|sed -e 's/ /-/g')
PROTOCOL=$(grep ^proto $x|sed -e 's/^M//g'|awk -F" " '{print $2}'|sed -e 's/ /-/g')
PORT=$(grep ^remote $x|sed -e 's/^M//g'|awk '{print $3}'|sed -e 's/ /-/g')

# Get IP address:

IP=$(route -n|grep -v ^"0.0.0.0"|grep -v ^"10.211.1."|grep -v ^"192.168.1.0"|grep -v ^"128.0.0.0"|grep -v ^Kernel|grep -v ^Destination|awk '{print $1}')

# Get geo data:

GEO_STRING=$(geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat $IP)
GEO_COUNTRY=$(echo $GEO_STRING|awk -F, '{print $2}'|sed -e 's/Rev 1: //' -e 's/ //g'|cut -c 1-2)
GEO_STATE=$(echo $GEO_STRING|awk -F, '{print $3}'|sed -e 's/ //g'|sed -e 's/N\/A/na/')
GEO_LONGSTATE=$(echo $GEO_STRING|awk -F, '{print $4}'|sed -e 's/ //g'|sed -e 's/N\/A/na/'|tr "'" "-")
GEO_CITY=$(echo $GEO_STRING|awk -F, '{print $5}'|sed -e 's/ //g'|sed -e 's/N\/A/na/')

mv $x ${VPN_WINNERS}${VPN_PREFIX}_${GEO_COUNTRY}_${GEO_STATE}_${GEO_LONGSTATE}_${GEO_CITY}_${IP}-${SERVER}_${PROTOCOL}${PORT}_${DATE}.ovpn

I really *loved* how elegant Marek's examples were in his 'vpnit' script,
but, I just couldn't extend his elegant *bash string operations* so I
resorted to the method I used above (which isn't elegant but it works).

[VPN user]

On Fri, 04 Mar 2016 15:51:18 +0000, Eef Hartman wrote:

> Interactive: grep "your username" /etc/passwd
> Default: ls -l /bin/sh

$ grep `whoami` /etc/passwd
$ grep $(whoami) /etc/passwd

Results in:
vpn-user:x:1000:1000:Vpn-user:/home/vpn-user:/bin/bash

> You _can_ always let a script use bash by putting
> #!/bin/bash
> as the FIRST line of that script.
> The _default_ though is /bin/sh and that will be using the above
> mentioned link.

Thanks for the advice.

All my scripts are based off of Marek's scripts, which all
start with the hash-bang bash line.

I have no need for more than one headache, so, the bash
shell is fine for me.

[VPN user]

For the benefit of anyone still interested, I made a major change
today in that now multiple programs can operate on the same config
file even if the file name has been moved, all the while the vpn
session is running.

What I do in order to ensure that multiple programs can access
the video file while the openvpn session is active is that I save
to /tmp a process-id-stamped text file containing the full path
to the config file.

If the vpngeo.sh program renames the file, then the vpngeo.sh
program updates that full path in the pid-stamped tmp file.

If subsequently vpnspeed.sh wants to rename the file, vpnspeed
gets the file name from that pid-stamped tmp file.

All this is possible because Marek kindly showed me how to get
the process ID and BitTwister showed me how to run the if operations.

# Abort if openvpn is not currently running
_openvpnPID=$(pgrep openvpn)
if [ $? -ne 0 ] ; then
echo "Oops: There's nothing to do as openvpn isn't running; aborting"
exit 1
else
# Define a common file for use by other programs
_vpnPIDtmpFile=/tmp/openvpn_${_openvpnPID}
fi

Once that common file is defined, the very next command after a
geographic-based or speed-based rename will be to update that
common file with the new filespec:
mv "$_fullFileSpec" "$_renameFileSpec"
echo $_renameFileSpec > $_vpnPIDtmpFile

With that in mind, now any program subsequent program can test for the
original filespec (based on the openvpn process id) or, if the original
file doesn't exist, the program gets a second chance by looking at the
temporary process-id related file's contents, as shown in this snippet:

# Define the filespec, even if the file has been moved since openvpn first ran:
if [ -f $_fullFileSpec ]; then
echo "fullFileSpec exists at $_fullFileSpec"
else
if [ -f $_vpnPIDtmpFile ]; then
_fullFileSpec=$(cat $_vpnPIDtmpFile)
echo "The renamed file exists at $_fullFileSpec"
else
echo "The $_vpnPIDtmpFile is missing; aborting ..."
exit 1
fi
fi

What this allows is MULTIPLE USE MODELS.

I can run multiple vpn-related programs now, and they can all operate
on the same config file, even if that config file has moved since
openvpn was initially started.

For example, I can run openvpn in a directory of new config files:
$ vpnrun
And, while VPN is running, I can change the name based on geography:
$ vpngeo (which will rename the original file)
Since that takes only seconds, I can then rename based on speed:
$ vpnspeed (which will re-rename by adding speedtest results)

Of course, other use models exist, eg., I can automate all of that,
but, I just wanted to let you know that I've added this manual
capability of having multiple programs access the config file,
even if the config file's location (and name) has changed.

Thanks for all the coding help and advice!

[Bit Twister]

On Sat, 19 Mar 2016 00:08:41 -0000 (UTC), VPN user wrote:

> All this is possible because Marek kindly showed me how to get
> the process ID and BitTwister showed me how to run the if operations.

But the lesson failed to take. :(

Before--------------

# Abort if openvpn is not currently running
_openvpnPID=$(pgrep openvpn)
if [ $? -ne 0 ] ; then
echo "Oops: There's nothing to do as openvpn isn't running; aborting"
exit 1
else
# Define a common file for use by other programs
_vpnPIDtmpFile=/tmp/openvpn_${_openvpnPID}
fi

After--------------

# Abort if openvpn is not currently running
_openvpnPID=$(pgrep openvpn)
if [ $? -ne 0 ] ; then
echo "Oops: There's nothing to do as openvpn isn't running; aborting"
exit 1

fi

# Define a common file for use by other programs
_vpnPIDtmpFile=/tmp/openvpn_${_openvpnPID}

> # Define the filespec, even if the file has been moved since openvpn first ran:
> if [ -f $_fullFileSpec ]; then
> echo "fullFileSpec exists at $_fullFileSpec"
> else
> if [ -f $_vpnPIDtmpFile ]; then
> _fullFileSpec=$(cat $_vpnPIDtmpFile)
> echo "The renamed file exists at $_fullFileSpec"
> else
> echo "The $_vpnPIDtmpFile is missing; aborting ..."
> exit 1
> fi
> fi

Look what you can do with negative logic:

# Define the filespec, even if the file has been moved since openvpn first ran:

if [ ! -f $_fullFileSpec ]; then
if [ ! -f $_vpnPIDtmpFile ]; then

echo "The $_vpnPIDtmpFile is missing; aborting ..."
exit 1
fi

_fullFileSpec=$(cat $_vpnPIDtmpFile)
fi