Find an unused local port and ranomly pick one from them each time.

[Hongyi Zhao]

Hi all,

I want to find an unused local port and randomly pick one from them each
time. At the same time, the method should immune from the race condition
between attempting to open the port and actually using it.

Based on the following website:

http://unix.stackexchange.com/questions/55913/whats-the-easiest-way-to-
find-an-unused-local-port

It seems the following shell code will do the job:

#!/bin/bash
read LOWERPORT UPPERPORT < /proc/sys/net/ipv4/ip_local_port_range
while :
do
PORT="`shuf -i $LOWERPORT-$UPPERPORT -n 1`"
ss -lpn | grep -q ":$PORT " || break
done
echo $PORT

But I'm not sure whether this method can immune from the race condition
between attempting to open the port and actually using it. Any hints on
this issue?

Regards
--
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.

[Bit Twister]

On Sat, 16 Jan 2016 02:31:52 -0000 (UTC), Hongyi Zhao wrote:
> Hi all,
>
> I want to find an unused local port and randomly pick one from them each
> time. At the same time, the method should immune from the race condition
> between attempting to open the port and actually using it.

Offhand I would say your design is flawed. There should be no race
condition between open and usage.

My guess is you are talking about a race condition between picking a
port and opening it.

Only method I know is to open port for exclusive use and if fails,
pick another port.

[Kaz Kylheku]

On 2016-01-16, Hongyi Zhao <hongy...@gmail.com> wrote:
> Hi all,
>
> I want to find an unused local port and randomly pick one from them each
> time. At the same time, the method should immune from the race condition
> between attempting to open the port and actually using it.
>
> Based on the following website:
>
> http://unix.stackexchange.com/questions/55913/whats-the-easiest-way-to-
> find-an-unused-local-port
>
> It seems the following shell code will do the job:
>
> #!/bin/bash
> read LOWERPORT UPPERPORT < /proc/sys/net/ipv4/ip_local_port_range
> while :
> do
> PORT="`shuf -i $LOWERPORT-$UPPERPORT -n 1`"
> ss -lpn | grep -q ":$PORT " || break
> done
> echo $PORT
>
> But I'm not sure whether this method can immune from the race condition
> between attempting to open the port and actually using it.

Of course it isn't immune. What steps in the script do you see that
lock down the port so that no other process can create a socket and bind
to it?

In your loop, you have to actually run the program which binds
the port that you think you have found. The program has to be designed
to accept a specified port as input, try to bind a socket to that port,
or else fail.

The script can detect that the program has failed, and then run the loop
again.

Basically, if your loop doesn't *acquire* the port before terminating,
there is a race.

[Hongyi Zhao]

On Sat, 16 Jan 2016 04:47:01 +0000, Kaz Kylheku wrote:

> Of course it isn't immune. What steps in the script do you see that lock
> down the port so that no other process can create a socket and bind to
> it?
>
> In your loop, you have to actually run the program which binds the port
> that you think you have found. The program has to be designed to accept
> a specified port as input, try to bind a socket to that port, or else
> fail.
>
> The script can detect that the program has failed, and then run the loop
> again.
>
> Basically, if your loop doesn't *acquire* the port before terminating,
> there is a race.

Thanks a lot for your analysis.

In fact, the background that I post this thread is the problem I
encountered as follows:

I want to find the usable vpngate's free vpn servers. The steps are
described as follows:

1- Obtain all of the OpenVPN Configuration File (.ovpn file) which are
using TCP protocol from the following website:

http://www.vpngate.net/en/

2- Runing on all of these .ovpn files in parallel with the following
command for each of the .ovpn file:

sudo openvpn --config the-ovpn-file --route-nopull --connect-retry-max 1
--remap-usr1 SIGTERM --tls-exit

Currently, I've 8000+ .ovpn files accumulated for long time. Therefor I
must run the above command in parallel to do the testing, i.e., put each
running of the above command into background like the following:

for i in list-of-all-the-ovpn-files
do
sudo openvpn --config $i --route-nopull --connect-retry-max 1
--remap-usr1 SIGTERM --tls-exit &> /dev/null &
done

When I do the above loop testing on all of these ovpn files, it seems
some of the connected openvpn instances will be
closed when I run a number of openvpn commands with the above method.

Furthermore, I inspected all of the ovpn config files, they all have the
following three lines in them besides other stuff:

proto tcp
remote <the-ip-of-vpn-server> <the-port-of-server>
nobind

After a further thinking, it seems that the above issue is cause by
the nobind option used in the config files. In detail, with the nobind
option, the openvpn client with using lport = 0, which means the openvpn
client will ask the kernel to allocate a port from it's
ip_local_port_range.

For my case, I running all of these openvpn client in parallel by putting
them into background, which maybe cause race condition for the kernel to
allocate *same* unused ports for each openvpn client instance.

Currently, I think the possible solutions for this issue are as follows:

Method 1:

Find all of the unused local port and ranomly pick one from them each
time for each openvpn client instance, i.e., the topic of this thread.

Method 2:

Sleep some time between each initial running of openvpn client instance:

for i in list-of-all-the-ovpn-files
do
sudo openvpn --config $i --route-nopull --connect-retry-max 1
--remap-usr1 SIGTERM --tls-exit &> /dev/null &
sleep 0.1
done

The sleep command will give enough time for the kernel to solve the race
condition and dynamically allocate appropriate local ports for each
openvpn client instance.

Method 3:

Comment out the nobind line in the config file, and then using the --
lport
option to assign a unused random ports for each openvpn client instance.

To summarize, the method 1 and 3 are based on the user-space's local-port
assign scheme, and the method 2 are based on the OS to assign the dynamic
local ports.

[Marc Haber]

Hongyi Zhao <hongy...@gmail.com> wrote:
>I want to find the usable vpngate's free vpn servers.

Why?

>The steps are
>described as follows:
>
>1- Obtain all of the OpenVPN Configuration File (.ovpn file) which are
>using TCP protocol from the following website:
>
> http://www.vpngate.net/en/

So they do publish public servers that are actually unusable?

Greetings
Marc
--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber | " Questions are the | Mailadresse im Header
Mannheim, Germany | Beginning of Wisdom " | http://www.zugschlus.de/
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834

[Hongyi Zhao]

On Sat, 16 Jan 2016 13:49:39 +0100, Marc Haber wrote:

>>I want to find the usable vpngate's free vpn servers.
>
> Why?

These servers are contributed by volunteers over the world, it maybe
opened and closed time to time. OTOH, you must select the ones which are
more fast and stable for your location.

>
>>The steps are described as follows:
>>
>>1- Obtain all of the OpenVPN Configuration File (.ovpn file) which are
>>using TCP protocol from the following website:
>>
>> http://www.vpngate.net/en/
>
> So they do publish public servers that are actually unusable?

I think the usability / stability are depended on the tester's criteria
and geo-locations.

[Marc Haber]

Hongyi Zhao <hongy...@gmail.com> wrote:
>On Sat, 16 Jan 2016 13:49:39 +0100, Marc Haber wrote:
>>>I want to find the usable vpngate's free vpn servers.
>>
>> Why?
>
>These servers are contributed by volunteers over the world, it maybe
>opened and closed time to time. OTOH, you must select the ones which are
>more fast and stable for your location.

So you intend to connect to hundreds of servers for testing which one
is best for you?

[Hongyi Zhao]

On Sat, 16 Jan 2016 15:21:21 +0100, Marc Haber wrote:

> So you intend to connect to hundreds of servers for testing which one is
> best for you?

Yes. And based on the best ones to construct the equal-cost multipath-
routing. And then based on the ipset + iproute2 to do the policy based
routing.

Regards

>
> Greetings Marc

[Kaz Kylheku]

On 2016-01-16, Hongyi Zhao <hongy...@gmail.com> wrote:

> For my case, I running all of these openvpn client in parallel by putting
> them into background, which maybe cause race condition for the kernel to
> allocate *same* unused ports for each openvpn client instance.

That would be an almost unimaginably spectacular programming blunder
in your kernel, I am afraid, which would show up when doing mundane
things like surfing the web (multiple passive sockets in your browser
getting the same local endpoint address and getting confused).

Almost any explanation for whatever you are seeing is more
plausible.

[Hongyi Zhao]

On Sat, 16 Jan 2016 14:50:42 +0000, Kaz Kylheku wrote:

> On 2016-01-16, Hongyi Zhao <hongy...@gmail.com> wrote:
>> For my case, I running all of these openvpn client in parallel by
>> putting them into background, which maybe cause race condition for the
>> kernel to allocate *same* unused ports for each openvpn client
>> instance.

Just my guess, seems I'm absolutely wrong ;-(

After all, the Linux kernel is developed and maintained by so many IT
genius. Such low level issue will not dare to appear in kernel level.

>
> That would be an almost unimaginably spectacular programming blunder in
> your kernel, I am afraid, which would show up when doing mundane things
> like surfing the web (multiple passive sockets in your browser getting
> the same local endpoint address and getting confused).
>
> Almost any explanation for whatever you are seeing is more plausible.

But from my experience, the following method will solve this issue:

for i in list-of-all-the-ovpn-files
do
sudo openvpn --config $i --route-nopull --connect-retry-max 1
--remap-usr1 SIGTERM --tls-exit &> /dev/null &

sleep 0.01
done

If I don't use the ``sleep 0.01'', most of the already established openvpn
connections maybe effected close suddenly along with the subsequent
starting of openvpn client instances.

If I decrease the duration of sleep command, say, to 0.001 second, and
then the issue will be appear again.

[Memory Eraser]

>
> I want to find the usable vpngate's free vpn servers. The steps are
> described as follows:
>
> 1- Obtain all of the OpenVPN Configuration File (.ovpn file) which are
> using TCP protocol from the following website:
>
> http://www.vpngate.net/en/
>
> 2- Runing on all of these .ovpn files in parallel with the following
> command for each of the .ovpn file:
>
> sudo openvpn --config the-ovpn-file --route-nopull --connect-retry-max 1
> --remap-usr1 SIGTERM --tls-exit
>
> Currently, I've 8000+ .ovpn files accumulated for long time. Therefor I
> must run the above command in parallel to do the testing, i.e., put each
> running of the above command into background like the following:

I suggest you to use Tor instead.

[Hongyi Zhao]

On Sun, 17 Jan 2016 10:49:54 +0800, Memory Eraser wrote:

> I suggest you to use Tor instead.

No, Tor has been blocked completely for my location.