possible problem with openvpn appliance version 02

Bob Andrews

unread,

Jan 23, 2012, 1:54:01 AM1/23/12

to openvpn-...@googlegroups.com

FYI:

I deployed version 02 a few days ago under vmware workstation 8. I used the vmware ovf tool
to convert the distribution into something usable by vmware.

Last night, I attempted to remotely connect, and was unable to get in. I connected in via an
alternate method, and found I could not ping the virtual machine from the local lan. Looking
at the VM's console, there were no interesting messages suggesting what might be wrong.

I tried the "ctrl-alt-f2" method to get a login prompt from the VM; this isn't usable in my
environment. I connect via VNC to the host running vmware; that's how I get to vmware.
When I attempt to use "ctrl-alt-f2", the key sequence is processed locally, not in the VNC
session.

I power-cycled the VM, and was immediately able to connect back into my network
via openvpn.

Fast-forward ~24 hours. I'm locally connected to the LAN now. I am no longer able to
ping the openvpn appliance running in the VM. Other VMs are still working OK.

Bob

Radu Constantinescu

unread,

Jan 23, 2012, 10:37:59 AM1/23/12

to openvpn-...@googlegroups.com

Bob,

Cannot help you unless you can get in and see what is in there.
If VNC is catching the ALT-Fx sequence you will have problems with any
linux machine (not being able to change to a different local console)
- so is a good idea to understand why is that happening and how can be
avoided. (the key sequence is ALT+F1, ALT+F2, not Ctrl+Alt+F2)

If you want to reactivate the console on tty1 go and edit
/etc/inittab, look for the line below and remove the # in front of it.
#1:2345:respawn:/sbin/mingetty tty1

Regards,
Radu

bob.442

unread,

Jan 23, 2012, 1:44:45 PM1/23/12

to openvpn-appliance

OK, got the console problem fixed. By the way, the network had
stopped working
again this morning. Next time it happens, I will be able to poke
around via the console.

Bob

Bob Andrews

unread,

Jan 24, 2012, 8:44:51 PM1/24/12

to openvpn-appliance

FYI: I have the VM in the state where it has lost the network, and I am
logged in on the console. I should point out that I did see something
similar to this on one of my other VMs (I run half a dozen of them 24/7).
I was able to get the network working again on the other VM by restarting
the network (via the script in /etc/init.d). So the problem with losing the
network in the openvpn appliance *may* be specific to my VMware
installation. Still, the openvpn appliance has lost the network approx.
every day or so since installing the 02 version, and the other VMs,
except for that one failure, have been working fine.

Are there any commands you would like me to run on the openvpn
appliance to see what is going on?

Bob

Radu Constantinescu

unread,

Jan 25, 2012, 10:17:02 AM1/25/12

to openvpn-...@googlegroups.com

ifconfig
ip route list
ip addr list
brctl show
cat /etc/sysconfig/network-scripts/ifcfg-eth0

The instance that I use is up since 5 days ago with no problems but
let's see about yours. I have it on a esxi 5 server.

Thanks,
Radu

Bob Andrews

unread,

Jan 25, 2012, 12:43:20 PM1/25/12

to openvpn-...@googlegroups.com

Here's the requested info. The second-to-last line looks suspicious.

Bob

---
+ ifconfig
br0       Link encap:Ethernet HWaddr 00:0C:29:05:F2:27
          inet addr:10.8.13.12 Bcast:10.8.13.255 Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe05:f227/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:355074 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1057 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:203334982 (193.9 MiB) TX bytes:65057 (63.5 KiB)

eth0      Link encap:Ethernet HWaddr 00:0C:29:05:F2:27
          inet6 addr: fe80::20c:29ff:fe05:f227/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:1429740951 errors:61532 dropped:1105 overruns:0 frame:0
          TX packets:1094 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4244786204 (3.9 GiB) TX bytes:70228 (68.5 KiB)
          Interrupt:177 Base address:0x1400

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:207 errors:0 dropped:0 overruns:0 frame:0
          TX packets:207 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:23111 (22.5 KiB) TX bytes:23111 (22.5 KiB)

tap0      Link encap:Ethernet HWaddr 92:FA:E5:9D:D4:D2
          inet6 addr: fe80::90fa:e5ff:fe9d:d4d2/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:371838 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:0 (0.0 b) TX bytes:211784128 (201.9 MiB)

+ ip route list
10.8.13.0/24 dev br0 proto kernel scope link src 10.8.13.12
default via 10.8.13.1 dev br0
+ ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0c:29:05:f2:27 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20c:29ff:fe05:f227/64 scope link
       valid_lft forever preferred_lft forever
3: sit0: <NOARP> mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0
4: tap0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 92:fa:e5:9d:d4:d2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::90fa:e5ff:fe9d:d4d2/64 scope link
       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
    link/ether 00:0c:29:05:f2:27 brd ff:ff:ff:ff:ff:ff
    inet 10.8.13.12/24 brd 10.8.13.255 scope global br0
    inet6 fe80::20c:29ff:fe05:f227/64 scope link
       valid_lft forever preferred_lft forever
+ brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.000c2905f227       no              tap0
                                                        eth0
+ cat /etc/sysconfig/network-scripts/ifcfg-eth0
#
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
---

Radu Constantinescu

unread,

Jan 25, 2012, 10:19:53 PM1/25/12

to openvpn-...@googlegroups.com

This is WRONG
> DEVICE=eth0
> BOOTPROTO=dhcp
> ONBOOT=yes

In the web interface go to IP Config and click change

The
cat /etc/sysconfig/network-scripts/ifcfg-eth0
must show a fixed ip config after reboot.

Radu

Bob Andrews

unread,

Jan 26, 2012, 12:06:04 AM1/26/12

to openvpn-...@googlegroups.com

OK, done. I guess I assumed that in the process of applying the saved config,
DHCP would end up getting disabled.

ifcfg-eth0 no longer shows DHCP.

Thanks,

Bob

Bob Andrews

unread,

Jan 26, 2012, 4:23:44 PM1/26/12

to openvpn-...@googlegroups.com

Looks like it wasn't DHCP. The VM lost the network again. Rebooting
brought it back. Prior to rebooting, I ran the script again to grab the
same outputs previously requested. Appended, below.

Bob

---
+ ifconfig
br0       Link encap:Ethernet HWaddr 00:0C:29:05:F2:27
          inet addr:10.8.13.12 Bcast:10.8.13.255 Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe05:f227/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

          RX packets:85386 errors:0 dropped:0 overruns:0 frame:0
          TX packets:613 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:62399629 (59.5 MiB) TX bytes:41209 (40.2 KiB)

eth0      Link encap:Ethernet HWaddr 00:0C:29:05:F2:27
          inet6 addr: fe80::20c:29ff:fe05:f227/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1

          RX packets:354322321 errors:393 dropped:396 overruns:0 frame:0
          TX packets:681 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1995924200 (1.8 GiB) TX bytes:48966 (47.8 KiB)

          Interrupt:177 Base address:0x1400

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1

          RX packets:125 errors:0 dropped:0 overruns:0 frame:0
          TX packets:125 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:14560 (14.2 KiB) TX bytes:14560 (14.2 KiB)

tap0      Link encap:Ethernet HWaddr BE:08:8F:AE:B3:60
          inet6 addr: fe80::bc08:8fff:feae:b360/64 Scope:Link

UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:90792 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:0 (0.0 b) TX bytes:64705862 (61.7 MiB)

+ ip route list
10.8.13.0/24 dev br0 proto kernel scope link src 10.8.13.12
default via 10.8.13.1 dev br0
+ ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0c:29:05:f2:27 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::20c:29ff:fe05:f227/64 scope link
       valid_lft forever preferred_lft forever
3: sit0: <NOARP> mtu 1480 qdisc noop
    link/sit 0.0.0.0 brd 0.0.0.0
4: tap0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 100

link/ether be:08:8f:ae:b3:60 brd ff:ff:ff:ff:ff:ff
inet6 fe80::bc08:8fff:feae:b360/64 scope link

       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
    link/ether 00:0c:29:05:f2:27 brd ff:ff:ff:ff:ff:ff
    inet 10.8.13.12/24 brd 10.8.13.255 scope global br0
    inet6 fe80::20c:29ff:fe05:f227/64 scope link
       valid_lft forever preferred_lft forever
+ brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.000c2905f227       no              tap0
                                                        eth0
+ cat /etc/sysconfig/network-scripts/ifcfg-eth0

# Generated by OvpnAppliance
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
IPADDR=10.8.13.12
NETMASK=255.255.255.0
GATEWAY=10.8.13.1
TYPE=Ethernet

Radu Constantinescu

unread,

Jan 26, 2012, 5:18:18 PM1/26/12

to openvpn-...@googlegroups.com

Well this time I see nothing wrong.

R

Bob Andrews

unread,

Jan 26, 2012, 5:31:31 PM1/26/12

to openvpn-...@googlegroups.com

Admittedly grasping at straws here...

Maybe it is the manner in which I tend to connect and disconnect. One of the things
I noticed the other day was that when I rebooted the VM, and did not connect with
openvpn to it, it stayed up for > 24 hours. Once I connect, it is rare to see the network
continue to work in the VM for more than 12 hours.

I tend to connect with half a dozen or so different laptops and desktops, from different
locations and external IP addresses, all using the same userid & password, but never
at the same time (except once, for testing, when both remote systems ended up with
the same IP address). Sometimes I explicitly disconnect when I am done, and sometimes
I forget and just shut the remote system off, or otherwise disconnect it from the internet.

I'll continue to pay attention to how I'm using it and continue trying to correlate that
with the network failures. I've also added "dmesg" to the list of commands to issue when
the network fails.

Bob

Bob Andrews

unread,

Jan 31, 2012, 12:46:08 AM1/31/12

to openvpn-...@googlegroups.com

A little more info. When the VM with the openvpn appliance in it loses the
network, it still transmits. With tcpdump, I can see it arp'ing from other
hosts on the LAN. The VM never sees the responses. The host running
VMware 8 does see the ARP responses...

And now, the strange thing -- I just paused the VM and resumed it, and the
network in the VM started working again! This is starting to smell like a
VMware issue, but my other VMs are not dropping like flies the way the
openvpn appliance is.

I'm still watching for more clues.

Bob

Radu Constantinescu

unread,

Jan 31, 2012, 9:33:02 AM1/31/12

to openvpn-...@googlegroups.com

It does smell like a vmware issue if you pause/unpause the the VM and
then is working.
I do not have vmware 8 and if I am not wrong you are using vmware 8
for linux - see what settings you have for the vmware switch
(promiscous, etc) but those are OK otherwise will not work.
Let me know if you see something that I will be able to investigate.
I have one of this up since 10 days ago on a ESXI server with no
problems. the vswitch is set to accept promiscuous mode and nothing
else.
09:31:23 up 10 days, 22:57, 1 user, load average: 0.00, 0.00, 0.00

Regards,
Radu

Bob Andrews

unread,

Feb 4, 2012, 12:13:07 AM2/4/12

to openvpn-...@googlegroups.com

After trying many things to narrow down the problem, I've stumbled onto what
appears to be the solution: install the vmware-tools in the VM. I've always
been under the impression that the vmware-tools were optional. The included
drivers enable shared drives and make the virtual console work better...but
my experience to this point has been that they didn't *have* to be installed.

Since installing them however, the openvpn appliance has been up, with a
functioning network, far longer than it ever has (using the second openvpn
release, under vmware workstation 8):

-> 00:10:43 up 1 day, 10:25, 1 user, load average: 0.00, 0.00, 0.00

Bob

Bob Andrews

unread,

Feb 4, 2012, 5:08:52 AM2/4/12

to openvpn-...@googlegroups.com

I may have spoke too soon. :-( The network stopped working again. One more
piece of info...on the console "ifconfig eth0 down; ifconfig eth0 up" caused the
network to start working again.

I'll watch it for a few more days; I may try going back to VMware Workstation 7
if I continue to reproduce the network failure.

Bob

Radu Constantinescu

unread,

Feb 6, 2012, 5:19:54 PM2/6/12

to openvpn-...@googlegroups.com

Bob,

I think that you are fighting a vmware/suse issue.
VMware tools are not requested for a vm to work but now that you have
them installed you cna change the interface type to VMXNET3 that is
the best choice with vmware tools installed.
Also be aware that vmware tols will try to synchronize the time in the
vm and ntpd does the same so stop one of them.

Radu

Bob Andrews

unread,

Feb 7, 2012, 3:08:44 AM2/7/12

to openvpn-...@googlegroups.com

I think you are likely correct about that. FYI: A couple days
ago, I set up a cron job on the VM which uses ping to test
the network every 15 minutes. If the network is down, it
repairs it using "ifconfig eth0 down; ifconfig eth0 up"....and
then sends me email. This helped. I did get periodic emails,
but my openvpn sessions were much more reliable.

Thanks for the ntpd tip. I issued this command to cause
ntpd not to run:

chkconfig --level 3 ntpd off

Earlier today, I found that VMware released Workstation 8.0.2
recently. I was running 8.0.1, so I upgraded. I haven't seen a
failure yet, but it is too soon to tell. The way things have been
going, I need to see things work for several days before I can
draw a conclusion that the problem is solved. Of course it will
be easy to draw the conclusion that it isn't solved if the network
in the VM fails again. :-(

Bob

Bob Andrews

unread,

Feb 15, 2012, 11:52:16 PM2/15/12

to openvpn-...@googlegroups.com

The appliance has been running under VMware Desktop 8.02 for a little
over a week now. For the first few days, no failure. Then a couple of them
half an hour apart. And then one or two more over the course of the next
few days. So whatever it is, it's still there. However, with my cron job
fixing the VM whenever it fails, it works well enough for my needs at this
time. I'm going to stop worrying about it for the time being. If I ever do
come across the root cause, I'll post it.

Bob

Bob Andrews

unread,

Mar 2, 2012, 12:05:36 PM3/2/12

to openvpn-...@googlegroups.com

In the fullness of time, I think the cause, or more accurately, the trigger, of the
mysterious network failure in the VM I've been seeing is starting to become
clear. The host running VMware workstation has several guests, one of
which is the openvpn client. Another one occasionally is used to download
linux distros, etc., with bittorrent. The failure rate in the VM has been quite
low recently, until last night. At the same time, the bittorrent VM wasn't doing
any downloads. Last night I started one. By morning, the openvpn appliance
had lost its network half a dozen times.

Thinking about this, since the appliance puts the ethernet in promiscuous mode,
it has to handle, in software, every packet going in and out of the bittorrent VM.
This shouldn't cause any problem, other than extra load on the host running VMware,
and perhaps some reduced performance. But it is causing a problem -- it is
probably triggering a flaw in a driver somewhere that is normally not triggered.

The VMware host has an extra ethernet port which is not currently being used.
When I get a little time, I'll create a second vmnet interface which is bound to the
second ethernet, and offload the bittorrent VM (or the appliance) to the second
interface. If my theory is correct, the problem will disappear, because the
appliance's promiscious interface will no longer have to deal with the bittorrent
traffic. The switch external to the VMware-serving host will filter all of that out,
because the VMware-serving host will use two separate physical interfaces for
bt and vpn.

Bob

Reply all

Reply to author

Forward