Cannot connect since 3.6.7 any more

[tempe...@gmail.com]

I am connecting to pw.openvpn.ipredator.se.

Since 3.6.7, the connection does not work any more.

I can reproduce this problem fully - When I downgrade to 3.6.5 or 3.6.6, I can connect again.

I also tried 3.8.8b2, which has the same problem.

I've copied the diagnostics from 3.8.8b2, using log level 4. It's 850 lines. Shall I post them here as a comment?

For now, I'm just posting the log part that's showing the progress, and which appear to be different, first from 3.6.6:

2016-09-20 20:40:54 us=266204 UDPv4 link local: [undef]

2016-09-20 20:40:54 us=266277 UDPv4 link remote: [AF_INET]46.246.39.130:1194

2016-09-20 20:40:54 us=266455 MANAGEMENT: >STATE:1474396854,WAIT,,,

2016-09-20 20:40:54 us=313168 MANAGEMENT: >STATE:1474396854,AUTH,,,

2016-09-20 20:40:54 us=313331 TLS: Initial packet from [AF_INET]46.246.39.130:1194, sid=869d7c79 d1712cfd

2016-09-20 20:40:54 us=313595 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this

2016-09-20 20:40:54 us=373694 VERIFY OK: depth=1, C=SE, ST=Bryggland, L=Oeldal, O=Royal Swedish Beer Squadron, OU=Internetz, CN=Royal Swedish Beer Squadron CA, emailAddress=<erased>

2016-09-20 20:40:54 us=375050 VERIFY OK: nsCertType=SERVER

And this from 3.6.8b2:

2016-09-20 20:28:09 us=598876 UDPv4 link local: [undef]

2016-09-20 20:28:09 us=598962 UDPv4 link remote: [AF_INET]46.246.35.2:1194

2016-09-20 20:28:09 us=599084 MANAGEMENT: >STATE:1474396089,WAIT,,,

2016-09-20 20:28:09 us=599323 write UDPv4: Host is down (code=64)

2016-09-20 20:28:11 us=751971 write UDPv4: Host is down (code=64)

I see that the IPs are different, but I don't think that's the issue, as the hostname has many IPs, and if they're randomly chosen, then I should have had problems randomly, even with the same TB versions. However, I've now installed the various versions multiple times, and it's always that 3.6.7 and later fails, while earlier ones don't.

I also have the impression that 3.6.7 uses a newer OpenVPN version (2.3.12). Maybe that's the issue here.

Let me know what information I shall gather, and how to post it. I have a web server, so I could also post a zip file if you want something larger.

[Tunnelblick developer]

Tunnelblick itself has nothing to do with actually making the connection; that's all done by OpenVPN, so I believe this is caused by some compatibility problem between the server's version of OpenVPN (whatever that is) and version 2.3.12 in Tunnelblick 3.6.7 and higher. (Tunnelblick 3.6.6 includes OpenVPN 2.3.11. (Both have the same version of OpenSSL – 1.0.2b, so that's not an issue.)

For that reason, posting more info here probably won't be helpful.

There isn't anything in the OpenVPN 3.2.12 change notes that would seem to cause the problem, so your best chance at fixing the problem is probably to work with iPredator.se.

However, if you want to verify that it is the new OpenVPN version that is causing the problem, you can copy a version of OpenVPN from one Tunnelblick application into another. When Tunnelblick is launched, it looks in a folder inside itself for the versions of OpenVPN to offer, so adding another one is easy: you just copy a folder. That will invalidate the application's digital signature, but you can ignore that for the purposes of testing. 

The OpenVPN binaries are located in folders in the Tunnelblick.app/Contents/Resources/openvpn folder. If you have Tunnelblick 3.6.6 installed, quit it, mount the Tunnelblick 3.6.7 disk image, and then copy the "Tunnelblick.app/Contents/Resources/openvpn/openvpn-2.3.12" folder from the copy of Tunnelblick in the disk image into the /Applications/Contents/Resources/openvpn/ folder. (You'll need a computer admin username/password to copy, and when you launch Tunnelblick it will complain about the digital signature, and then you'll probably have to enter the computer admin username/password again to secure the modified version of Tunnelblick.)

After adding the newer version of OpenVPN to Tunnelblick 3.6.6, be sure to select the configuration and set its "OpenVPN version" in the "Settings" tab of the "Configurations" panel of the "VPN Details" window.

If you need more detailed instructions, let me know.

Once you have (I assume) verified that the problem is with OpenVPN 2.3.12, you should contact iPredator or consult with OpenVPN experts:

• OpenVPN FAQ
• OpenVPN HOWTO
• OpenVPN 2.3 Man Page
• OpenVPN Documentation
• OpenVPN Users Forum
• OpenVPN Users Mailing List

[tempe...@gmail.com]

I added OpenVPN 2.3.12 to TB 3.6.6 and activated it in the Settings. I was able to connect. This would suggest that it's not problem with the newest OpenVPN version.

Then I did the reverse, i.e. I added OpenVPN 2.3.11 to TB 3.6.7. This did also let me connect. This would suggest that it's indeed a problem with OpenVPN.

I re-did this twice, just to be sure. I also checked the log to make sure it was really the intended OpenVPN version that got used.

Since this did make little sense, I did some more digging. One thing that I find suspicious is that TB 3.6.7 shows OpenVPN 2.3.11 as the default, even though the alias "default" inside its openvpn folder points to the 2.3.12 version. How can that be?

Still looking into this, but I just wanted to give this update in case you have some thoughts.

[tempe...@gmail.com]

Oh, I forgot to add one important bit: Now, after modding the 3.6.7 by adding 2.3.11, it does suddenly connect even with 2.3.12, too. So, with these mods all four combinations of TB and OpenVPN suddenly work.

I'll now see if I can break it again.

[tempe...@gmail.com]

I just re-installed 3.6.7 fresh from the dmg, and now that works, too. Meaning I can't reproduce the issue any more.

So, it seems that some other resident part outside of the app is involved in fixing this. Where would I look for those, seeing if I can detect something there, by comparing with previous Time Machine backups? Probably something related to OpenVPN.

[tempe...@gmail.com]

Update: The problem resurfaced after a reboot. Will now do the tests with the different openvpn versions once again, with reboots in between.

[Temperest]

Alright - it does appear to be a OpenVPN 2.3.12 problem, after all. When I reboot between tests, it clearly behaves such as it working with 2.3.11 and failing with 2.3.12, no matter which TB version I use.

I'll take this up with the other groups now.

Thanks for all your quick help here, that's much appreciated!

[Tunnelblick developer]

Thanks for the update(s).

Perhaps there is something wrong with a run-time library or kext? Do you have any non-Apple kexts loaded?

/usr/sbin/kextstat | grep -v com.apple

will list non-Apple kexts that are loaded.

If you have a "tap" configuration (as opposed to a "tun" configuration), Tunnelblick will load its own kext to connect, and unload it upon disconnect. For most "tun" configurations, OpenVPN will use the "utun" driver built into OS X, but some configurations require the "tun" kext, in which case Tunnelblick will load/unload it dynamically. Examination of the Tunnelblick/OpenVPN log should show the kext being loaded. (I'm not sure it logs unloads.)

[Temperest]

The only non-Apple kext is Little Snitch. And I should have considered it already, but totally forgot.

And if I disable LS, reboot and connect, then it works even with 2.2.12.

So, it's a combination of both LS and OpenVPN 2.3.12 that causes the issue, it appears now.

I like to keep using LS, though, as I use its automatic profile switching to make sure that some connections are blocked if the VPN is not connected (something, the VPN suddenly disconnects, and then I don't want my apps communicated over the unsecured connection).

Now, with another party being part of the problem, this isn't getting any easier to resolve, though. Damn.

[Temperest]

On  the tun/tap thing, the only related part I found in the log is this:

2016-09-20 23:36:47 us=593019 Opening utun (connect(AF_SYS_CONTROL)): Resource busy
2016-09-20 23:36:47 us=594370 Opened utun device utun1

Is that what you had in mind?

[Tunnelblick developer]

I don't think it is Little Snitch itself, though – I think it is some "rule" or setting that Little Snitch is using. I don't use it myself, but a lot of people do, and I don't remember seeing problems with it and OpenVPN, at least in the last couple of years.

[Temperest]

On Tuesday, September 20, 2016 at 11:39:16 PM UTC+2, Tunnelblick developer wrote:

I don't think it is Little Snitch itself, though – I think it is some "rule" or setting that Little Snitch is using. I don't use it myself, but a lot of people do, and I don't remember seeing problems with it and OpenVPN, at least in the last couple of years.

Well, yes. I never had problems before with LS + TB either, and I haven't made any changes to the LS rules recently, either. But 2.3.12 changed that for me. At least, if others soon report similar issues, my report may help identify a pattern.

[Temperest]

Alright - I've figured it out - and it was like you suspected:

I had authorized the "openvpn" process to access the VPN server. However, the new openvpn version was seen as a different version by Little Snitch, which means I had to add a new rule for this new version - which I didn't realize. The problem was that Little Snitch did not clearly mark both openvpns as different versions - it lists them as if they're the same until you explicitly add the other one - then you have two entries, both with the same name and no way to tell their difference.

With normal apps, this is less of a problem because then LS can show their versions from the Info.plist - but with system tools like openvpn, there's no common version info LS can read, and it makes no attempt to identify them otherwise more clearly.

Oh well, the fun of unix tools and the lack of proper standards :)

But I've solved it now, and am happy to report it's no one's fault but my own. No general problem with OpenVPN whatsoever.

Again, thanks for all your help figuring out my own mistakes :-D

Also, if you like a free license for my OS X apps (iClip, Find Any File), let me know.

[Tunnelblick developer]

OK, thanks for posting such a clear explanation of the problem and how you solved it.

Finding the version of command-line tools is a pain because there is no universal option to do so. Often "--version" works, but not always. And even then, --version sometimes produces a lot of extraneous output.

But I would think (in total ignorance of how Little Snitch works) that Little Snitch could notice that the "openvpn" program that is being blocked has the same name as a different program named "openvpn" that is not being blocked and let you know that, so you would have an idea about what's happening.

If you can use wildcards in the rules for Little Snitch (maybe write the rules manually?), maybe you could unblock (or allow, or whatever they call it)  "/Applications/Tunnelblick.app/Contents/Resources/openvpn/*/openvpn". That way every future update of OpenVPN won't cause the same problem.

That is, until I put the OpenVPN binaries where macOS wants them to be (outside of Resources) -- but I can't do that unless I rename the OpenVPN binaries to be something like "openvpn-2.3.12", which is a pain. (They should be in Contents/MacOS, but you can't have a directory structure there, so binaries need unique names. AFAICT, the only problem with having them in Resources is that they are signed and then Resources is sealed, which is sort of double-protecting them, which Apple says is inefficient.)

Earlier you mentioned about the "Default" OpenVPN. Tunnelblick itself doesn't use it. It is there so it can be used by shell programs so they don't need to know the exact version of OpenvPN that is included in a particular version of Tunnelblick.