DNS resolution using pushed DNS-server doesn't work after Mac OS Ventura update

[Tatu Teräväinen]

Hi,

I just upgraded to Mac OS Ventura (13.0) and after that DNS resolution through the VPN connection doesn't work anymore.

Tunnelblick seems to successfully set the DNS servers pushed by the OpenVPN server, but Mac doesn't use those servers when resolving domains. My theory is that either DNS-cache flushing fails somehow when setting the DNS servers (trying to manually flush didn't work either though..) or the OS somehow discards our local DNS server (e.g. maybe if it tries to resolve something through the primary DNS-server before the connection is complete and decides it doesn't work and decides to ignore it after that and use the fallback (8.8.8.8) pushed after that - a bit like ubuntu's systemd-resolved behaves). 

Did some tests in terminal after connecting to VPN:

dig someserver.example.com -> queries the dns server via VPN and finds correct IP, works as expected
scutil —dns -> resolver#1 lists correct DNS servers set by the VPN connection
ping someserver.example.com -> ping: cannot resolve <<exampledomain>>: Unknown host

I tried to flush dns caches after connecting to VPN, but no success:
sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

Couldn't find much information about Ventura update, but atleast System Settings -screen has changed quite a bit so maybe they messed with networking internals as well.

Anyone have any ideas I could try out?

Br,

Tatu

scutil --dns output:

DNS configuration

resolver #1

  search domain[0] : openvpn

  nameserver[0] : 172.30.0.3

  nameserver[1] : 8.8.8.8

  flags    : Request A records

  reach    : 0x00000002 (Reachable)

Some logs below from the connection:

13:22:01 *Tunnelblick:  DNS servers '172.30.0.3 8.8.8.8' were set manually

                           13:22:01 *Tunnelblick:  DNS servers '172.30.0.3 8.8.8.8' will be used for DNS queries when the VPN is active

                           13:22:01 *Tunnelblick:  NOTE: The DNS servers include one or more free public DNS servers known to Tunnelblick and one or more DNS servers not known to Tunnelblick. If used, the DNS servers not known to Tunnelblick may cause DNS queries to fail or be intercepted or falsified even if they are directed through the VPN. Specify only known public DNS servers or DNS servers located on the VPN network to avoid such problems.

                           13:22:01 *Tunnelblick:  Flushed the DNS cache via dscacheutil

                           13:22:01 *Tunnelblick:  /usr/sbin/discoveryutil not present. Not flushing the DNS cache via discoveryutil

                           13:22:01 *Tunnelblick:  Notified mDNSResponder that the DNS cache was flushed

                           13:22:01 *Tunnelblick:  Not notifying mDNSResponderHelper that the DNS cache was flushed because it is not running

                           13:22:01 *Tunnelblick:  Setting up to monitor system configuration with process-network-changes

                           13:22:01 *Tunnelblick:  End of output from client.up.tunnelblick.sh

                           13:22:01 *Tunnelblick:  **********************************************

2022-11-03 13:22:01.321974 Initialization Sequence Completed

2022-11-03 13:22:01.321995 MANAGEMENT: >STATE:1667474521,CONNECTED,SUCCESS,192.168.255.6,85.156.139.198,1194,,

2022-11-03 13:22:02.551541 *Tunnelblick: DNS address 172.30.0.3 is being routed through the VPN

2022-11-03 13:22:02.663507 *Tunnelblick: DNS address 8.8.8.8 is being routed through the VPN

[Svet Bajlekov]

Hi Tatu,

Myself and a colleague also experienced something very similar after upgrading to Ventura. I actually signed up to this mailing list just to post about it...

In my case DNS resolution would work as expected for a while, but after some time (e.g. half an hour) it would cease working. And similarly, resolving with dig would still work, but ping and similar (e.g. trying to load websites in the browser) wouldn't work - it would just hang.

Then a couple of days after the upgrade the problem went away and I haven't experienced it since, hence why I didn't end up sharing. But I suspect it will come back, since I didn't change anything specific. Anyway, just wanted to let you know you're not the only one.

Cheers,

Svet

[Tatu Teräväinen]

Hi Svet,

Thanks for letting me know, that sounds like a similar issue indeed. Hopefully there's some fix in the pipeline.

Fyi: Apparently testing DNS with ping was a bad idea (https://groups.google.com/g/tunnelblick-discuss/c/7GaWje4EtyM/m/dP-ytzTZAwAJ). Anyway, I experience the same trouble via Safari as well, like you guys.

Br,

Tatu

[david jimenez]

Hi folks,

in my company we are suffering same issue. For some reason for some employees the dns works fine but for others after a while stopped working and force you to restart tunnelblick. The issue started after upgrading to Ventura last week. I hope someone can find a solution.

Best regards

David

[Doug Horner]

Interestingly, when the DNS stops resolving nslookup still works but other resolvers do not.

Do doing nslookup aaaaaa.com will work but all other apps fail.  I did a tcpdump and the odd thing is the Ethernet addresses do not show in the tcpdump for the lines that are working.

tcpdump -e -nn host 172.27.233.123

20:28:02.070938 18:3e:ef:e0:b0:67 > d4:e2:cb:dc:15:6a, ethertype IPv4 (0x0800), length 87: 10.0.0.78.59858 > 172.27.233.123.53: 32056+ PTR? lb._dns-sd._udp.med-web.com. (45)
20:28:02.070989 18:3e:ef:e0:b0:67 > d4:e2:cb:dc:15:6a, ethertype IPv4 (0x0800), length 82: 10.0.0.78.52119 > 172.27.233.123.53: 41956+ PTR? 78.0.0.10.in-addr.arpa. (40)
20:28:02.071082 18:3e:ef:e0:b0:67 > d4:e2:cb:dc:15:6a, ethertype IPv4 (0x0800), length 78: 10.0.0.78.64726 > 172.27.233.123.53: 37540+ Type65? dev.wikigdrive.com. (36)
20:28:02.071140 18:3e:ef:e0:b0:67 > d4:e2:cb:dc:15:6a, ethertype IPv4 (0x0800), length 78: 10.0.0.78.62608 > 172.27.233.123.53: 45955+ A? dev.wikigdrive.com. (36)
20:28:02.071209 18:3e:ef:e0:b0:67 > d4:e2:cb:dc:15:6a, ethertype IPv4 (0x0800), length 75: 10.0.0.78.53187 > 172.27.233.123.53: 44019+ Type65? docs.google.com. (33)
20:28:02.071730 18:3e:ef:e0:b0:67 > d4:e2:cb:dc:15:6a, ethertype IPv4 (0x0800), length 75: 10.0.0.78.55214 > 172.27.233.123.53: 28437+ A? docs.google.com. (33)
20:28:02.249234 AF IPv4 (2), length 61: 192.168.242.200.63947 > 172.27.233.123.53: 18569+ A? aaaaaaa.com. (29)
20:28:02.276617 AF IPv4 (2), length 537: 172.27.233.123.53 > 192.168.242.200.63947: 18569 1/13/14 A 213.186.33.5 (505)
20:28:02.756041 18:3e:ef:e0:b0:67 > d4:e2:cb:dc:15:6a, ethertype IPv4 (0x0800), length 81: 10.0.0.78.50988 > 172.27.233.123.53: 41139+ A? wss-primary.slack.com. (39)
20:28:07.575126 AF IPv4 (2), length 65: 192.168.242.200.64681 > 172.27.233.123.53: 63259+ A? aaaaaaaaaaa.com. (33)
20:28:08.109521 AF IPv4 (2), length 541: 172.27.233.123.53 > 192.168.242.200.64681: 63259 1/13/14 A 64.90.54.20 (509)
20:28:16.970922 18:3e:ef:e0:b0:67 > d4:e2:cb:dc:15:6a, ethertype IPv4 (0x0800), length 76: 10.0.0.78.64382 > 172.27.233.123.53: 23157+ Type65? drive.google.com. (34)
20:28:16.971031 18:3e:ef:e0:b0:67 > d4:e2:cb:dc:15:6a, ethertype IPv4 (0x0800), length 76: 10.0.0.78.55844 > 172.27.233.123.53: 17590+ A? drive.google.com. (34)
20:28:16.971081 18:3e:ef:e0:b0:67 > d4:e2:cb:dc:15:6a, ethertype IPv4 (0x0800), length 82: 10.0.0.78.60694 > 172.27.233.123.53: 22213+ A? p42-content.icloud.com. (40)
20:28:19.543067 18:3e:ef:e0:b0:67 > d4:e2:cb:dc:15:6a, ethertype IPv4 (0x0800), length 81: 10.0.0.78.50988 > 172.27.233.123.53: 41139+ A? wss-primary.slack.com. (39)

I also sniffed from the DNS server and it does not even see the packets.  But it does for nslookup commands.

[Doug Horner]

d4:e2:cb:dc:15:6a is the default router, so it’s as if the routing for the dns is not using the routing table.  I can ping the DNS server by IP but the DNS request tacks on the Ethernet and tries to send it to the default router rather than send it via the tunnel.

[Svet Bajlekov]

Thanks for sharing that Doug, sounds like an interesting finding!

After a week or two of not facing this issue, it occurred again earlier today. I tried to troubleshoot - e.g. checking the logs in the Console app for anything funny - but didn't get anywhere. I didn't do a packet dump though.

The slightly weird thing this time was that simply reconnecting the Tunnelblick VPN didn't fix it (as I believe it did previously), and neither did a more general network restart. I had to completely close and reopen many of the apps that were affected by it (Safari, Outlook) for them to have working DNS resolution again. Not sure why...maybe because I let the problem linger for a while this time - while troubleshooting - the apps ended up giving up on DNS resolution more generally. Just figured I'd share in case this provides any further hints as to what's causing this.

Best,

Svet

--
You received this message because you are subscribed to a topic in the Google Groups "tunnelblick-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tunnelblick-discuss/CpusBhU7Ob8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tunnelblick-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tunnelblick-discuss/53ac037c-ed1e-4c0c-9978-6033b79e13e8n%40googlegroups.com.

[Enrico]

Hi, same problem for me with Ventura.

I made a test with viscosity client, same problem after some time dns do no work properly.

Example, nslookpu resolve my internal name but traceroute give host not found.

[NUC Skylaker]

https://www.reddit.com/r/MacOSBeta/comments/wau5l5/macos_ventura_public_beta_2_safari_dns_question/ - here you can find the solution.

[Svet Bajlekov]

Thanks for the pointer!

Not sure that's the solution though. If I'm not mistaken, the upshot of that thread is summarized by these two comments:

> It looks like Ventura is skipping non DNSSEC responses if there are DNSSEC servers on the list. If you remove 8.8.8.8 and 8.8.4.4 it should start working.

and 

> Thanks!! Only thing I could find on this. This helped me resolve an issue that was driving me crazy. We have a small number of Mac users at my company, one user upgraded and could no longer reach our internal sites. We also had 8.8.8.8 as tertiary DNS, once I removed that it worked flawlessly. It was definitely skipping over our local servers.

In our case at least, there are no public DNS servers set as secondary/tertiary. Just our private server, 172.20.142.5. This is confirmed in the Tunnelblick connection log:

> Changed DNS ServerAddresses setting from '8.8.8.8 8.8.4.4' to '172.20.142.5'

> DNS servers '172.20.142.5' will be used for DNS queries when the VPN is active

and also by checking what servers are active after connection (just that private one).

Maybe there's some sort of glitch related to what's described in the Reddit thread, but I don't think it's that specifically. The theory around it being a routing issue seems more plausible (but not sure how to fix...haha).

Svet

To view this discussion on the web visit https://groups.google.com/d/msgid/tunnelblick-discuss/28287675-c7b5-4d6d-be5e-6df39ff94aefn%40googlegroups.com.

[Ryan Williams]

In my configuration the only thing that I've found to avoid the issue is to manually set my DNS Server settings to list the VPN's DNS server first and then my home network's DNS server second - both manually set and not dynamic.  Once I did that my stalled connections all came back to life and DNS still works after I disconnect from the VPN since my normal servers are listed second

[Tatu Teräväinen]

Thanks for the replies everyone!

Indeed, for me removing 8.8.8.8 from the pushed DNS servers seemed to do the trick. Now that VPN-server pushes only 172.x.x.x DNS queries are resolved using the pushed server and everything works (at least for now). In my local network (VPN off) I don't use public 8.8.8.8 so maybe that's the difference to Svets situation? My local DNS servers support DNSSec but don't use DNS-over-TLS, so I'm guessing Ventura prefers any DNS-server that uses DNS-over-TLS when available / listed in the settings. 

Actually now that I tried to google Ventura + DNS-over-TLS I found this: https://twitter.com/jacopoj_/status/1586390728761933825?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E1586390728761933825%7Ctwgr%5E027fef3b8b0ca39f6d9218e173db33c2dd1b7d86%7Ctwcon%5Es1_&ref_url=https%3A%2F%2Fwww.redditmedia.com%2Fmediaembed%2Fygpaj8%3Fresponsive%3Dtrueis_nightmode%3Dfalse

-Tatu

To view this discussion on the web visit https://groups.google.com/d/msgid/tunnelblick-discuss/1b195a70-d4a4-4a2e-8c28-c93a1cff59e0n%40googlegroups.com.

[Carlos Luis Cubas Morera]

Same issue here. We haven´t found any workaorund for that.

OpenVPN Cliente had similar issue, and they have fix It, in the his last release:

https://forums.openvpn.net/viewtopic.php?t=35018

At the moment, we are migrating our users to OpenVPN Client when they experience this issue.

[Tunnelblick developer]

Anyone with this problem should make sure that Tunnelblick's "Monitor network settings" is set, and that the settings on the "While Connected" tab of Tunnelblick's "Advanced" settings window are set appropriately. (The defaults are to restore settings when a value changes to the pre-VPN value, or to restart the connection when a value changes to anything else.)

[Svet Bajlekov]

On Wednesday, December 14, 2022 at 12:38:19 PM UTC+1 Tunnelblick developer wrote:

Anyone with this problem should make sure that Tunnelblick's "Monitor network settings" is set, and that the settings on the "While Connected" tab of Tunnelblick's "Advanced" settings window are set appropriately. (The defaults are to restore settings when a value changes to the pre-VPN value, or to restart the connection when a value changes to anything else.)

Thanks! Just to confirm, I believe those were indeed the settings I was using when experiencing the issue (if I'm not mistaken they are the defaults?). The issue didn't appear to be that my DNS server was being reset/switched; it looked more like a subtle connection/routing issue that exactly matched what Doug Horner mentioned in a previous email.

FWIW, I recently opted for the workaround that someone else mentioned, which is to not set DNS via Tunnelblick, but to hardcode the servers in the OS Network settings - with the primary server being the VPN's DNS (always the same IP in my case), and the secondary/tertiary being my router, public DNS, etc. Not an ideal solution (and probably not viable in many cases) but it has worked surprisingly well in my case.

Svet

[Brian Reid]

We have a similar issue happening...only on Ventura and intermittantly.  As a band-aid we've found that changing the ipv6 setting to local only or back again seems to fix the DNS issue temporarily.

[Ash McConnell]

Same problem for me too, I tried using OpenVPN Client Connect, but VPN DNS didn't seem to work at all (using the same config as tunnelblick)

[Doug Horner]

It looks like mDNSResponder somehow stops sending DNS queries using the routing table.   When I can not resolve hosts, tcpdump shows DNS queries going out on the wifi with the mac address of the wifi default router instead of the tun interface!  172.16.1.12 is my DNS server and my internet router is just going to drop them.  The routing table show 172.16.0.0/16 should go thru the tun interface, but its not!  mDNSresponder is somehow ignoring the system routes! Anyone know the best way to relay this back to Apple?  

[Tunnelblick developer]

Does anyone have this problem when their configuration file includes persist-tun?

[Drew Gulino]

yes, we are having this issue with persist-tun in our config.

[Tunnelblick developer]

Everyone: try removing "persist-tun" from your OpenVPN configuration file and see if that helps.

[Trey Ethridge]

I tried removing "persist-tun" and it didn't seem to have any effect.  I still received the DNS error dialog and had DNS issues.

[Tunnelblick developer]

Trey,  does your initial connection work and the error happens on reconnect, or does the initial connection fail with the error?

[Trey Ethridge]

It usually works for a period of time after I restart the laptop.  I'm using a macbook pro with an m1 chip.  Then it gets into a state where I get the DNS error dialog every time until I restart the laptop again.  I have been able to get it to work by restarting Tunnelblk on occasion, but that doesn't always work.

--

You received this message because you are subscribed to a topic in the Google Groups "tunnelblick-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tunnelblick-discuss/CpusBhU7Ob8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tunnelblick-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tunnelblick-discuss/daf2126a-086e-4284-9cde-b94c17870c7fn%40googlegroups.com.

[Sam Weaver]

I troubleshooted this a bit with Trey today, I help maintain the VPN he was connecting to.

The DNS dialog box "DNS does not appear to be working" is a red herring- we use private DNS servers that are not routed over the VPN, that dialog box is expected.

The behavior we're seeing is strangeness, one of either:

(a) DNS servers being set to some unexpected result, OR
(b) DNS queries failing in some contexts (system-level DNS) while succeeding in other contexts (Chrome seems to resolve just fine) while connected to VPN

- Sam

[Tunnelblick developer]

Thanks, Sam. Please be aware of another thread on this subject, at 

     https://github.com/Tunnelblick/Tunnelblick/issues/736

On Monday, January 30, 2023 at 3:21:36 PM UTC-5 Sam Weaver wrote:

The DNS dialog box "DNS does not appear to be working" is a red herring- we use private DNS servers that are not routed over the VPN, that dialog box is expected.

What the message means is that Tunnelblick was unable to reach tunnelblick.net after the VPN was established even though it could reach tunnelblick.net before the VPN was established. Is your private DNS server not resolving tunnelblick.net?

The behavior we're seeing is strangeness, one of either:

(a) DNS servers being set to some unexpected result, OR
(b) DNS queries failing in some contexts (system-level DNS) while succeeding in other contexts (Chrome seems to resolve just fine) while connected to VPN

I suspect that Chrome bypasses the OS-level DNS system by sending encrypted DNS queries to Google's DNS servers.

[Yousif Hassan]

My organization has been using Tunnelblick for 10 years and we love it.  It's really very frustrating to see what problems Apple has caused with Ventura.  That said, I have switched all my users on Ventura to OpenVPN Client Connect and things are back to normal.  Their thread suggests there is a watchdog function they have added to restore DNS settings when something changes.  This sounds a lot like what Tunnelblick can do with the "monitor network settings" function.  I am holding out hope that Tunnelblick can also resolve this issue, as Tunnelblick is superior in most ways to OpenVPN Client Connect.  With the OpenVPN Client Connect app, you can't edit profiles inline, you can't connect to more than one VPN at a time, and it does not support Layer 2 tunnels at all.

Thank you for your hard work on Tunnelblick throughout the years.

[Yousif Hassan]

I have found a stable solution that works for my company with Tunnelblick on Ventura/M1 and wished to share it.

We reverted back to "Set nameserver (3.0b10)" for "Set DNS/WINS" in VPN settings.  This allows both "split DNS" scenarios as well as additional pushed search domains to work correctly.  It required that we modify the OpenVPN CLIENT configuration file to remove a few keywords that caused TB to reject the file when this "set nameserver (3.0b10)" option was set: mssfix, fragment, and for some reason pull.  Those were accepted when the "set nameserver" option was used, but not when set "nameserver (3.0b10)" was used.

We use pfSense on the server side, but the upshot is that it's version 2 not 3, so it's sending "push dhcp-option DOMAIN-SEARCH" and "push dhcp-option DOMAIN" and things appear to work from a DNS perspective.

This is definitely a change in Ventura's handling of DNS, but TB's older nameserver handling method works and scutil --dns verifies that everything is set correctly and stays that way.

I hope this helped someone.

[Tunnelblick developer]

Thanks, Yousiff! Very interesting report.

Can you elaborate on the problem with misfit, fragment, and pull being in the configuration file? For example, was Tunnelblick rejecting the keywords when you tried to install the configuration containing them, or when you tried to connect the VPN? Can you provide screenshots or text of the error message that appeared? (Email privately to devel...@tunnelblick.net if you prefer.)

[Tomek Lutelmowski]

The same issue here with Fortinet VPN Client (7.2) on Ventura. After connecting to VPN, at random (minutes after connection), the resolv.conf file is changed by OS to remove VPN DNS servers and put back previous ones. 

[Mike Michel]

You can disable that mac changes the dns settings back to the original in advanced settings (ignore dns-server changes after connection) but it won't help either. My VPN connection is fine,  resolv.conf is ok, I can curl internal stuff using the IP and scutil --dns shows also the vpn dns servers BUT dns is just not working anymore. I have to restart the vpn connection and then I am fine for a while again.

[David Japaridze]

It seems that there is an issue with DNSSEC, so if you have any DNS server in the list, that supports  Domain Name System Security Extensions, other DNS servers, regardless of their order will be ignored.

you can find response here

It looks like Ventura is skipping non DNSSEC responses if there are DNSSEC servers on the list. If you remove 8.8.8.8 and 8.8.4.4 it should start working.

Hope it will help, at least it fixed problem for me.

---

აღნიშნული მეილი და თანდართული დოკუმენტ(ებ)ი შეიძლება შეიცავდეს კონფიდენციალურ ინფორმაციას და გამიზნულია მხოლოდ დასახელებული ადრესატ(ებ)ის საყურადღებოდ. იმ შემთხვევაში, თუ აღნიშნული ინფორმაცია თქვენთვის არ არის განკუთვნილი, გთხოვთ დაუყოვნებლივ აცნობოთ გამომგზავნს, უკან გაუგზავნოთ მეილი და წაშალოთ აღნიშნული ინფორმაცია თქვენი სისტემიდან.
The information  (including its attachments) in this email is confidential and may have legal privileges. We intend it only for the use of the individual or entity we've addressed the communication to. If you have received this email by mistake please delete it and notify the sender.