NxCloud 3.4.8 - DNS Timeout

[Josh]

Hi,

I upgraded to 3.4.8 recently and out of the blue today, my server stopped responding to DNS lookups. I can run 'nslookup google.com [myserverip]' and the lookup will time out.

I checked the error logs and don't see anything. Looking at a TCP dump of the traffic into my server I can see clients requesting DNS lookups but my server never replies to them or even does a DNS lookup to the upstream DNS.

Any thoughts on how I can troubleshoot this?

Thanks in advance,

Josh

[Jinhee]

Did you restart it? Probably you lost the connection between your NxFilter and upstream DNS sever. If you setup alter mail you should get alerted about that. Otherwise send me your log file. support @ nxfilter.org

[Josh]

Yes, I did try restarting. I also checked and I can query my upstream DNS providers from my server.

After restarting I can use NxCloud for around 1min before it stops responding. Eventually I start getting queue full emails; however, I am not sure why as I can query my upstream DNS server while this is happening. I also tried switching my upstream DNS to another resolver and that also did not solve the problem.

[Jinhee]

When you do the connection check to your upstream server, did you do it on your NxFilter machine? That 'Queue full' error only  happens when it can't get any response from its upstream server. Which upstream server do you use? If you don't have many users going with Google public DNS server is OK. When you have bigger number of users you could use a local recursive server, something like Deadwood server.

From our tutorial,

Using local recursive DNS server

One of the possible cause of performance degrading for NxFilter would be the latency to its upstream server. This is not the case when you have just several hundred users as NxFilter has its own caching. But if you have several thousand users this could be an issue. So we added local recursive DNS option.

However, this doesn't mean that NxFilter does recursive DNS query by itself. Rather you install a recursive DNS server into the server having NxFilter already installed and make NxFilter to use the recursive DNS server as its upstream DNS server. If you install something like MaraDNS's Deadwood recursive DNS server and set it to listen to UDP/10053 on '127.0.0.1' then you add the following line into '/nxfilter/conf/cfg.properties' file.

    local_resolver_port = 10053

And then restart NxFilter.

[Josh]

Yes, I queried my upstream DNS from my NxCloud server.

I was initially using Google Public DNS but I switched to OpenDNS and another private DNS resolver to eliminate that issue. 

I could try setting up a local DNS resolver however I am a little sceptical as to if that will resolve the issue seeing as my upstream DNS appears to be working fine.

[Jinhee]

So you were using Google DNS at the time? And restarting didn't fix your problem. This means you still have the problem? How did you  test your upstream server? Something like this?

  dig @8.8.8.8 www.google.com

[Josh]

Yes exactly.

I used 'nslookup google.com 8.8.8.8' and 'nslookup google.com 8.8.4.4'.

[Josh]

I tried dig as well and it resolves to 8.8.8.8 fine.

At the moment I am watching tcpdump of port 53. I can see my request enter my server when I use nslookup on my IP; however, no packets ever leave my server. No lookup is ever sent to my upstream DNS and no response is ever sent back to my PC.

[Jinhee]

That's weird. Enable debugging and send me your log file.

[Jinhee]

For how to enable debugging, see this,

  https://groups.google.com/forum/#!searchin/nxfilter200/Before$20you%7Csort:relevance/nxfilter200/iyoNK8vl1_8/Wiw7yGtKCQAJ

[Josh]

Will do.

I also just fired up a new VM. Installed a fresh copy of NxCloud 3.4.8, copied my config.h2.db over to the new machine and the symptoms appeared on the new server after a min or so. I also tried installing a copy of NxCloud 3.4.7 and had the same results on the new server.

At the moment the new server is running 3.4.7 but the issue is still happening so I'll email you a copy of the debug log once I enable it and collect some debug info.

Thaks for your quick responses. I appreciate it!

[Jinhee]

What happens if you don't copy your config then? If you set it up for just testing with your own IP.

[Josh]

I just emailed you the log.

I have not tried that but I will see what happens and let you know.

[Josh]

On a fresh install, everything works fine. Looks like the issue lies somewhere in my config.

Any ideas on how I can proceed from here?

[Josh]

Ok, I am back up and running.

I was not able to resolve this issue so I installed a fresh copy and reset everything back up by hand.

I still have a copy of the config which causes the issue. Would you like me to email it to you so you can do some further testing on your end to see what causes the issue?

[Jinhee]

Maybe there's something changed in your IP association setup or IP based ACL. I found this on your log,

  INFO [01-16 10:17:28] - DnsStats.flush, udp_cnt = 0, acl_drop_cnt = 0, query_cnt = 29, custom_cnt = 5, auth_redi_cnt = 14, auth_drop_cnt = 0

This means that everybody gets redirected to the login page. Maybe you had to test your server with dig or nslookup. Not your upstream server, your NxFilter. I guess it was sending you its own IP address. But when you get that 'Queue full' error log or alert email, that was a real one.

In long term plan, we might go for an integrated recursive DNS server. But the thing is that you still can do that using third party DNS server. So this one is not that urgent.

[Josh]

Yes, the redirection was done on purpose. 

The server was not resolving any requests and I did not want the internet to be down for my customers. So I installed Nginx on a VM and had it load balance port 53 between my server IP and the two public Google IP's. Here is an article on what I did to keep things going:

https://www.nginx.com/blog/load-balancing-dns-traffic-nginx-plus/

However, because of this, the load balancer would occasionally send a few queries through to my server to test if it was working. After I would restart NxCloud it would respond for about one min, and some requests would get through from the load balancer which would be redirected due to it not knowing wich IP it came from.

[Jinhee]

Not so sure about your DNS load balancing. I don't know if there's any benefit. Could be just a redundancy. It's not like NxFilter sends your requests to its upstream server everytime. It has its own cache. Do you see any performance enhancing or stability there? Was it not related to your problem even?

[Josh]

No, I quickly set that up when the NxCloud failed. I just did it as a temporary fix.

Were you able to tell anything else in the debug log about what happened? I can't seem to find anything and it is so bizarre that if I restore from backup the exact same thing happens.

[Jinhee]

No. I can't think of anything special. If you send me your config backup I can test it on my side.

[Josh]

I emailed you a copy of the config.

[Jinhee]

Just tested with your config DB. Since I don't associate my IP address I always get 45.55.114.xxx which is your block IP. It means it works fine. Actually it can't be from config DB unless you have some IP based ACL there.

[Josh]

Hmm, Ok. That is odd. 

I have things running again. I'm not sure what caused this but if I run into it again I'll let you know.

Thanks for all your help Jinhee!