Can this be caused by NxFilter misconfiguration?

[Todor Todorov]

Dear Jinhee,

Recently we are frequently encountering errors like the ones in the attached screenshots. Especially on cert1.png, the sites have nothing in common.

After 1-2 minutes suddenly the error is gone.

Is it possible, due to misconfiguration on our side, NxFilter to return the wrong IP from cache or something?

NxFilter is v4.3.5.4 and the upstream DNS servers we use are 1.1.1.1 and 8.8.8.8, without DoH.

[Jahastech]

Seems like you are blocked on HTTPS. When it gets forwarded or redirected on HTTPS it happenes. Read this,

  http://nxfilter.org/tutorial/i-faq.php#hide-ssl

[Todor Todorov]

Dear Jinhee,

Thank you, but it is not that, if you take a look at the screenshots, the certificate for the domain replying is different than the domain requested, as if NxFilter returned the wrong IP and the browser contacted the wrong vhost.

[Jahastech]

So you are saying that they are not blocked but they make such certificate errors telling that the certificates are from different sites?

Test NxFilter with those sites. See if the response IP for those sites are correct. Or see if they are same with the responses from your upstream servers.

Do you have have any redirection rules?

[Todor Todorov]

Thank you,

Yes, that is correct, certs are from different sites. And no redirection rules.

Looking more carefully into this:

In the first case both sites are hosted on Amazon AWS.

In the second, both on Akamai.

It is possible that my NxFilter DNS cache for them got stale and the returned IPs changed in the meantime.

My settings are:

Response cache size: 100000

Current Size : In-memory = 39,774 / Persistent = 40,724 / Negative = 0

Use Persistent Cache: yes

Use Negative Cache: no

Minimal Responses: no

Minimum Cache TTL: 0

Block Cache TTL: 0

Do you see any obvious errors with the settings?

[Jahastech]

If there's a connection timeout or no response from your upstream server then your persistent cache works. Then there's a possibility to get an old IP for the site. However, a persisten cache also gets refreshed after 24 hours. Still there's a possibility though.

Do you see that a lot? How many users do you have? And how many requests a day?

[Jahastech]

Sorry it's not like we use persistent cache where there's timeout or no response from upstream servers. But we use it on certain conditions. Disable it for the time we will come up with something.

[Todor Todorov]

Hi,

I see that 5-6 times every day on my workstation, i imagine all other users get that with the same frequency.

Users are around 600 with around 2 million requests for 24 hours.

I've just got the same error trying to open access.redhat.com, certificate present was from www.candlepowernyc.com

I immediately began resolving the IP addresses on both, 10 seconds interval:

First resolve:

access.redhat.com is 2.21.248.237 (on Akamai)

www.candlepowernyc.com is 3.226.162.70 (on AWS)

Second resolve:

access.redhat.com is 92.123.12.128 (on Akamai) - changed!

www.candlepowernyc.com is 3.226.162.70 (on AWS)

Third resove:

access.redhat.com is 23.46.168.248 (on Akamai) - changed again! (access.redhat.com opened)

www.candlepowernyc.com is 18.210.160.240 (on AWS) changed!

Disabling Persistent Cache for now, will report back in a day.

[Jahastech]

Persistent Cache feature is so nice. At the moment, I use NxFilter as my DNS server but there's no upstream server for that. I just put some fake one. But I still can browse my every day websites. This means even if there's a nationwide DNS failure your DNS resolution stil works by NxFilter. However, if these CDN sites change these IPs that much frequently and they actually on/off their content service along with that IP address change we will have a real problem with that.

[Jahastech]

I tested those 2 sites for some time. But I get the same IP always.

For access.redhat.com,

;; ANSWER SECTION:
access.redhat.com.      3421    IN      CNAME   access.redhat.com.edgekey.net.
access.redhat.com.edgekey.net. 21137 IN CNAME   e133.b.akamaiedge.net.
e133.b.akamaiedge.net.  19      IN      A       104.74.153.93

For www.candlepowernyc.com,

;; ANSWER SECTION:
www.candlepowernyc.com. 21599   IN      CNAME   aws-http-redirect-alb-830774459.us-east-1.elb.amazonaws.com.
aws-http-redirect-alb-830774459.us-east-1.elb.amazonaws.com. 59 IN A 18.210.160.240
aws-http-redirect-alb-830774459.us-east-1.elb.amazonaws.com. 59 IN A 3.226.162.70

[Todor Todorov]

I agree, that's why i kept it on.

I've disabled Persistent Cache 15 minutes ago, but still status showing Persistent = 40,746.

Should i restart NxFilter to clear it?

[Jahastech]

No. That's the number of Persistent Cache data in your DB. When you disable it, NxFilter just stops using it. But it still keeps the data for later use.

[Todor Todorov]

Thank you.

About the changing IPs, Akamai is Geo CDN, so, for the same hostname, you're receiving their IP in Korea, and I in Europe, but i guess you already know that.

[Jahastech]

Yeah, but what I am saying is that they don't need to stop their service for your changed IP or the old IP. And the IP addresses of candlepowernyc are in my DNS answers. Means those 2 IPs are both valid. Do you have that problem with candlepowernyc?

And as a test, I created a redirection on my NxFilter for access.redhat.com. I use '23.46.168.248' which is your IP. But it works fine. Do you think they really on/off their service along with the IP change in DNS service?

[Jahastech]

Maybe it's not about on/off. Just don't know how their service works. Anyway, it might be related to Persistent Cache.

[Jahastech]

Any update from your testing? If there's a real problem we can fix it by having Persistent Cache only works when there's a network failure.

[blougaville]

Just wanted to let you know that I've been seeing this same behavior for a while now. We are using NxCloud and have noticed this on several of our customers.

Here's the description from a customer:

I keep getting these errors. Sometimes if wait a few minutes and refresh it resolves itself to the correct address after a minute or two, but it is frustrating.

I looked at the problem and they were trying to go to www.cdwg.com but they were getting a certificate error about www.burkert.com. It's always two totally unrelated websites.

After reading this thread, I have disabled persistent cache on our NxCloud and will see if that improves the situation. I have attached a screenshot of our DNS > Config page in NxCloud.

[Jahastech]

Did you test it with nslookup or dig? If you get a wrong address from NxCloud, it would be about a connection problem. If you get a certificate error, it might be from other things such as an embedded site got blocked. So, you need to confirm that with nslookup or dig. However, we can see if your Persistent Cache way solves the problem first.

[blougaville]

I will try to test with nslookup next time I'm on-site and encounter the certificate error myself. The problem is I'm not often on-site and by the time my customers tell me of the problem it has resolved itself.

I seriously doubt it's an embedded site. First, if you go to the sites in the example screenshot I attached, you can see that those two websites have nothing to do with each other. Second, I don't see any blocks being logged from the users that report the problem around the time of the problem. Also, I have only seen this problem with customer networks who are using NxCloud so I really believe the problem has something to do with the DNS caching in NxCloud.

I will definitely report back in a few days how it goes now that I've disabled persistent cache.

Thanks!

[Jahastech]

If it's from Persistent Cache, those site's IPs should? be changed frequently. Might be but we don't have a solid proof. We could make it working only if there's a problem with its upstream DNS server but at the moment, we just leave it as we can't be 100% sure about this one.

[Jahastech]

The common thing is that they are all related to Akamai. Maybe we can exclude these domains from Persistent Cache if we can fix their common characteristics.

[blougaville]

That would be great! Let me know if you want me to try anything. For now, I'll report back in a few days to let you know if my users encounter the same errors with persistent cache turned off.

[Jahastech]

There may be another reason. And Persistent Cache is not related to this problem if that's the case. Persistent Cache only works when there's no in-memory cache. It can't be reoccuring if it's from Persistent Cache. We will try to fix the problem with the next version.

[blougaville]

Just wanted to follow up to say that my customers haven't reported any certificate errors like the ones described in previous posts after I disabled Persistent Cache in NxCloud.

[Jahastech]

OK. If it works without Persistent Cache, that's good. However, if it happens it should happen without Persistent Cache as we use a temporary cache for the expired responses. We can disable this temporary cache but it's also related to performance enhancing so we can't give it up easily. With the latest version of NxFilter, we exclude those frequently updated domains from Temporary Cache though. We will watch it.

[gpdur...@gmail.com]

Hello, I am having the same issue and I am not using NxCloud. This has been occurring for many releases. However for me I wait up to 1 minute and it corrects itself. The very interesting part is that it can occur with the same website over a short time and often this disturbs the connection. When it does I do an nslookup and it returns the correct IP address, it is the cert that shows an issue. Akamai is Microsoft owned and I have trouble with this for varying websites.

I do not know how to turn off persistent cache so if someone can remind me of this within NxFilter, I would be thankful. 

My configuration is and has been that nxfilter points to my internal DNS which is pointing to various DNS with 1.1.1.1 being the most recent within the last few months. Again this has occurred prior to then. I also have root hints enabled so it gets updated. The time on the computers all match and are synchronized with a global NIST NTP server. Yes this is my home network and I really would like to get this solved. It has been over a year I am experiencing this. 

Thank you!

[Jahastech]

You can turn off Persistent Cache on 'DNS > Setup'. Can you get me those domains you have troubles with? If we can recreate it on our side then we cna fix it.

[Jahastech]

Another one to look at is  'Minimum Cache TTL' on 'DNS > Setup'. You can set it to 0 to turn it off.

We don't use Google forum anymore. Try Reddit forum next time,

  https://www.reddit.com/r/nxfilter/

On Wednesday, September 23, 2020 at 11:27:37 PM UTC+9 gpdur...@gmail.com wrote: