WU Updates and DNS

[G Hammer]

I recently regraded my LAN and in doing so forgot to have my WeeWX machine use external DNS. It was pointed to my pihole DNS servers.

Because of that, I see that the WU update is not very well behaved.

It seems to call for a new DNS lookup every time it fires.

With rapid fire on, that is 85000+ DNS calls daily!

rtupdate.wunderground.com 85026
wx.lan 87934

That number of lookups raised warnings on my pihole machines, they want more resources to handle the high volume of calls.

I reconfigured the WeeWX machine to use external DNS, but the issue is just being pushed elsewhere.

Nothing on my network makes that many DNS lookup calls, not even close.

Every OS I'm aware of has a DNS cache, WeeWX doesn't seem to use it though.

However, WU has a TTL of only 60 seconds, but that is still around 24X less.

Any thoughts?

[Joel Bion]

Should WeeWX, or any application, be aware of how DNS gets the address, including the use of a cache? I tend to think “no” as that breaks “layering” (leaving the network details out of the application, in this case)

Usually you get the operating system’s networking stack, or DNS supporting application, to do dns caching for you. For example, using unbound, or dnsmasq or if you use systemd, there’s a resolver with that, too. 

Are you running one?

Sent from my iPhone

On Jan 4, 2024, at 6:30 AM, G Hammer <gary....@gmail.com> wrote:

I recently regraded my LAN and in doing so forgot to have my WeeWX machine use external DNS. It was pointed to my pihole DNS servers.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/6cdab24d-9998-4e39-b29a-d69a430ed45an%40googlegroups.com.

[Joel Bion]

To make my point more precisely, unless weewx is somehow explicitly indicating it wants a fresh, not cached, entry (can an app even do that?!) I don’t see how weewx can be at fault here. Does your weewx-running machine run a local caching DNS resolver? Have you verified its use and configuration using tools like “dig”?

Sent from my iPhone

On Jan 4, 2024, at 8:06 AM, Joel Bion <jpb...@gmail.com> wrote:

Should WeeWX, or any application, be aware of how DNS gets the address, including the use of a cache? I tend to think “no” as that breaks “layering” (leaving the network details out of the application, in this case)

[rcst...@gmail.com]

Short of weewx including a dnscacher, which would only solve the issue for weewx on a computer, not everything else, not sure what can be done. Maybe include a note that users may want to install dnsmasq on their weewx box if they use rapidfire? At least then the traffic is all local to the box…

 

Rightly or wrongly, DNS has become a huge underpinning of nearly everything on modern OSes and software. There’s a reason I have this image hanging on my wall in my office

https://images.app.goo.gl/TzCnAXacUgBRk5LDA

 

-Ryan Stasel

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CA514115-415E-46DC-A543-A63D38E52838%40gmail.com.

[Greg Troxel]

I agree with Ryan and Joel. It is normal to call gethostbyname when you
need it, and if that's an issue the user should set up a caching
resolver. It is a likely source of bugs and a definite source of
complexity to have another caching layer in the daemon.

Besides, WU should support mqtt with a long-lived connection. Someone
who uses it should file a bug report with them :-)

[rcst...@gmail.com]

Fwiw, I had the same response when my Meteobridge was talking to Pihole...
it just hit DNS like crazy. Thankfully it's just DNS, and that's what DNS
servers are meant to do. You can certainly install dnsmasq on whatever you
have weewx running on and let it handle that load, but as Greg mentioned,
you might end up with oddities. The "cloud" has made this all infinitely
more complicated because of super low TTLs, DNS round robin, location based
resolution, etc. So if you're adding caching, you may end up in the
situation where WU takes an endpoint down and your rapidfires stop updating
until the various caches all catch up.

My suggestion... just exclude those entries in pihole so they don't screw up
the graph scaling (that was my issue with Meteobridge, it just made every
other client queries counter impossible to judge).

Good luck!

-----Original Message-----
From: Greg Troxel <g...@lexort.com>
Sent: Thursday, January 4, 2024 8:53 AM
To: rcst...@gmail.com
Cc: 'Joel Bion' <jpb...@gmail.com>; 'G Hammer' <gary....@gmail.com>;
'weewx-development' <weewx-de...@googlegroups.com>
Subject: Re: [weewx-development] WU Updates and DNS

[G Hammer]

I'm staying with my 'send it outside for DNS resolution'.

I'm certain that pihole is doing caching, but it still needs to respond to the query.

I've just never seen pihole raise an issue that there were too many requests and that it wanted more resources. Albeit the difference was instead of 2 cores, it wanted 2.8 cores <g>.

[rcst...@gmail.com]

Yes. And yeah, pihole is running dnsmasq itself, and just adding entries for known problems to 127.0.0.1.

 

And no, I’ve never seen pihole raise a “load too high” message or anything. I’ve got it running on same raspberry pi my weewx runs on… chugs along every day, all day, no issues. One of these days I’ll move pihole over to docker… maybe. =)

--

You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/e3f4883b-aa69-4a4d-b223-28c599353a11n%40googlegroups.com.

[Vince Skahan]

Your pihole is misconfigured if it is out of resources with that few a number of queries.

If WU sets their TTL short, all consumers are going to query often because nobody's caching nameserver (pihole, local, google, or anybody else) will have a valid cache.  That is just how DNS works.  Things are working 'exactly' as WU wants them to work.  They 'want' you to query often.  Do a "dig -q=ANY" and you'll see how WU is currently set up on their end:

• three actual AWS systems speaking ipv4
• behind an AWS load balancer for both ipv4 and ipv6 queries
• and both the rapidfire and non-rapidfire WU hostnames point to the load balancer
• with 60 second TTL on the actual computers they load balance across

If you check your pihole logs, you'll see why you do so many queries per day:

• each query causes 'two' lookups, one for A records, one for AAAA records
• so doing the math you get the essentially one call/second average you are seeing

Go to your pihole and look at long term data => top lists and tell us how many lookups it reports to WU yesterday.

• I did 576 such lookups here with rapidfire 'off', all resolved via google DNS
• For the last month the math works out the same per day so checking yesterday is good enough as a check

As I think Greg suggested, you 'could' run a caching nameserver on your weewx system to only look up when the caches expire, but I wouldn't bother.

[Vince Skahan]

I might add that I've been running 'dig' a lot on my system this morning while constructing my previous answer, and I 'do' see my pihole caching.   Many answers are from cache.  When the TTL has expired it looks up to google.  So it's working as expected.  Again, if your pihole is out of compute I'd suggest your pihole is misconfigured somehow.

(I'm running mine in docker on a several year old i3 NUC that is basically at zero load average)

[G Hammer]

I am happy enough with sending the DNS lookups external.

Saves configuring pihole to not log/ignore those calls.

If misconfigured, it ran for literally years with no warnings until the infernal WU lookups were pointed it way.

Seems happy enough now.

[Joel Bion]

Further thoughts:

When I think of even the more complexly configured servers, they may support email, web services, things like that. This means they are, for network use purposes, frequently responsive to a remote request vs. primarily generative of traffic for their own reasons.Many data generative applications that maintain a regular stream of data to a remote host open that stream only once - and on reconnect if needed, for any reason.

weather data sensors and collection applications that feed to weather data storage sites are data generators for this discussion’s purposes and neither client nor server distributions are optimized for them. 

Further, to save resources on a sensor, and on the machine receiving the sensory data, rarely is any lasting connection (in an TCP/UDP way) established. In fact, in some, like weather underground, the authentication credentials and the sensory payload are given in a single packet, using HTTP GET or whichever. Single packet updates.

Pihole and other tools that provide recursive DNS resolvers are typically done in external boxes - not installed in typical “server” installations. This is usually great, but isn’t so great for the data generator use case.

to avoid hammering other devices, one of a couple things are done on the data generator box, all in the spirit of “fix the problem as close to the creator of the problem as possible”

A) a recursive DNS server is placed resident on the data generating box - this fixes the problem at the box level but adds complexity for end users. Tools like metro bridge/meteohub where a turnkey box+utility run should probably do this. 

B) the application generating the traffic is built to try to be a gentle user of DNS by doing its own caching. 

I’m a big fan of (A) because writing a barely functional DNS resolver for single-application use is easy but writing a really good one requires a lot of subtle domain-specific (pun intended) knowledge. And there are excellent examples like unbound+getdns+stubby (and others!) that do it so well.

Joel

Sent from my iPhone

On Jan 4, 2024, at 10:14 AM, G Hammer <gary....@gmail.com> wrote:

I am happy enough with sending the DNS lookups external.

Saves configuring pihole to not log/ignore those calls.

If misconfigured, it ran for literally years with no warnings until the infernal WU lookups were pointed it way.

Seems happy enough now.

<pihole.png>

On Thursday, January 4, 2024 at 1:04:57 PM UTC-5 Vince Skahan wrote:

I might add that I've been running 'dig' a lot on my system this morning while constructing my previous answer, and I 'do' see my pihole caching.   Many answers are from cache.  When the TTL has expired it looks up to google.  So it's working as expected.  Again, if your pihole is out of compute I'd suggest your pihole is misconfigured somehow.

(I'm running mine in docker on a several year old i3 NUC that is basically at zero load average)

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/1ad89bff-c8fe-4f37-a4f5-6ca4213fc322n%40googlegroups.com.

<pihole.png>

[G Hammer]

As a happy end, I installed unbound on the WeeWX machine with an added local zone to allow the machine to see a couple of local targets needed for backup and NTP.

pihole is happy and the stats are relevant now without all the WeeWX lookups.

[Joel Bion]

Great news! :)

Sent from my iPhone

On Jan 9, 2024, at 6:29 AM, G Hammer <gary....@gmail.com> wrote:

As a happy end, I installed unbound on the WeeWX machine with an added local zone to allow the machine to see a couple of local targets needed for backup and NTP.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/6e63aabf-9451-4950-8878-0b11c8f1486an%40googlegroups.com.