Best practices when a service starts receiving bad data

[bell...@gmail.com]

Suppose a WeeWX service is augmenting loop packets. After some time passes, it starts to receive bad data, logs this information, and ignores the bad data. If the service continues to receive bad data it will ‘flood’ the log with these error messages. I see a few possible ways to stop this.

1. Throttle the logging. For example, after X errors only log every Nth message.
2. Have the service shut itself down. This would probably require accessing the engine’s internal callbacks data structure or an enhancement to allow ‘unbinding’ to events.
3. Raise an exception, causing WeeWX to terminate.
   At first glance 3 seems a bit severe. But if this data is critical, it could make sense to stop WeeWX. Option 2 intrigues me if the problem is due to the computer being stressed (CPU, RAM, network, etc.). Stopping the service might help determine if the service is the cause or not. Option 1 seems the simplest and leaves the possibility that good data will start arriving and everything will ‘self correct’ and is the option I am leaning towards.

Thoughts?
Thanks. - rich

[Vince Skahan]

4.  Validate your input to prevent accepting garbage in completely in your service.  That's separate from logging or not.

1a. Make your logging do something like how networking does exponential backoff so you get 'some' indications but if things run really amok you don't flood your logs.   Log it if you see N messages.   Back off til you see N*2, then N*4, etc. unless you have some definable quiet period to let logging reset its settings.   Easier said than done I guess :-)

5. Use cron or logwatch or equivalent to look for those messages in your syslog and perhaps trigger some command to tell your service to back off on its logging.

[Peter Fletcher]

Vince's 1a is good, and not particularly difficult to implement. What I also do in some of my household control services (though I haven't felt the need to do it for anything in weewx) is send myself an email for serious problems. Depending on the severity of the problem, I either send a single email message when it first occurs or send messages at specified intervals until the problem is fixed.

[bell...@gmail.com]

Vince and Peter,

Thanks for your thoughts! I will be working on some type of logging throttle.

rich