Small Python 'daemon' script to check if weewx is running

[Constantine Samaklis]

I thought I would give something back to the group since I have gotten good help to my questions. I have created a Python utility script that can be ran periodically (cron?) to check if weewx is running and if it is not, it will send out an email with a number of lines from the end of the log (/var/log/messages, or whatever log you want) that would show the error that weewx encountered. The script will try to restart weex a number of pre configurable times, in case the error was a fluke rather than a fatal one.

The script will stop trying to send messages after it has reached the indicated restarts times in the configuration times, and you will need to delete the flagFile that has been created. (or write a 0 in it). The script has a configurable log rotation, where you can indicate the log rotation interval (in days) and the days of logs to keep. 

logOutput is the temporary tail of the messages log (or could be the syslog or whatever you specify) that will be emailed as an attachment if weewx is found not to be running.

I hope everyone finds this somewhat useful. If you are somewhat paranoid about your weather station being up 24/7 like me then you probably will. :-)

Questions and comments are welcome.

#configuration file for checking routine

[mail]

fromaddr=us...@gmail.com

toaddr=toU...@gmail.com

msgSubj=weewx process is not running

body=weewx process is not running. Please check the attached log and address the issue

userName=gmailu...@gmail.com

password=yourPwd

smtpServer=smtp.gmail.com:587

[logging]

logDir=/home/user/scripts

logFile=checkWeewx.log

logOutput=tail.log

linesToTail=200

logToTail=/var/log/messages

logRotationDayInterval=1

logDaysToKeep=5

[weewx]

#how many times to try and restart the weewx process

restarts=3

flagLogDir=/home/user/scripts

flagFile=weewxNotRunning.txt

startCommand=sudo /etc/init.d/weewx start

You can launch the script by typing: Python ./checkWeewx.py /pathToConfigFile/checkWeewx.config

The configuration format:

#configuration file for checking routine

[mail]

fromaddr=us...@gmail.com

toaddr=toU...@gmail.com

msgSubj=weewx process is not running

body=weewx process is not running. Please check the attached log and address the issue

userName=gmailu...@gmail.com

password=yourPwd

smtpServer=smtp.gmail.com:587

[logging]

logDir=/home/user/scripts

logFile=checkWeewx.log

logOutput=tail.log

linesToTail=200

logToTail=/var/log/messages

logRotationDayInterval=1

logDaysToKeep=5

[weewx]

#how many times to try and restart the weewx process

restarts=3

flagLogDir=/home/user/scripts

flagFile=weewxNotRunning.txt

startCommand=sudo /etc/init.d/weewx start

[Thomas Keffer]

Thanks, Constantine. I hope people find your script useful.

However, I want to say that if weewx crashes or becomes unresponsive, for any reason, that's a bug and I want to know about it! Run Constantine's script, but send me the log. My goal is to engineer out any glitches, not paper over them.

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Tom Keffer

kef...@threefools.org
+1 541-386-8891 (h)
+1 541-490-9507 (c)

Skype: tkeffer

[Constantine Samaklis]

I added a step to check if the mySQL database is up and running, as it may be that the db is down and weewx will just put some connectivity entry in the syslog and will keep trying to connect. Instead, if connectivity to the database is down then you will get an email letting you know. (It actually did happen to me, I run mySQL on a different machine, where the mySQL box was off but weewx was working seemingly fine, but I had no archive values getting stored)

On Friday, September 13, 2013 7:44:28 PM UTC-4, Constantine Samaklis wrote:

I thought I would give something back to the group since I have gotten good help to my questions. I have created a Python utility script that can be ran periodically (cron?) to check if weewx is running and if it is not, it will send out an email with a number of lines from the end of the log (/var/log/messages, or whatever log you want) that would show the error that weewx encountered. The script will try to restart weex a number of pre configurable times, in case the error was a fluke rather than a fatal one.

The script will stop trying to send messages after it has reached the indicated restarts times in the configuration times, and you will need to delete the flagFile that has been created. (or write a 0 in it). The script has a configurable log rotation, where you can indicate the log rotation interval (in days) and the days of logs to keep. 

logOutput is the temporary tail of the messages log (or could be the syslog or whatever you specify) that will be emailed as an attachment if weewx is found not to be running.

I hope everyone finds this somewhat useful. If you are somewhat paranoid about your weather station being up 24/7 like me then you probably will. :-)

Questions and comments are welcome.

#configuration file for checking routine

[mail]

fromaddr=us...@gmail.com

toaddr=toU...@gmail.com

msgSubj=weewx process is not running

body=weewx process is not running. Please check the attached log and address the issue

userName=gmailuserName@gmail.com

password=yourPwd

smtpServer=smtp.gmail.com:587

[logging]

logDir=/home/user/scripts

logFile=checkWeewx.log

logOutput=tail.log

linesToTail=200

logToTail=/var/log/messages

logRotationDayInterval=1

logDaysToKeep=5

[weewx]

#how many times to try and restart the weewx process

restarts=3

flagLogDir=/home/user/scripts

flagFile=weewxNotRunning.txt

startCommand=sudo /etc/init.d/weewx start

You can launch the script by typing: Python ./checkWeewx.py /pathToConfigFile/checkWeewx.config

The configuration format:

#configuration file for checking routine

[mail]

fromaddr=us...@gmail.com

toaddr=toU...@gmail.com

msgSubj=weewx process is not running

body=weewx process is not running. Please check the attached log and address the issue

userName=gmailuserName@gmail.com

[Thomas Keffer]

I must admit, this is an area that has not been tested extensively. There is an assumption that, if you use MySQL, that the server will always be up. The code that deals with this could be hardened up.

-tk

--

You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Arild Halvorsen]

This looks exciting. How do I implement this solution in weewx? Where do I put the files. Which information is nødvenidig in weewx.conf?

[Constantine Samaklis]

The script file and configuration file can be anywhere you want.

You can launch the script by typing: python <path>/checkWeewx.py <path>/checkWeewx.config

There are no changes required in the weewx.conf. You will need to just fill the pertinent information for your setup in checkWeewx.config as it is the configuration file that drives the python script.

[pligg...@gmail.com]

I got this running as a cronjob, and I get a mail when weewx is not running when the cronjob is executed. But I don't get the script to start weewx again.

This is the last part of checkweewx.config:

[weewx]
#how many times to try and restart the weewx process
restarts=3

flagLogDir=/home/pliggen/scripts
flagFile=weewxNotRunning.txt
startCommand=sudo /etc/init.d/weewx start

If I run /etc/init.d/weewx start in terminal it starts.

Any tips where to check to get this working?

[Leon Shaner]

Hi, hi,

The sudo implies you have this in your own / non-root crontab.  Is that correct?

If instead, you use "sudo crontab -e" so root is calling your script, then you can omit the sudo in the script.  That much alone may be the solution to your problem.

Meanwhile, you didn't say what OS you're running, but on newer systems, those RC / init.d scripts are often presented via legacy wrappers for use by systemctl.  In that case the systemctl mechanism for restarting the service might be less susceptible to foreground vs. background / disconnected usage.   Meaning where you are using "sudo /etc/init.d/weewx start" from command-line and it works, but doesn't work from cron, substituting the systemctl equivalent may be a way out.

You could try these from command line to verify your system has the legacy RC / init.d wrappers, then substitute the the "sudo systemctl start weewx" of calling the init.d script directly...

Here again, however, putting any of these in a script that runs from the root crontab will avoid needing to prefix with sudo...

# Check status:

$ sudo systemctl status weewx

# Restart

$ sudo systemctl restart weewx

# Stop

$ sudo systemctl stop weewx

# Start

$ sudo systemctl start weewx

Regards,

Leon

--

Leon Shaner :: Dearborn, Michigan (iPad Pro)

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Constantine Samaklis]

Correct. The script should be called from the root crontab as only root can start the weex service without getting prompted for a password. The command should be either service weewx start or systemctl start weewx

On Tuesday, April 30, 2019 at 3:46:18 PM UTC-4, Leon Shaner wrote:

Hi, hi,

The sudo implies you have this in your own / non-root crontab.  Is that correct?

If instead, you use "sudo crontab -e" so root is calling your script, then you can omit the sudo in the script.  That much alone may be the solution to your problem.

Meanwhile, you didn't say what OS you're running, but on newer systems, those RC / init.d scripts are often presented via legacy wrappers for use by systemctl.  In that case the systemctl mechanism for restarting the service might be less susceptible to foreground vs. background / disconnected usage.   Meaning where you are using "sudo /etc/init.d/weewx start" from command-line and it works, but doesn't work from cron, substituting the systemctl equivalent may be a way out.

You could try these from command line to verify your system has the legacy RC / init.d wrappers, then substitute the the "sudo systemctl start weewx" of calling the init.d script directly...

Here again, however, putting any of these in a script that runs from the root crontab will avoid needing to prefix with sudo...

# Check status:

$ sudo systemctl status weewx

# Restart

$ sudo systemctl restart weewx

# Stop

$ sudo systemctl stop weewx

# Start

$ sudo systemctl start weewx

Regards,

Leon

--

Leon Shaner :: Dearborn, Michigan (iPad Pro)

On Apr 30, 2019, at 3:10 PM, plig...@gmail.com wrote:

I got this running as a cronjob, and I get a mail when weewx is not running when the cronjob is executed. But I don't get the script to start weewx again.

This is the last part of checkweewx.config:

[weewx]
#how many times to try and restart the weewx process
restarts=3
flagLogDir=/home/pliggen/scripts
flagFile=weewxNotRunning.txt
startCommand=sudo /etc/init.d/weewx start

If I run /etc/init.d/weewx start in terminal it starts.

Any tips where to check to get this working?

Den söndag 11 oktober 2015 kl. 02:45:50 UTC+2 skrev Constantine Samaklis:

The script file and configuration file can be anywhere you want.

You can launch the script by typing: python <path>/checkWeewx.py <path>/checkWeewx.config

There are no changes required in the weewx.conf. You will need to just fill the pertinent information for your setup in checkWeewx.config as it is the configuration file that drives the python script.

On Saturday, October 10, 2015 at 5:22:12 PM UTC-4, Arild Halvorsen wrote:

This looks exciting. How do I implement this solution in weewx? Where do I put the files. Which information is nødvenidig in weewx.conf?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx...@googlegroups.com.

[pligg...@gmail.com]

Hi Leon,

and thanks for the help! I was running my own non-root crontab. So I tried to put the command in sudo crontab -e.
Stopped the service and tried it and now the service started as expected. Great! And it worked with /etc/init.d/weewx start as startcommand.
I'm not sure I understood everything you wrote, I'm quite new at this, but you made it clear what to try.

One more question, it seems that when I run it via sudo crontab there were some issue with the logging,
hade to remove home/pliggen/scripts from the "logDir=" line.

Got this email message:

sh: 1: cannot create /home/pliggen/scripts//var/log/syslog.txt: Directory nonexistent
Starting program...
Traceback (most recent call last):
File "/var/www/html/scripts/checkWeewx/checkWeewx.py", line 159, in <module>
sendMailFailure(logDir + "/" + logOutput, config, additionalText)
File "/var/www/html/scripts/checkWeewx/checkWeewx.py", line 86, in sendMailFailure
f = file(filename)
IOError: [Errno 2] No such file or directory: '/home/pliggen/scripts//var/log/syslog.txt'

So I removed the logDir file-path so it looks like this now:

[logging]
logDir=
logFile=checkWeewx.log
logOutput=/var/log/syslog.txt
linesToTail=200
logToTail=/var/log/messages
logRotationDayInterval=1
logDaysToKeep=5

And now I get this message in the email when the cronjob runs if the service is stopped:

sh: echo: I/O error
sh: echo: I/O error
Starting program...

It doesn't matter as long as the service now seems to start but if you see something obvious I'm glad to hear!

Thanks, and sorry for my bad english.

[Leon Shaner]

Hi, hi,

I didn't look at the mechanics of the script too closely, but before you had shown:

flagLogDir=/home/pliggen/scripts

And just now you showed:

logOutput=/var/log/syslog.txt

Which I glean from the error is being concatenated to:

 /home/pliggen/scripts//var/log/syslog.txt

And that can't work if you don't first "mkdir -p  /home/pliggen/scripts//var/log/" except I doubt very much you really want such a convoluted path down to the checkWeewx.log.  ;-)

If you want to keep the log in your home dir, you probably want:

logOutput=checkWeewx.log

But it will be owned by root in your home dir.

I wouldn't be messing with the syslog directly, but if you want the checkWeewx.log to be next to syslog, you could do this:

flagLogDir=/var/log

logOutput=checkWeewx.log

So if I have gleaned correctly, that will give you a log owned by root, here:

/var/log/checkWeewx.log

The author of the script can validate the above, if I have it right or wrong.  ;-)

Regards,

Leon

--

Leon Shaner :: Dearborn, Michigan (iPad Pro)

[Constantine Samaklis]

You are definitely on the right path :-)

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/9yskDGu9vI0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[pligg...@gmail.com]

Ok, thanks, now everything works perfect! Hope that the script never has to start weewx but occasionally the service has stopped and now it will start unattended again!
And thanks to the author for a great script!

BR Mikael

[James Bellanca]

I know this threads a little old, but thanks for the script. I just implemented it now.

It appears that after it detects WeeWx is down, it restarts it, but then leaves the weewxNotRunning.txt file out there with the number of retries in it. So, say it has to restart WeeWx, successfully does so, and a week later does it again, etc. After the 3rd week, because it counts as a "retry" and the file's left out there, it will stop. So, in other words, you have to know, if you get the email, make sure you go out there and delete the weewxNotRunning.txt manually to reset the counter, or it'll stop working after the 3rd time it happens. I'm not missing anything, right?

[Constantine Samaklis]

Yes, that is correct. The idea is that there is something really going wrong if you have this happening multiple times. (You can bypass that by setting the restarts in the config to a higher number but I would not advise to do so)

If it is a one off, then resetting the number in the flag file to 0 would work best and is a safer approach.

--

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/9yskDGu9vI0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3e10898a-5ec9-4ee1-b4ea-aceb85e8602ao%40googlegroups.com.

[Silvio Schömann]

Hello everyone, I have an error message 'too few arguments' What am I doing wrong ?

[Silvio Schömann]

I'll answer myself :) Didn't read right and python executed wrong. correct: python /checkWeewx.py /pathToConfigFile/checkWeewx.config i.: python /checkWeewx.py

Greetings Silvio