Looking for Python process WatchDog systems : Any alternatives to this ?

[David Lyon]

Hi all,

I'm looking for something to run and automatically restart python scripts on a server.

I found this:

 - https://github.com/sebastien/monitoring

His presentation is at:

 - http://www.slideshare.net/ffunction/fabric-cuisine-and-watchdog-for-server-administration-in-python

Are there any better/more-popular tools to do the same thing?

[Rhys Elsmore]

I have continually used Supervisord - http://supervisord.org/

Even has a built in XMLRPC for remote controlling/monitoring your processes :)

David Lyon

1 July 2013 4:05 PM

--
You received this message because you are subscribed to the Google Groups "sydneypython" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sydneypython...@googlegroups.com.
To post to this group, send email to sydney...@googlegroups.com.
Visit this group at http://groups.google.com/group/sydneypython.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Hal Ashburner]

supervisor?
http://supervisord.org/

[eugene.va...@gmail.com]

http://circus.readthedocs.org

Newcomer but funky!

EuGeNe -- follow me on http://twitter.com/3kwa

[Dylan Jay]

On 01/07/2013, at 4:06 PM, Rhys Elsmore <m...@rhys.io> wrote:

I have continually used Supervisord - http://supervisord.org/

Even has a built in XMLRPC for remote controlling/monitoring your processes :)

We use this on all our deployments and even have devaloped several plugins for it such as mr.rubber

https://pypi.python.org/pypi/mr.rubber

The other python based one I know about is Circus

https://pypi.python.org/pypi/circus/

I believe it's under heavy development. I know there are reasons Tarek decided to develop a new one instead of use supervisord but I forget what they were.

[Dylan Jay]

On 01/07/2013, at 4:10 PM, Dylan Jay <dj...@pretaweb.com> wrote:

On 01/07/2013, at 4:06 PM, Rhys Elsmore <m...@rhys.io> wrote:

I have continually used Supervisord - http://supervisord.org/

Even has a built in XMLRPC for remote controlling/monitoring your processes :)

We use this on all our deployments and even have devaloped several plugins for it such as mr.rubber
https://pypi.python.org/pypi/mr.rubber

Sorry, that wasn't very clear. We use supervisord, not circus.

[David Lyon]

Thanks guys.

Supervisor was really what I was expecting. When I looked more closely I noticed there was a lot more documentation and the project is much more significant.

However, when I tweaked 'monitoring', I got it to do what I wanted fairly easily. Here was my script to get 'geany' running and stay running. Every time you close it, the process restarts:

from monitoring import *
__doc__ = """How to ensure that Geany stays up and running."""
Monitor(
    Service(
        name="geany-ensure-up",
        monitor=(
            ProcessInfo(command='geany',
                freq=Time.ms(1000),
                fail=[
                    Incident(
                        # If we have 5 errors during 5 seconds...
                        errors=5,
                        during=Time.s(5),
                        actions=[
                            # NOTE: Restart will make the process a child of the monitoring, so
                            # you might prefer to use something like upstart
                            Restart("geany")
                        ]
                    )
                ]
            )
        )
    )
).run()

[Ben De Luca]

while True;  do  python myprogram.py ; sleep 5 ; done

Do people have non trivial software out there running for years or more? 

[David Lyon]

On Tue, Jul 2, 2013 at 3:41 PM, Ben De Luca <bde...@gmail.com> wrote:

while True;  do  python myprogram.py ; sleep 5 ; done

Do people have non trivial software out there running for years or more?

Funnily enough, you read our servers mind/core-dump. :-)

As it turns out, the 'monitor' library didn't work. Firstly because Python 2.5 was
all that there was on the server. After installing Python 2.7 from source it still
didn't work properly as it had dependencies on what I presume is a modern
linux kernel. The Kernel on the Server is Linux 2.5 (??) and can't be upgraded
as the distro discontinued.

Reason it hasn't been upgraded is that it has been running for years. Uptime
on that server is about 3 years. It does work in the Insurance industry.

The code you posted above, after a day of frustration, actually looks like it
would be the best fit for what we have know.

(Management decision: time to upgrade the machine after 7 years of
 continous operation).

[Rhys Elsmore]

Can I ask why you passed on supervisor? (I have setup many different processes using it, and would be happy to help/shoot you some advice)

[Ben De Luca]

In terms of software that bash line does most of what people want from an auto start, im sure the other 5% will take another 1000 lines to write though.

I have a few processes that are rock solid unless the file system goes away, and thats how I deal with them. Mostly there are just programs that start/monitor our distributed resource scheduler (GridEngine), which keeps services running as needs be. 

 

[David Lyon]

On Tue, Jul 2, 2013 at 5:13 PM, Rhys Elsmore <m...@rhys.io> wrote:

Can I ask why you passed on supervisor? (I have setup many different processes using it, and would be happy to help/shoot you some advice)

I don't have a good answer for that.

What I should actually do, is revisit supervisor and see if it can be made to work on the same server. My guess is that it may just well work, but we will see.

Thanks for the assistance. :-)

[Xuanyi Chew]

I have used Monit in the past. Fork the Cookbook currently also uses Monit + statsD. I feel Monit is lighterweight than Supervisord, and I'm a fan of lightweightedness. You might want to check that out too.

Xuanyi Chew
+61403928398

[Dylan Jay]

On 02/07/2013, at 5:33 PM, Xuanyi Chew <che...@gmail.com> wrote:

> I have used Monit in the past. Fork the Cookbook currently also uses Monit + statsD. I feel Monit is lighterweight than Supervisord, and I'm a fan of lightweightedness. You might want to check that out too.

I'd say monit is more heavyweight than supervisord. I've used both.

If you want to consider non python alternatives (like monit), then I've also used upstart which works well enough.

[Graham Dumpleton]

On 02/07/2013, at 5:54 PM, Dylan Jay <dj...@pretaweb.com> wrote:

> On 02/07/2013, at 5:33 PM, Xuanyi Chew <che...@gmail.com> wrote:
>
>> I have used Monit in the past. Fork the Cookbook currently also uses Monit + statsD. I feel Monit is lighterweight than Supervisord, and I'm a fan of lightweightedness. You might want to check that out too.
>
> I'd say monit is more heavyweight than supervisord. I've used both.

Define heavyweight. Supervisord consumes starting memory of about 10MB+, which is a lot when a PaaS uses it and counts it towards your memory usage. Really eats into things when trying to cram stuff into a 32MB instance.

Graham

[Dylan Jay]

:)

Monit is heavy to me in features and complexity of the configuration. But since supervisord is python it will consume more memory than monit or upstart or pretty much anything written in C.


>
> Graham