Implementing a monitoring system with django.

[Marc Aymerich]

Hi,
I'm considering to implement a simple monitorization system that
basically gathers some data from a set of nodes (<1000 nodes). The
gathered data should be stored in order to let users perform some
queries and maybe also generate some graphs.

I'm looking for advice in what components are best suited for each of
the following parts:

1) Storage: maybe something nonsql like MongoDB? or perhaps RRD?
2) Data gathering: celery periodic tasks?
3) Graphs: rrd? or some java script framework for graphing?

thanks for your comments!!
--
Marc

[Bill Freeman]

While this may be a fine opportunity to learn about some of these techniques (NoSQL, celery), unless I misunderstand the potential bandwidths involved, the problem fits more pedestrian stuff pretty well.

If I were doing this I'd use PostgreSQL and a cron (or equivalent on your OS) triggered django management command to gather the data and stuff it into the models.  Certainly django can provide display of the data to the user.

For graphing, I've used YUI (yahoo) tools in the past (I'm sure that google has competitive offerings), but it required that the user have Flash.  While pretty much everyone has Flash, I'd rather go for HTML5 or SVG (and a few people may have browsers that don't do these).  Maybe modern frameworks can be had that detect the browser's capabilities and respond accordingly.  (The YUI stuff was driven by a table of data in the DOM, and that approach allows the support decision to be deferred to the browser side.)  Depending on load and update rate, you could also generate png or jpg files on the fly in a view that IMG tags in your HTML refer to, which works even if the user has javascript disabled, but loads the server a bit.  You would probably want to encode a timestamp or generation sequence into the generated image path name, so that you wouldn't have to worry about cacheing, including in the browser, keeping you from showing a fresh image when the user refreshes the page.

Bill

[Marc Aymerich]

Hi bill thanks for your feedback!
You're right, maybe I can start using a traditional RDBMS like
postgres and switch when things starts getting slow, if even it really
happens at all.

By the way, I've been looked at some nosql storage backends, and
couchdb seems the more appropriate option for storing time-series data
in django.

regarding the graph stuff, the project is somehow related with
opensource and openeverything, so flash is not really an option. I'll
look at some js framework.

thanks again!
--
Marc

[ephan]

Depending on what you need for the graphs,have a look at flot,jqplot or rapheal.js(heavy).There is also high charts althought there are some licensing issues you may want to avoid.

If you get stuck with the charts,let me know..am running some.

[Tom Evans]

Any reason not to use something like Nagios or Cacti? We use Nagios at
$JOB, it's great.

Cheers

Tom

[mulianto]

hi 

what are you monitoring in the 1000 nodes, web server load, db load , network interface stats, cpu load, memory usage, or specific thing.

i would use centralized monitoring that collect from all the server in centralized location, with low overhead in periodical  time like 10 seconds and save in rrd format which many tools already available out there.

i prefer cacti for view the graph from rrd data collected. and for the collector use collectd and store it in centralized collectd server.

this what i done before to monitoring my server ˆ⌣ˆ 

Sent from my iPhone

On 12 Des 2012, at 13:34, Ken Hill <kennet...@gmail.com> wrote:

I've been considering a very similar Django to help simplify some of my daily server system admin tasks (e.g. monitor and trend CPU uase, memory usage, disk space, etc.). I'd like to try CouchDB to store the data with a Django UI to make the app usable to non-technical users.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/KNDp3QSOrx4J.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

[Marc Aymerich]

On Wed, Dec 12, 2012 at 7:59 AM, ephan <eph...@gmail.com> wrote:
> Depending on what you need for the graphs,have a look at flot,jqplot or
> rapheal.js(heavy).There is also high charts althought there are some
> licensing issues you may want to avoid.
>
> If you get stuck with the charts,let me know..am running some.

Thanks Ephan,
those graphs looks really nice! I'm looking into them.

> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/django-users/-/pxDyxFeEwBoJ.

>
> To post to this group, send email to django...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.

--
Marc

[Marc Aymerich]

Hi Tom,
We already have developed a web interface for controlling those nodes
in Django, so I think that it would be easy to run our own monitoring
system than integrate Nagios into our existing interface.


--
Marc

[Ryan Nowakowski]

Checkout Graphite[1]. It seems like it would handle your graphing
requirement.

1. http://graphite.wikidot.com/start

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.

[Sergiy Khohlov]

"Pro python system administration" by Rytis Sileika published by
Apress covers this area. Jinja2 is used for generating templates. You
also can use django for this task.

Many thanks,

Serge


+380 636150445
skype: skhohlov


2012/12/17 Ryan Nowakowski <tub...@fattuba.com>: