Suitability for thousands of devices?

[Simon]

I'm looking for a collector to use to collect data for potentially
thousands of devices (definitely thousands of individual queries.)

I like some features of RRDBot (like the ability to specify an
interface name and do a table lookup) but I don't see any information
on whether or not it forks.

Is this collector suitable for doing thousands (tens of thousands..)
of queries in a 5 minute interval? Does it run the queries
sequentially?

[Stef Walter]

Simon wrote:
> I'm looking for a collector to use to collect data for potentially
> thousands of devices (definitely thousands of individual queries.)

I query many hundreds of SNMP agents with tightish poll times (10 to 30
seconds).

> I like some features of RRDBot (like the ability to specify an
> interface name and do a table lookup) but I don't see any information
> on whether or not it forks.
>
> Is this collector suitable for doing thousands (tens of thousands..)
> of queries in a 5 minute interval? Does it run the queries
> sequentially?

It runs queries in parallel, but does not fork or use multiple threads.
It uses a single open udp socket and it's a goal to be efficient. The
RRD updating is far more expensive than the SNMP bits.

If you find it to be inefficient for a certain work load, I'd love to
hear about it. Contributions would be even better.

Cheers,

Stef Walter

[Bart van den Heuvel]

Hi,

 

I'm also about to use rrdbot on a somewhat larger scale. My first tests are also proving that the queries run parallel with good performance. A multi thread/fork architecture would be nice if you run rrdbot for large scale polling on a multi core server.

 

I will post my result when i run my first big load.

Regards, Bart

 

2009/3/18 Stef Walter <stef...@memberwebs.com>

[Stef Walter]

Bart van den Heuvel wrote:
> I'm also about to use rrdbot on a somewhat larger scale. My first tests
> are also proving that the queries run parallel with good performance. A
> multi thread/fork architecture would be nice if you run rrdbot for large
> scale polling on a multi core server.

That's true, I can see how it would be. I'm not sure I have the time to
get into that any time soon. But I'd be supportive of an effort that
integrates such a feature in a sane manner.

Cheers,

Stef

[Simon]

I've decided to test RRDBot out for the tool I am creating. I'll be
polling about 2,000 devices for testing, probably somewhere between
8,000 - 12,000 total queries every 5 minutes or so. I'm currently
running an older revision of this tool using MRTG which required me to
distribute the polling over two servers. I'll let you know the results
with RRDBot for reference sake.

Thanks for the input you have provided.

[Stef Walter]

Simon wrote:
> I've decided to test RRDBot out for the tool I am creating. I'll be
> polling about 2,000 devices for testing, probably somewhere between
> 8,000 - 12,000 total queries every 5 minutes or so. I'm currently
> running an older revision of this tool using MRTG which required me to
> distribute the polling over two servers. I'll let you know the results
> with RRDBot for reference sake.

Awesome. I'd love to hear the results.

Cheers,

Stef