[rrd-users] Newby questions

[Pablo Chacin]

Greetings all

I'm new to rrdtools. I'm actually still evaluating it as as tool for
storing monitoring metrics.
After reading the documentation and several discussions in this list and
also in other forums
I'm still confused about a couple of things.
1. How are subsecond timestamt reported? In several places it is
mentioned the existence of these timestamps, but can't find any example
of how to specify it in the update command, except when using "N" as
timestamp.

2. Supposing subsecond timestamps are supported by rrdtools, are they
also supported by rrdcached?

3. Finally, is there any patch to add SUM function for totalizing a
metric over a period. This is a very important requirement as we need to
count the total number of certain events for minute, hour and day. I'm
aware of the approach of using the average and multiplying by the
reporting period, but this will lead to a approximated value. Even when
I understand the concerns about it not working properly in all
situations, I think is a very common requirement. Simply can't believe
nobody has done this ever, unless I'm completely wrong about the need of
such function.

many thanks in advance


--
Pablo Chacin
R&D Engineer
SenseFields SL
Tlf (+34) 93 418 05 85
Baixada de Gomis 1,
08023 Barcelona (Spain)
http://www.sensefields.com/


_______________________________________________
rrd-users mailing list
rrd-...@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[Tobias Oetiker]

Today Pablo Chacin wrote:

> Greetings all
>
> I'm new to rrdtools. I'm actually still evaluating it as as tool for
> storing monitoring metrics.
> After reading the documentation and several discussions in this list and
> also in other forums
> I'm still confused about a couple of things.
> 1. How are subsecond timestamt reported? In several places it is
> mentioned the existence of these timestamps, but can't find any example
> of how to specify it in the update command, except when using "N" as
> timestamp.

rrdtool update 1351545246.3897:394.33

> 2. Supposing subsecond timestamps are supported by rrdtools, are they
> also supported by rrdcached?

I think the version on github does support it ... the released
version not.

>
> 3. Finally, is there any patch to add SUM function for totalizing a
> metric over a period. This is a very important requirement as we need to
> count the total number of certain events for minute, hour and day. I'm
> aware of the approach of using the average and multiplying by the
> reporting period, but this will lead to a approximated value.

everything in rrdtool is a 'rate' so even if there was such a
function the math would still be the same ... can yo elaborate in
what way you expect the result of the calculation to be an
aproximation?

> Even when
> I understand the concerns about it not working properly in all
> situations, I think is a very common requirement. Simply can't believe
> nobody has done this ever, unless I'm completely wrong about the need of
> such function.

cheers
tobi

> many thanks in advance
>
>
>

--

Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch to...@oetiker.ch ++41 62 775 9902 / sb: -9900

[Pablo Chacin]

Hi Tobi

> 3. Finally, is there any patch to add SUM function for totalizing a
> metric over a period. This is a very important requirement as we need to
> count the total number of certain events for minute, hour and day. I'm
> aware of the approach of using the average and multiplying by the
> reporting period, but this will lead to a approximated value.
> everything in rrdtool is a 'rate' so even if there was such a
> function the math would still be the same ... can yo elaborate in
> what way you expect the result of the calculation to be an
> aproximation?
>

According to what I read in the previous discussions on this list
(https://lists.oetiker.ch/pipermail/rrd-developers/2005-April/001405.html)
The presence of unknown values may produce wrong totals when
calculating the total from the average. For example:

> 0:UNKN 300:10 600:10 900:10
>
> consolidated and then mulitplied by the period will produce the same value
> as if all entries were 10, and not with one UNKN (given the cf is 0.5). A
> SUM function would nicely return 30.
>

Regards

Pablo

Pablo Chacin
R&D Engineer
SenseFields SL
Tlf (+34) 93 418 05 85
Baixada de Gomis 1,
08023 Barcelona (Spain)
http://www.sensefields.com/

[Tobias Oetiker]

Today Pablo Chacin wrote:

>
> Hi Tobi
>
> > 3. Finally, is there any patch to add SUM function for totalizing a
> > metric over a period. This is a very important requirement as we need to
> > count the total number of certain events for minute, hour and day. I'm
> > aware of the approach of using the average and multiplying by the
> > reporting period, but this will lead to a approximated value.
> > everything in rrdtool is a 'rate' so even if there was such a
> > function the math would still be the same ... can yo elaborate in
> > what way you expect the result of the calculation to be an
> > aproximation?
> >
>
> According to what I read in the previous discussions on this list
> (https://lists.oetiker.ch/pipermail/rrd-developers/2005-April/001405.html)
> The presence of unknown values may produce wrong totals when
> calculating the total from the average. For example:
>
> > 0:UNKN 300:10 600:10 900:10
> >
> > consolidated and then mulitplied by the period will produce the same value
> > as if all entries were 10, and not with one UNKN (given the cf is 0.5). A
> > SUM function would nicely return 30.
> >

you can track the amount of unknown time too ... but you can use
the TOTAL function will ignore unknonwn values ...

otoh, why would you want to count UNKNOWN as ZERO ? isn't more
corect to asume that the values during the time where you have no
information were the same as in over the rest of the interval ?

cheers
tobi

Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch to...@oetiker.ch ++41 62 775 9902 / sb: -9900

[Pablo Chacin]

On 10/29/2012 11:57 PM, Tobias Oetiker wrote:
> you can track the amount of unknown time too ... but you can use
> the TOTAL function will ignore unknonwn values .

Didn’t know this. Could you please point-me to

> otoh, why would you want to count UNKNOWN as ZERO ? isn't more
> corect to asume that the values during the time where you have no
> information were the same as in over the rest of the interval ?

In my case, no. I may have gaps in the collection which, if i understand
correctly
will be traduced as unknown.

--
Pablo Chacin
R&D Engineer
SenseFields SL
Tlf (+34) 93 418 05 85
Baixada de Gomis 1,
08023 Barcelona (Spain)
http://www.sensefields.com/

[Pablo Sanchez]

[ Comments below, in-line ]

On 10/29/2012 06:57 PM, Tobias Oetiker wrote:

> Today Pablo Chacin wrote:
>
>> For example:
>>
>>> 0:UNKN 300:10 600:10 900:10
>>>
>>> consolidated and then mulitplied by the period will produce the
>>> same value as if all entries were 10, and not with one UNKN (given
>>> the cf is 0.5). A SUM function would nicely return 30.
>>>
>

> otoh, why would you want to count UNKNOWN as ZERO ? isn't more
> corect to asume that the values during the time where you have no
> information were the same as in over the rest of the interval ?

Hey Tobi,

I realize RRD is not an ANSI SQL compliant database (*grin*) ... some
confusion might arise that in a database, NULL (unknown) is not
sum()'d:

--8-<---8-<---8-<---8-<---8-<---8-<---8-<--
19:05:23 plugh > insert into tobi values (1);

1 row created.

Elapsed: 00:00:00.00
19:05:31 plugh > insert into tobi values (4);

1 row created.

Elapsed: 00:00:00.01
19:05:36 plugh > insert into tobi values (8);

1 row created.

Elapsed: 00:00:00.00
19:05:38 plugh > insert into tobi values (null);

1 row created.

Elapsed: 00:00:00.00
19:05:44 plugh > commit;

Commit complete.

Elapsed: 00:00:00.10
19:05:46 plugh > select * from tobi;

SOME_COLUMN
-----------
1
4
8


Elapsed: 00:00:00.01
19:05:49 plugh > select sum(some_column) from tobi;

SUM(SOME_COLUMN)
----------------
13

Elapsed: 00:00:00.00
19:05:58 plugh >
--
Pablo Sanchez - Blueoak Database Engineering, Inc
Ph: 819.459.1926 Blog: http://pablo.blog.blueoakdb.com
Fax: 760.860.5225 (US)

[j.hal...@seznam.cz]

>I realize RRD is not an ANSI SQL compliant database

Correct. Even the rrd isn't a database at all.

jh

---------- Původní zpráva ----------
Od: Pablo Sanchez <pa...@blueoakdb.com>
Datum: 30. 10. 2012
Předmět: Re: [rrd-users] Newby questions

[Pablo Sanchez]

[ Comments below, in-line ]

On 10/29/2012 10:51 PM, j.hal...@seznam.cz wrote:
>
> >I realize RRD is not an ANSI SQL compliant database
>
>

> Even the rrd isn't a database at all.

Actually, it is a database but not in the way you're defining it. )

http://en.wikipedia.org/wiki/Database

Cheers,

[Tobias Oetiker]

Today Pablo Chacin wrote:

> On 10/29/2012 11:57 PM, Tobias Oetiker wrote:
> > you can track the amount of unknown time too ... but you can use
> > the TOTAL function will ignore unknonwn values .
>

> Didn?t know this. Could you please point-me to

you can use a CDEF to turn UNKNOWN to 0 and all the rest of the
entries to 1 calculate the AVERAGE. This will tell you how much
'UNKNOWN' is contained in your data stream.

> > otoh, why would you want to count UNKNOWN as ZERO ? isn't more
> > corect to asume that the values during the time where you have no
> > information were the same as in over the rest of the interval ?

> In my case, no. I may have gaps in the collection which, if i understand
> correctly will be traduced as unknown.

yes ... and thus, if you build a sum over all the entries, the most
correct solution is that the SUM is also unknown. RRDtool has the
option of ignoring the unknown entries using the TOTAL function
(which will cause them to essentially be treated as 0).

or if you take the average and and multiply with the total
interval, you assume the unknwon samples to be equal to the
average.

cheers
tobi

>
>
>

--
Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
http://it.oetiker.ch to...@oetiker.ch ++41 62 775 9902 / sb: -9900

[Pablo Chacin]

I think that "it is not a database" is not the point.

The point is: is the way unknown values handle
coherent with the calculation of totals?

As I see it, there's no fundamental reason not to
be able to calculate totals in a RRD. Other implementations
of the concept, as StatsD and Graphite allows it.

On 10/30/2012 03:51 AM, j.hal...@seznam.cz wrote:

>I realize RRD is not an ANSI SQL compliant database

Correct. Even the rrd isn't a database at all.

jh

---------- Původní zpráva ----------
Od: Pablo Sanchez <pa...@blueoakdb.com>
Datum: 30. 10. 2012
Předmět: Re: [rrd-users] Newby questions

[ Comments below, in-line ]

On 10/29/2012 06:57 PM, Tobias Oetiker wrote:
> Today Pablo Chacin wrote:
>
>> For example:
>>
>>> 0:UNKN 300:10 600:10 900:10
>>>
>>> consolidated and then mulitplied by the period will produce the
>>> same value as if all entries were 10, and not with one UNKN (given
>>> the cf is 0.5). A SUM function would nicely return 30.
>>>
>

> otoh, why would you want to count UNKNOWN as ZERO ? isn't more
> corect to asume that the values during the time where you have no
> information were the same as in over the rest of the interval ?

--
Pablo Sanchez - Blueoak Database Engineering, Inc
Ph: 819.459.1926 Blog: http://pablo.blog.blueoakdb.com
Fax: 760.860.5225 (US)

_______________________________________________
rrd-users mailing list
rrd-...@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

_______________________________________________
rrd-users mailing list
rrd-...@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[Pablo Chacin]

On 10/30/2012 07:35 AM, Tobias Oetiker wrote:
> yes ... and thus, if you build a sum over all the entries, the most
> correct solution is that the SUM is also unknown. RRDtool has the
> option of ignoring the unknown entries using the TOTAL function
> (which will cause them to essentially be treated as 0).

Unfortunately, I can do this only when retrieving data, in a VDEF,
not in a CDEF to be actually stored that way, right?

By the way, can I use VDEF in a export? from documentation it is
not evident when VDEF are allowed.

Finally, as this requirement comes frequently, will you eventually
consider to incorporate TOTAL (or SUM) CDEF a feature of RRDTools?

Thanks

--
Pablo Chacin
R&D Engineer
SenseFields SL
Tlf (+34) 93 418 05 85
Baixada de Gomis 1,
08023 Barcelona (Spain)
http://www.sensefields.com/

[Simon Hobson]

Pablo Chacin wrote:

>The point is: is the way unknown values handle
>coherent with the calculation of totals?
>
>As I see it, there's no fundamental reason not to
>be able to calculate totals in a RRD.

You can compute totals in RRD - the problem is that the way it does
it isn't the way you want to do it *in this application*. Which ever
way it's done, there will be a lot of people for whom that is not the
right way. Even in one setup the requirements may change - for
example ...

At work I use RRD to log traffic on our network. We sell on some of
our bandwidth to other tenants in the building, but we don't charge
by volume of data or impose monthly caps.
Now consider these scenarios :

1) We have a failure in data collection, but the network is still up.
In this case, traffic is still flowing, but we aren't recording it.
The most sensible option would be to assume that the traffic in the
gap is roughly the average of the rates either side of the gap.

2) We have an actual outage (extended power failure so we lose our
monitoring as well as the line).
In this case the sensible option is to assume the data was zero during the gap.

3) As in 1 - we have a collection failure but the network is still
running. If we were imposing caps or charging by the gigabyte then
we'd probably have to assume zero for charging/capping purposes as we
wouldn't have the data to back up any other assumption. But for our
own traffic management/planning we'd want to assume data was still
flowing.

So there you are - which way should RRD do it ? Either way is both
right and wrong. As it is, it's possible to work around it either way
if you use the right calculations when extracting the data.

If you want to assume zero, then this RPN will do the conversion for
you, then work with the result :
aa=a,UNKN,0,a,IF
which means: if a is unknown then substitute 0, else use a.

--
Simon Hobson

Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed
author Gladys Hobson. Novels - poetry - short stories - ideal as
Christmas stocking fillers. Some available as e-books.

[Pablo Chacin]

Thanks Simon, that's a very compelling explanation on how to
handle this situation. Probably it worth putting in the RRDTools'
wiki to prevent others coming regularly and asking the same
question.


Regards

P.S. Just one final question are these RPNs available when fetching
and exporting data? I don't want to actually graph data, but to
export it to other applications. Documentation is not clear.

On 10/30/2012 08:51 AM, Simon Hobson wrote:
> You can compute totals in RRD - the problem is that the way it does
> it isn't the way you want to do it *in this application*. Which ever
> way it's done, there will be a lot of people for whom that is not the
> right way. Even in one setup the requirements may change
>

> So there you are - which way should RRD do it ? Either way is both
> right and wrong. As it is, it's possible to work around it either way
> if you use the right calculations when extracting the data.
>
> If you want to assume zero, then this RPN will do the conversion for
> you, then work with the result :
> aa=a,UNKN,0,a,IF
> which means: if a is unknown then substitute 0, else use a.
>


--

Pablo Chacin
R&D Engineer
SenseFields SL
Tlf (+34) 93 418 05 85
Baixada de Gomis 1,
08023 Barcelona (Spain)
http://www.sensefields.com/