Wrong graphic date/time

[jms667]

Hello Dimitri,

 

First of all, thanks a lot for your great tool, I'm a SAN storage guy and using it in order to graph stuff like servicetime, io/s, KB/s, queue, etc ... on Linux and AIX hosts.

 

Recently I have deployed dimstat client on several AIX hosts. Compilation and running the agent were ok.

Then I created 2 custom stat reports, one based on the command 'iostat -b', and other one based on 'iostat -Dl'.

 

There is an issue with the one based on 'iostat -b'. Data collecting is working fine but date/time are wrong into the graph (the graphic time take more and more advance as compared to the 'real' time), and I don't find the reason. Perhaps it is related to the way I modify the collected data :

 

- the main collect script iostat_mod is calling 2 others scripts :

 

- iostat_mod1 is replacing in a dirty way K (kilo), M (mega), G (giga), etc .. suffix by  zeroes to avoid any issue when data are pushed into the database

 

- iostat_mod2 is dividing each value by 5 in order to get io and KB value per second (and not a cumulated value of 5 seconds -> iostat -b 5)

 

################

bash-3.2# cat /etc/STATsrv/bin/iostat_mod
/etc/STATsrv/bin/iostat_mod1 | /etc/STATsrv/bin/iostat_mod2

 

################

bash-3.2# cat /etc/STATsrv/bin/iostat_mod1
/usr/bin/iostat -b 5 | /usr/bin/sed -e "s/\([0-9]*\).\([0-9]*\)K/\1\2000/g" -e "s/\([0-9]*\).\([0-9]*\)M/\1\2000000/g" -e "s/\([0-9]*\).\([0-9]*\)G/\1\2000000000/g" -e "s/\([0-9]*\).\([0-9]*\)S/\1\2000/g"

################

bash-3.2# cat /etc/STATsrv/bin/iostat_mod2
while read A1 A2 A3 A4 A5 A6 A7 A8 A9
   do
     if [[ $A9 = "" || $A9 = "werr" ]]
        then
          echo "$A1 $A2 $A3 $A4 $A5 $A6 $A7 $A8 $A9"
        else
          B2=`echo $A2 | cut -d'.' -f1`; B2=`expr $B2 / 5`
          B3=`echo $A3 | cut -d'.' -f1`; B3=`expr $B3 / 5`
          B4=`echo $A4 | cut -d'.' -f1`; B4=`expr $B4 / 5`
          B5=`echo $A5 | cut -d'.' -f1`; B5=`expr $B5 / 5`
          echo "$A1 $B2 $B3 $B4 $B5 $A6 $A7 $A8 $A9"
     fi
   done

 

 

I also noticed something strange on the client side.

They are multiple iostat_mod process running. I don't have this behavior with linux hosts collecting scripts.

 

################################

bash-3.2# ps -ef | grep iostat_mod | grep -v grep
    root  5243096 18088142   8 11:43:12      -  4:27 sh -c /etc/STATsrv/bin/iostat_mod
    root 18088142 21889098   0 11:43:12      -  0:00 sh -c /etc/STATsrv/bin/iostat_mod
    root 19529878 18088142   0 11:43:12      -  0:00 sh -c /etc/STATsrv/bin/iostat_mod

 

 

Any help would be appreciated :-)

Thanks.

 

Jean-Marc.

 

[Dimitri]

Hi Jean-Marc,

The main issue I'm suspecting here is that you're using a fixed time
interval of 5sec, while dim_STAT will expect to execute your script
with its own time interval (/etc/STATsrv/bin/iostat_mod %i -- where
%i will be replaced by the interval value) -- so, if you're starting
a collect with 10sec time interval (for ex.) you'r script is not
taking care about and will continue to deliver stats every 5sec.. -
and as the result, your data will be out of sync (in fact dim_STAT is
not collecting data with live timestamps because stats are very often
coming from system commands which a re buffering their output, so the
measurement arriving at any moment may be related to several intervals
back, etc. -- to avoid "time jumps", dim_STAT is using a "serial
number" (SNO), and based on SNO * Timeout + Start_Time we're getting
the corresponding time..)

so, if the supposition is correct, all you need is to adapt your
script to consider time interval..

if not -- then I'll need more info ;-)

regarding multiple iostat_mod processes: looks like there is a father
process with PID: 18088142 and his 2 sons.. - is it possible that AIX
is simply cutting 1 and 2 in names? ;-) -- to check you may just
rename them to see if it's so..

Rgds,
-Dimitri

On 6/13/14, jms667 <jms...@gmail.com> wrote:
>
>
> Hello Dimitri,
>
>
>
> First of all, thanks a lot for your great tool, I'm a SAN storage guy and
> using it in order to graph stuff like servicetime, io/s, KB/s, queue, etc
> ... on Linux and AIX hosts.
>
>
>
> Recently I have deployed dimstat client on several AIX hosts. Compilation
> and running the agent were ok.
>
> Then I created 2 custom stat reports, one based on the command 'iostat -b',
>
> and other one based on 'iostat -Dl'.
>
>
>
> There is an issue with the one based on 'iostat -b'. Data collecting
> is working fine but date/time are wrong into the graph (the graphic time
> take more and more advance as compared to the 'real' time), and I don't
> find the reason. Perhaps it is related to the way I modify the collected
> data :
>
>
>

> - the main collect script *iostat_mod* is calling 2 others scripts :
>
>
>
> -* iostat_mod1* is replacing in a dirty way K (kilo), M (mega), G (giga),

> etc .. suffix by zeroes to avoid any issue when data are pushed into the
> database
>
>
>

> - *iostat_mod2* is dividing each value by 5 in order to get io and KB value

>
> per second (and not a cumulated value of 5 seconds -> iostat -b 5)
>
>
>
> ################
>
> bash-3.2# cat /etc/STATsrv/bin/

> *iostat_mod*/etc/STATsrv/bin/iostat_mod1 | /etc/STATsrv/bin/iostat_mod2
>
>
>
> ################
>
> bash-3.2# cat /etc/STATsrv/bin/
> *iostat_mod1*/usr/bin/iostat -b 5 | /usr/bin/sed -e

> "s/\([0-9]*\).\([0-9]*\)K/\1\2000/g" -e
> "s/\([0-9]*\).\([0-9]*\)M/\1\2000000/g" -e
> "s/\([0-9]*\).\([0-9]*\)G/\1\2000000000/g" -e
> "s/\([0-9]*\).\([0-9]*\)S/\1\2000/g"
>
>
> ################
> bash-3.2# cat /etc/STATsrv/bin/

> *iostat_mod2*while read A1 A2 A3 A4 A5 A6 A7 A8 A9

> --
> You received this message because you are subscribed to the Google Groups
> "dim_STAT" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dimstat+u...@googlegroups.com.
> To post to this group, send email to dim...@googlegroups.com.
> Visit this group at http://groups.google.com/group/dimstat.
> For more options, visit https://groups.google.com/d/optout.
>

[jms667]

Hi Dimitri,

 

Thanks for your quick answer.

You're right I am using fixed collect interval in the script (5 sec), but it's the same interval wich is set in the collect report (Add-On STAT).

I am doing the same thing for an other report and all is well synchronized.

But anyway I followed your advise and used the %i parameter as followed, but unfortunatly I still have the same issue.

 

dimstat collect report (Add-On STAT) :

iostat  %i

 

/etc/STATsrv/bin/iostat_mod
bash-3.2# cat /etc/STATsrv/access
command  iostat         /etc/STATsrv/bin/iostat_mod

 

/etc/STATsrv/bin/iostat_mod
bash-3.2# cat /etc/STATsrv/bin/iostat_mod
/etc/STATsrv/bin/iostat_mod1 $1 | /etc/STATsrv/bin/iostat_mod2

/etc/STATsrv/bin/iostat_mod1
bash-3.2# cat /etc/STATsrv/bin/iostat_mod1
/usr/bin/iostat -b $1 | /usr/bin/grep hdisk | /usr/bin/sed -e "s/\([0-9]*\).\([0-9]*\)K/\1\2000/g" -e "s/\([0-9]*\).\([0-9]*\)M/\1\2000000/g" -e "s/\([0-9]*\).\([0-9]*\)G/\1\2000000000/g" -e "s/\([0-9]*\).\([0-9]*\)S/\1\2000/g"

/etc/STATsrv/bin/iostat_mod2
bash-3.2# cat /etc/STATsrv/bin/iostat_mod2

while read A1 A2 A3 A4 A5 A6 A7 A8 A9
   do
     if [[ $A9 = "" || $A9 = "werr" ]]
        then
          echo "$A1 $A2 $A3 $A4 $A5 $A6 $A7 $A8 $A9"
        else
          B2=`echo $A2 | cut -d'.' -f1`; B2=`expr $B2 / 5`
          B3=`echo $A3 | cut -d'.' -f1`; B3=`expr $B3 / 5`
          B4=`echo $A4 | cut -d'.' -f1`; B4=`expr $B4 / 5`
          B5=`echo $A5 | cut -d'.' -f1`; B5=`expr $B5 / 5`
          echo "$A1 $B2 $B3 $B4 $B5 $A6 $A7 $A8 $A9"
     fi
   done

Regards,

JM.

[Dimitri]

Hi Jean-Marc,

may you send me then:
- an example of your script output (10-15 iterations, just to have
an idea, better to pipe)..
- your "iostat" Add-On export (just export it as a file via
Home->Add-Ons-> Export)

I'm pretty curious what is going wrong in your case..

Rgds,
-Dimitri

On 6/16/14, jms667 <jms...@gmail.com> wrote:
> Hi Dimitri,
>
> Thanks for your quick answer.
> You're right I am using fixed collect interval in the script (5 sec), but
> it's the same interval wich is set in the collect report (Add-On STAT).
> I am doing the same thing for an other report and all is well synchronized.
> But anyway I followed your advise and used the %i parameter as followed,
> but unfortunatly I still have the same issue.
>

> *dimstat collect report (Add-On STAT) :*
> iostat %i
>
>
> */etc/STATsrv/bin/iostat_mod*bash-3.2# cat /etc/STATsrv/access
> command iostat /etc/STATsrv/bin/iostat_mod
>
>
> */etc/STATsrv/bin/iostat_mod*bash-3.2# cat /etc/STATsrv/bin/iostat_mod
> /etc/STATsrv/bin/iostat_mod1 $1 | /etc/STATsrv/bin/iostat_mod2
>
> */etc/STATsrv/bin/iostat_mod1*

> bash-3.2# cat /etc/STATsrv/bin/iostat_mod1
> /usr/bin/iostat -b $1 | /usr/bin/grep hdisk | /usr/bin/sed -e
> "s/\([0-9]*\).\([0-9]*\)K/\1\2000/g" -e
> "s/\([0-9]*\).\([0-9]*\)M/\1\2000000/g" -e
> "s/\([0-9]*\).\([0-9]*\)G/\1\2000000000/g" -e
> "s/\([0-9]*\).\([0-9]*\)S/\1\2000/g"

> */etc/STATsrv/bin/iostat_mod2*

[jms667]

Hi Dimitri,

Please have a look at the files attached.
(regarding script output, each new iteration is marked by "## new line added manually").

I noticed something strange which could be the problem.
When executing 'iostat_mod1' script, all is well synchronized (iostat iteration are sync with the real time display).
When executing 'iostat_mod' script (==> /etc/STATsrv/bin/iostat_mod1 $1 | /etc/STATsrv/bin/iostat_mod2), iostat iteration are NOT synchronized with the display. Piping a script into an other is probably not a good idea in order to modify on the fly the data in my case ?

If a create a new dimstat report (Home->Add-Ons->Integrate New Add-On STAT) using only the script /etc/STATsrv/bin/iostat_mod1, all is ok, date and time are sync with the 'real time'.

Regards,
JM.

[Dimitri]

Hi Jean-Marc,

I think your problem is coming from the unexpected new lines printed
in the middle of the output of your scripts... - don't know if AIX
iostat is doing so, or your script itself, but most of "multi-line"
stat commands are using an "empty line" as output iterations
separator..

so, if you can find a way change it via your script (have empty line
only on the end of each iteration) -- then it's ok... - if not, then
you have to delete all empty line on fly and find any other pattern
which will match in your output the "separator" and declare it during
Add-On creation instead of the empty line....

NOTE: in your mod2 script you also should take care about a time
interval and not divide your data always by 5 ;-)

Rgds,
-Dimitri

>> > email to dimstat+u...@googlegroups.com <javascript:>.

>> > To post to this group, send email to dim...@googlegroups.com

>> <javascript:>.

[jms667]

Hi Dimitri,

Empty lines are part of the output of the AIX iostat command.
I tried to declare a new separator (each iteration ends with *------*) into Add-On creation page, but the problem still remains.
Pretty busy these days, but I will try soon to modify my script in oder to process data output on the fly in only one script (and avoid to pipe the output from one script into an other).
When it will be done, I will come back here, and hopefully with good news :-)
thanks agin for dimstat and for your support.

Regards,
JM.