Advise on selecting correct values to be analized

[bianca...@gmail.com]

Hello, 

I am using Dim stat for the first time and I would like to ask for your advise regarding what values I should choose when I start the monitoring/collecting of data and what values to pick when I do the analysis using Graph view.

I am starting an application on a pc, and I would like to monitor for that specific java process the CPU usage, Memory and IO.

When I choose I/O stat, for example, I get "No data to draw" message, do I have to make some settings somewhere so that I can collect also I/O statistics? Or what does the message mean?

Thank you very much in advance for your support!

My kind regards,

Bianca

 

[Dimitri]

Hi Bianca,

first of all: on which OS you're running your Java application ?..

let's suppose it's Linux, then :
- you can see "java" process RAM and CPU usage via "Linux ProcLOAD"
or "Linux psSTAT"
- the difference between ProcLOAD and psSTAT is that the first one
is grouping (SUM) all the stats for all processes having the same
name, while the second one is showing the stats "per process" and to
each process can be identified by the name composed of <process
name>-<PID>, like in the following output example :

PNAME-PID UsrTM SysTM CPU% MinF MajF PRI NI Thr
VmSIZE VmLCK VmRSS VmData VmSTK VmEXE VmLIB VmPTE CUsrTM CSysTM
init-00001 0.00 0.00 0.0 0 0 20 0 1
23596 0 1644 332 88 140 2480 64 0.00 0.00
ksoftirqd/6-00029 0.00 0.01 0.0 0 0 20 0 1
0 0 0 0 0 0 0 0 0.00 0.00
ksoftirqd/20-00085 0.00 0.01 0.0 0 0 20 0 1
0 0 0 0 0 0 0 0 0.00 0.00
...

so, you can find then from collected stats :
- CPU% : global CPU usage by the process(es)
- UsrTM : User CPU Time
- SysTM : System CPU Time

NOTE: the last 2 stats are representing the corresponding time in
seconds and this time is related to the time interval you're using for
your stats collect and not to the number of CPU/cores you have on your
system (for ex. if you started your collect with 10sec time interval
and see your Java process using 5sec in UsrTM and 1sec in SysTM =
means you're using 6sec of CPU Time of a single CPU core (so, 60% of
this CPU core is used by your Java process), while if UsrTM growing to
30sec then you're using 3 CPU cores 100% (30sec CPU Time during
10sec), and so on..

for RAM usage by your process: VmSIZE is probably the most important,
while you may also be interested by RSS and others..

for IO usage by your process: there is no "easy way" (psSTAT is not
reporting them yet), so you can find some info from "/proc/PID/io",
the only problem is that the reported "IO" are generic (it can be any
read()/write() -- file, IP socket, etc.) -- so, this could give an
idea at least about the "IO" activity on the process, but then it's up
to you to correlate it with real I/O load on your system..

for general I/O stats on Linux you can get all the info from "Linux
iostat (v.10) Statistic(s)" Add-On.

Well, this all is supposing you're using Linux.. -- if you're using
another UNIX, the is similar, but different ;-)

Rgds,
-Dimitri

> --
> You received this message because you are subscribed to the Google Groups
> "dim_STAT" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to dimstat+u...@googlegroups.com.
> To post to this group, send email to dim...@googlegroups.com.
> Visit this group at https://groups.google.com/group/dimstat.
> For more options, visit https://groups.google.com/d/optout.
>

[manesa bianca]

Hello Dimitri,

Thank you very much for replying to my post so quickly, you are very kind.

I am using x86_64 GNU/Linux.

I am able to get statistics for CPU% and VM size using LpsSTAT. Also, for I/O I am using Lio_STAT_v10, choose the partition ( ex. "vda" ) then rKB/s and wKB/s; and for these is ok, I get data to have the graph draw. 

I am curios, for example, why I cannot use  "psSTAT", where I could filter after a certain PID and get CPU% and MEM% ( is this the same as CPU% and VM size from LpsSTAT ? ). For psSTAT I always get "No data to draw", does this mean I need to activate this stat somewhere? In access file I looked and it's there...seems to be activated; could I do something else to activate this STAT also?

Thank you very much for your response.

Have a good day!

Regards,

Bianca

> email to dimstat+unsubscribe@googlegroups.com.

> To post to this group, send email to dim...@googlegroups.com.
> Visit this group at https://groups.google.com/group/dimstat.
> For more options, visit https://groups.google.com/d/optout.
>

--
You received this message because you are subscribed to a topic in the Google Groups "dim_STAT" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dimstat/aVzYGfi5b9w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dimstat+unsubscribe@googlegroups.com.

[Dimitri]

Hi Bianca,

regarding LpsSTAT :
- when you start a new STAT collect: be sure you selected the LpsSTAT checkbox
- once your collect is started: you can see in your STAT-service log
messages that LpsSTAT was started
- then from the Main Page go to Analyze
- on Analyze Page: select Single-Host Analyze, Active ONLY, Show
STAT(s) Status; then click on [Analyze]
- on Analyzer Page you'll see now your active STAT collect(s) with
corresponding STAT(s) Status
- look on the STAT(s) Status column: you should find there LpsSTAT:N
value, where N will be the number of currently collected rows for
LpsSTAT
- you can reload the page in your browser to refresh the data: be
sure N value is growing (means your LpsSTAT data are collecting)

if you don't see the LpsSTAT:N in Status column for your collect =>
then something was missed in the previous steps, and you need to
recheck it once again..

once you see LpsSTAT:N in your Status, you can then select your
collect and click on the [LpsSTAT] button.
then, on LpsSTAT Analyzer page :
- select your process from the list
- select CPU% (or VMSIZE) from Graph values
- click on [Go] and see corresponding graphs !

NOTE: processes are starting an ending, their PID are changing, so
pretty quickly it becomes painful to follow them.. -- so, you may then
instead of selecting them from the list just provide the process name
pattern. For ex. if you're looking for per-process stats for all your
"java" programs running you may check "Use Select Pattern" option for
PNAME (process name) and enter "java" into the field (or any other
regular expression you want) => this will then always show any "java"
processes stats for the corresponding time interval (if there was no
such processes during selected time: then you'll have just no data
shown (so better to try first with processes constantly running on
your system to check the feature, etc.))

Rgds,
-Dimitri

>> > email to dimstat+u...@googlegroups.com.

>> > To post to this group, send email to dim...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/dimstat.
>> > For more options, visit https://groups.google.com/d/optout.
>> >
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "dim_STAT" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/
>> topic/dimstat/aVzYGfi5b9w/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> dimstat+u...@googlegroups.com.

>> To post to this group, send email to dim...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/dimstat.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --

> You received this message because you are subscribed to the Google Groups
> "dim_STAT" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to dimstat+u...@googlegroups.com.

[manesa bianca]

Hi Dimitri,

I have successfully used DimSTAT so far, but I would like to ask you 2 things:

1. How are the collections retrieved from STATservice ( client ) towards dimSTAT server? does the server make an ssh connection with the clients...how exactly and what does it need to retrieve the counters? doe an IP connection that doesn't allow ssh would work?

2. If I have collected smth for 8 hours, the I stopped all collectors and started again after 1-2 days. How can I still visualize the data taken in those 8 hours...in case the collectors were stopped in the meanwhile ?

Thank you very much in advance for your reply.

Kind regards,

Bianca

>> > email to dimstat+unsubscribe@googlegroups.com.

>> > To post to this group, send email to dim...@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/dimstat.
>> > For more options, visit https://groups.google.com/d/optout.
>> >
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "dim_STAT" group.
>> To unsubscribe from this topic, visit https://groups.google.com/d/
>> topic/dimstat/aVzYGfi5b9w/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to

>> dimstat+unsubscribe@googlegroups.com.

>> To post to this group, send email to dim...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/dimstat.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "dim_STAT" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to dimstat+unsubscribe@googlegroups.com.

> To post to this group, send email to dim...@googlegroups.com.
> Visit this group at https://groups.google.com/group/dimstat.
> For more options, visit https://groups.google.com/d/optout.
>

--
You received this message because you are subscribed to a topic in the Google Groups "dim_STAT" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dimstat/aVzYGfi5b9w/unsubscribe.

To unsubscribe from this group and all its topics, send an email to dimstat+unsubscribe@googlegroups.com.

[Dimitri]

Hi Bianca,

the following are the answers to your questions:

1) by default STAT-service is listening on the IP port 5000 (but you can change it by editing the STAT-service script on your machine) -- when you're starting a new stats collect via Web interface of dim_STAT there will be a fork of several scripts/processes running in background and connecting to corresponding IP port, reading the output data from the IP socket and sending them to dim_STAT database. So, you don't need SSH for this, the IP port should be just accessible from the dim_STAT server. Let me know if you need any more details..

Also, you can find other info about STAT-service from User's Guide: http://dimitrik.free.fr/UserGuide/MultiPage/Report_1_page4.html#note_4963

NOTE: just a reminder about the new v5 of STAT-service is available for all supported platforms, and this version is allowing much more flexibility (to be installed in any directory, etc.) -- for Linux the full 64bit version is available from here: http://dimitrik.free.fr/STATsrv-v5-Linux-x64.tgz

2) in every Analyze page you can select whatever time interval you're interesting (between date & time) -- however I'd say such a way to select time intervals very quickly becomes painful ;-)) -- much more simple is to add LOG Messages to the given time periods and then quickly select a needed time interval by Messages (between Messages, after a Message, etc.) -- for the same reasons whatever graph you're seeing on Analyze page you'll always find on the bottom of the page an entry field to add a LOG Message corresponding to the graph interval (begin / end of graph, % or exactly time, etc.). As well there is a special Admin page fully dedicated to LOG Messages management (you can manually add them for any time period to any stat collect, change them, copy, etc. - and even load from a file ;-))

while also keep in mind that you can add LOG Messages automatically directly from the host from which the stats are collected -- see the following chapter: http://dimitrik.free.fr/UserGuide/MultiPage/Report_1_page10.html#note_4987

again, let me know if you need any more details..

Rgds,

-Dimitri

[manesa bianca]

Hi Dimitri,

Thank you so much for all the details, it is clear now.

One more question, in case I install Dim-STAT server on a machine on which I cannot access the GUI intf. to start the counters...is there a possibility to start them from a command line and them take the DB, import it on another machine and analyze the data with DIM-STAT from this other machine where I have the possibility to access it's GUI so I can generate the graphs ?

I am in the situation where I need Dim-STAT on a private network but I cannot access it's GUI...is there any workaround through this?

Thank you.

Kind regards,

Bianca

[Dimitri]

Hi Bianca,

the situation with "paranoid" users and private networks is pretty common, and to be honest, I was many times surprised to see what kind of workarounds users are inventing to get their stuff done ;-)) => some are using CURL to send HTTP requests via shell scripts, some are involving dim_STAT CLI to generate live graphs, etc. etc. -- but, unfortunately, I cannot call such "workaround" simple to implement & use..

specially that once you have to deal with "sensible" environments, private network, high security, etc. => you're also dealing with a "human factor" (e.g. if there are any issues on the server - you're the fist to be accused because of your scripts/ open ports/ inter host communications/ and so on).. 

so, to manage such "sensible" cases I've added from a long time to dim_STAT a script-based tool which is called EasySTAT ;-)

  - EasySTAT is by default shipped within STAT-service package

  - and generally what you have to do is just deploy STAT-service on the host

  - then just start EasySTAT.sh script with corresponding options

  - this will collect you all requested stats into a given directory

  - once the collect is finished, you just copy it to the machine where you have your fully functional dim_STAT running

  - the directory will also automatically contain the scrip to load the data into your dim_STAT database

  - you may reedit the script to match your own preferences

  - then you just run the script and see your data loaded ;-))

advantages:

  - you don't need to install anything else to monitored servers except STAT-service

  - you can run EasySTAT via cron, etc. -- e.g. you don't need any human intervention

  - you don't need to open any special ports on the server, etc.

  - you always have a copy of "original" stat commands outputs, which can be reviewed in case of some doubts

  - you can script the whole process whenever you copy the obtained data yourself or users are sending them to you by email/ dropbox/ cloud/ etc..

  - and maybe the most important that any sysadmin may get a look on what your scripts are doing and what exactly the data you've collected, so everything remains transparent, and then everybody will be just happy ;-))

more details and HOWTO about EasySTAT you can find here:

  - http://dimitrik.free.fr/UserGuide/MultiPage/Report_1_page8.html#note_4979

NOTE: since STAT-service v5 usage of EasySTAT is yet more simple now - the list of stat to collect is moved to external file (.env-easystat) rather to be part of EasySTAT.sh script itself, so you just need to edit the ".env-easystat" file to get the list of stats you need (generally just comment in/out some lines in this file, or add few lines more if needed)..

Rgds,

-Dimitri