Performance metrics

[Ioannis Stoilis]

Hi,

I am evaluating pysphere for performance metric collection via Python,
but the getting started wiki page is more focused to the
administration of the VMs.

Can you give a small example on how to do it?

Ideally, I'd like something that returns a host list, their available
performance metrics and their values.

Thank you in advance!

[Seba]

Hi!
I'm still writing the documentation and haven't reached that part
yet. I'm on holidays until Monday. I'll provide you with an example
first thing on Tuesday.

Regards,

Sebastián

[Ioannis Stoilis]

Thank you, looking forward for it!

--
Ioannis Stoilis

[Guo-Wei Su]

Hi,

  I have same problem how to use performance_manager object?

[Seba]

Hi Guo,

First of all I want to apologize for the delayed answer.

First of all, I encourage you to use the current trunk version for
this example to work. If you don't know how to download and install
that version, let me know and I can provide you with a src
distribution or a windows executable.

#as always we first have to login:

from pysphere import *
s = VIServer()
s.connect("yourhost", "username", "password")

# then retrieve the available hosts
hosts = s._get_hosts()

#this is a dictionary of host names an managed object references. Eg.
{'192.168.100.1': 'host-5583', '192.168.100.2': 'host-185', ...}

#now instance the performance manager

pm = s.get_performance_manager()

#to retrieve the available perf metrics from, for example, the first
host you run
pm.get_entity_counters(hosts['192.168.100.1'])

#This will return a long list of counter names and their ids:
{'actav1': 265,
'actav15': 271,
'actav5': 268,
'active': 25,
'activewrite': 316,
.....
'used': 14,
'utilization': 305,
'vmmemctl': 82,
'write': 325,
'zero': 33}

#Now to get the values of the one you are interested in, for example,
'used', 'write', and 'zero', you should run
statistics = pm.get_entity_statistic(hosts['192.168.100.1'],
[14,325,33])

#that will return a list of EnitityStatistics objects, to display the
values you might run:
for sta in statistics:
print sta.counter
print sta.description
print sta.value
print sta.time

Hope this helps,

Regards

Sebastian

[Seba]

Guo,
Where I wrote:
hosts = s._get_hosts()

I meant:

hosts = s.get_hosts()

[Ioannis Stoilis]

It works very nice, but I have the following issue: The metrics don't
seem to come with a unit.

The viperformace Perl lib I am using(and want to replace) returns
something like this:
Counter: Core utilization
Instance :
Description: CPU utilization of the corresponding core (if
hyper-threading is enabled) as a percentage during the interval (A
core is utilized, if either or both of its logical CPUs are utilized)
Units: Percent
Sample info : 20,2011-12-19T02:21:40-07:00,20,2011-12-19T02:22:00-07:00
Value: 871,642

Which is very useful, as I am appending them to the metric name before
publishing, so they can make some sense without lookinto into docs and
specs.

Do you think you can export that piece of information as well?

--
Ioannis Stoilis

[Seba]

Hi Ioannis!

You are right, I missed unit information completely. I've just added
it to pysphere, so please check the trunk version.
Besides unit info I also added group info (for example CPU, Memory,
etc) where the counter belongs to.

So when you did:

for sta in statistics:
print sta.counter
print sta.description
print sta.value
print sta.time

Now you can also do:

for sta in statistics:
print sta.counter
print sta.description
print sta.value
print sta.time

print sta.group #e.g. 'mem'
print sta.group_description #e.g. 'Memory'
print sta.unit #e.g. 'kiloBytes'
print sta.unit_description #e.g. 'KB'

Hope this helps!

Sebastian Tello

On Dec 19, 2:28 pm, Ioannis Stoilis <ioan...@stoilis.gr> wrote:
> It works very nice, but I have the following issue: The metrics don't
> seem to come with a unit.
>
> The viperformace Perl lib I am using(and want to replace) returns
> something like this:
> Counter: Core utilization
> Instance :
> Description: CPU utilization of the corresponding core (if
> hyper-threading is enabled) as a percentage during the interval (A
> core is utilized, if either or both of its logical CPUs are utilized)
> Units: Percent
> Sample info : 20,2011-12-19T02:21:40-07:00,20,2011-12-19T02:22:00-07:00
> Value: 871,642
>
> Which is very useful, as I am appending them to the metric name before
> publishing, so they can make some sense without lookinto into docs and
> specs.
>
> Do you think you can export that piece of information as well?
>

[Ioannis Stoilis]

Works like a charm, thank you :)

--
Ioannis Stoilis

[Davim]

I've been reading this
http://pubs.vmware.com/vsphere-50/topic/com.vmware.wssdk.apiref.doc_50/vim.PerformanceManager.html
and it seams that there is a 'utilization' for CPU and the same for
memory, how do I get them both? if I use:

statistics = pm.get_entity_statistic(h,['utilization'])

It returns the CPU utilization in percentage for each CPU core, how do
I get the memory utilization?

and the

[Davim]

Ok I've managed to get the IDs I need using:

from pysphere import *
server = VIServer()
server.connect("10.39.31.115", "Administrator", "wit$admin$09")
print 'Connected to ' + server.get_server_type() + ' ' +
server.get_api_version()

hostlist = server.get_hosts() #returns dictionary with host MORs
and names
print 'Available Hosts:'
pprint(hostlist);
print '\n'

#now instance the performance manager

pm = server.get_performance_manager()

#to retrieve the available perf metrics

for h in hostlist:
try:
refresh_rate =
pm.query_perf_provider_summary(h).RefreshRate
metrics = pm.query_available_perf_metric(h,
interval_id=refresh_rate)
counter_obj = pm.query_perf_counter([metric.CounterId for
metric in metrics])
for c in counter_obj:
print c.GroupInfo.Key + ' ' + c.NameInfo.Key + ' ' +
c.RollupType + ' : ' + str(c.Key)
except Exception, ex:
print ex

server.disconnect()

And then instead of using the Counter names I use the Counter IDs
discovered with the code above.

Could you modify the get_entity_statistic method so one could specify
the counter group and the counter name insted of just the counter
name, since there are multiple counters with the same name in
different groups.

regards,
Luis Davim

On Jan 31, 11:28 am, Davim <luis.da...@gmail.com> wrote:
> I've been reading thishttp://pubs.vmware.com/vsphere-50/topic/com.vmware.wssdk.apiref.doc_5...

[Seba]

Hey Luis,

     Thanks for noticing that, I thought those name keys were unique but clearly they aren't.

Please check trunk (revision 57), I've changed the keys in the dictionary returned by "get_entity_counters" (which is internally called by "get_entity_statistic") from CounterName to GroupName.CounterName.

I believe now they should be unique. E.g:

>> pm = s.get_performance_manager()

>> pm.get_entity_counters(h)

{'cpu.idle': 13,

 'cpu.reservedCapacity': 9,

 'cpu.usage': 2,

  ...

 'disk.usage': 101,

  ...

 'mem.state': 60,

 'mem.swapin': 280,

 'mem.usage': 16,

  ...

 'net.usage': 115,

  ...

 'rescpu.samplePeriod': 285,

 'sys.resourceCpuUsage': 260,

 'sys.uptime': 122}

>> pm.get_entity_statistic(h, "net.usage")

#same than pm.get_entity_statistic(h, ["net.usage"])

# or pm.get_entity_statistic(h, 115)

[<host-444:usage:Usage::15:kiloBytesPerSecond:2012-01-31 12:23:46.782000>]

Hope it helps,

Regards,

Sebastian.

2012/1/31 Davim

[Seba]

I've just seen this new message of yours. Already done!

2012/1/31 Davim

[Davim]

Wow. That was fast :)

Thanks a lot.

> >http://pubs.vmware.com/vsphere-50/topic/com.vmware.wssdk.apiref.doc_5...

[rabinnh]

Seba,

Just curious. If I submit a bunch of counters, the object that is
returned does not include the same group name. For example if I
fetch:

net.usage

What I get back is

MOR: ha-host
Counter: usage
Group: Network
Description: Usage
Instance:
Value: 0
Unit: KBps
Time: 2012-02-12 23:40:18.839569

MOR: ha-host
Counter: usage
Group: Network
Description: Usage
Instance: vmnic0
Value: 0
Unit: KBps
Time: 2012-02-12 23:40:18.839617

The problem is that I have to manually map the "net." part to the
group "Network"., although the "Counter" property is fine. It doesn't
make the code very adaptable. Is there any way to get back the same
"key" I used to get counter, or at least name the group the same.

Because fetching counters has pretty high latency, I tend to group
them together. When I get the response, I attempt to map it to the
counters in the request. I am finding it very difficult to do that.

Rick

[Seba]

Hi Richard,

  What is shown there is the group (long) description, but you can get the group key too.

for stat in  pm.get_entity_statistic(h, "net.usage") :

    print stat.group_description #prints "Network"

    print stat.group #prints "net"

    print stat.description #prints "Usage"

    print stat.counter #print "usage"

Hope that helps.

Regards,

Seba

2012/2/12 rabinnh

[rabinnh]

Seba,

Understand that. But the problem is this; let's say that you submit
10 metrics using the strings and you get a valid return. You need to
map the values back to the sttrings that you used to identify the
counters when you made the call if you want to use the values in any
sort of calculation.

I understand from looking at the code what you are doing; if people
submit labels you convert them into the counter IDs, fetch the values,
and return them.

What I propose is that you simply ensure that you return the counter
ID as well, as I have done in this modified viperformance.py file. I
marked the changes with "#RB". Below are the small changes that I
made.

At least this way if the caller wishes to, they can convert the
strings to counter IDs prior to the call and make a dictionary. Then
when the call returns they can use the returned counter ID to map back
to the strings.

Rick

class EntityStatistics:
# RB
def __init__(self, mor, counter_name, counter_desc, counter_key,
group_name, group_desc,
unit_name, unit_desc, instance_name,
value,time_stamp):
self.mor = mor
self.counter = counter_name
self.description = counter_desc
# RB
self.key = counter_key
self.group = group_name
self.group_description = group_desc
self.unit = unit_name
self.unit_description = unit_desc
self.instance = instance_name
self.value = value
self.time = time_stamp
def __str__(self):
return "MOR: %s\nCounter: %s\nGroup: %s\nKey: %s\nDescription:
%s\n" \
"Instance: %s\nValue: %s\nUnit: %s\nTime: %s" % (
self.mor, self.counter,
self.group_description,
# RB
self.key,self.description,
self.instance, self.value,
self.unit_description,
self.time)
# RB
def __repr__(self):
return"<%(mor)s:%(counter)s:%(description)s:%(key)s" \
":%(instance)s:%(value)s:%(unit)s:%(time)s>" %
self.__dict__


class PerformanceManager:

def __init__(self, server, mor):
self._server = server
self._mor = mor

def _get_counter_info(self, counter_id, counter_obj):
"""Return name, description, group, and unit info of a give
counter_id.
counter_id [int]: id of the counter.
counter_obj [list]: An array consisting of performance
counter information for the specified counterIds."""
for c in counter_obj:
if c.Key == counter_id:
# RB
return (c.NameInfo.Key, c.NameInfo.Label, c.Key,
c.GroupInfo.Key,
c.GroupInfo.Label, c.UnitInfo.Key,
c.UnitInfo.Label)
# RB
return None, None, None, None, None, None, None

[rabinnh]

BTW, what I was saying is that using get_entity_statistic, my stat
object has a "group" attribute that returns "Network". My post shows
"print stat".

On Feb 13, 10:12 am, Seba <argo...@gmail.com> wrote:

[rabinnh]

I see what you are talking about. Your "__str__" function doesn't
display all the information. Stupid of me . . .

Thanks.

On Feb 13, 10:12 am, Seba <argo...@gmail.com> wrote:

[Seba]

Rick,

    You are right though. Even you can map them using stat.group and stat.counter, the object doesn't hold the counter key.

So there would be no easy way to match the object to the metric requested if using the counter ids instead of the string group.name notation.

I added a counter_key attribute to EntityStatistics in trunk revision 61.

So you can use counter_key, or  counter + "." + group to match de results back depending if you are passing counter ids or the string notation.

for stat in pm.get_entity_statistic(h, [336, 'sys.uptime', 331])

    print stat.counter_key

    print "%s.%s" % (stat.counter, stat.group)

Thanks,

Seba.

2012/2/13 rabinnh

[Antonio Ken Iannillo]

Hi,

I tried your code for getting performance metrics from a single Esxi host.

I had no problem in getting the performance manager, but when i get the entity counters i have:

>>> pm.get_entity_counters(hosts['ha-host'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pysphere/vi_performance_manager.py", line 117, in get_entity_counters
    sampling_period = self._check_and_get_interval_by_id(entity, interval)
  File "/usr/local/lib/python2.7/dist-packages/pysphere/vi_performance_manager.py", line 197, in _check_and_get_interval_by_id
    summary = self.query_perf_provider_summary(entity)
  File "/usr/local/lib/python2.7/dist-packages/pysphere/vi_performance_manager.py", line 299, in query_perf_provider_summary
    qpps_entity.set_attribute_type(entity.get_attribute_type())
AttributeError: 'str' object has no attribute 'get_attribute_type'

Any clue? There is another way to get the entity host in another way when you connect directly to an esxi host?