Get datastore stats

[Mike Zupan]

Looking to bring my graphs outside of vsphere and I know of get_datastore() but can't figure out how to get the basic summary out of each datastore. Like total/used/free of space on the LUN

thanks!

[krisdigitx]

so far i can only see these attributes

datastore.numberWriteAveraged: 655361
datastore.totalWriteLatency: 655365
datastore.totalReadLatency: 655364
datastore.write: 655363
datastore.datastoreIops: 655367
datastore.numberReadAveraged: 655360
datastore.sizeNormalizedDatastoreLatency: 655366
datastore.read: 655362


storagePath.Read and storagePath.write are also good for graphing.



-krisdigitx

[Mike Zupan]

Maybe I'm missing it but I don't see those attributes in the object

#!/usr/bin/env python

from pysphere import *

server = VIServer()

server.connect("1.2.3.4", "admin", '****')

for ds,name in server.get_datastores().items():

    print dir(ds)

[krisdigitx]

try this:

#!/usr/bin/env python26
import os
import sys
import subprocess
from pysphere import *
def ESXconnect():
    server = VIServer()
    server.connect("xx.xx.xx.xx","root","xxxxxx")
    hosts = server.get_hosts()
    print hosts
    host = [k for k,v in hosts.items() if v == 'esx-server-name'][0]
    pm = server.get_performance_manager()
    for key,value in pm.get_entity_counters(host).items():
        print key + ": " + str(value)
    stats = pm.get_entity_statistic(host, [65599,65619])
    for stat in stats:
        print stat.description
        print stat.value
def main():
    ESXconnect()
if __name__ == "__main__":
    main()

[Mike Zupan]

That seems to get it on a per server.. but what if I want to see what
datastore on whole is doing since many esx servers share the same
datastore?

[Mike Zupan]

So is there no way to do this easily?

[Seba]

Hey!
 Sorry I couldn't reply before, I've been quite busy lately.

You can used the performance manager over a wide range of managed
entities, not only hosts. However there are some issue when retrieving
performance data from datastores (see http://communities.vmware.com/thread/286747,
http://communities.vmware.com/message/1623870#1623870)

I've changed some code to skip a validation (when vsphere sdk says
that not current metrics are available for a datostore) however I
can't get any values on my environment. But you could check if this
works on yours (you'll need the latest trunk revision)

from pysphere import VIServer

server = VIServer()
server.connect(HOST, USER, PASSWORD)

pm = server.get_performance_manager()

for ds_mor, name in server.get_datastores().items():
    print "ENTITY STATISTICS FOR DATASTORE", name
    counters = pm.get_entity_counters(ds_mor)
    print "  Available counters:", ",".join(counters.keys())
    stats = pm.get_entity_statistic(ds_mor, counters.values())
    if not stats:
        "  No perf values could be obtained."
    for stat in stats:
        print " *", stat

server.disconnect()

On the other hand you might find the values you need from the
datastore object properties directly, For instance:

from pysphere import VIServer, VIProperty

server = VIServer()
server.connect(HOST, USER, PASSWORD)

for ds_mor, name in server.get_datastores().items():
    props = VIProperty(server, ds_mor)
    print "DATASTORE:", name
    print "  Type:", props.summary.type
    print "  Capacity:", props.summary.capacity
    print "  Free space:", props.summary.freeSpace
    if hasattr(props.summary, "uncommitted"):
        print "  Uncommited:", props.summary.uncommitted

server.disconnect()


You can get any property following the path names described in the
vSphere SDK api reference (http://www.vmware.com/support/developer/vc-
sdk/visdk41pubs/ApiReference/vim.Datastore.html)
Keep in mind that some properties might not be present (the ones with
the red star mark)

Regards,

Sebastian.

[Mike Zupan]

Thanks for the reply and examples this is a great resource in python. The second example worked better for me and is a lot quicker to pull the data also. 

[PyFiend]

I've been toying around with trying to get VMWare stats back with
Python. Funny thing is everything appears to work just fine with the
exception of server.get_datastores(), which seemingly doesn't return
anything. Could this be due to the version of ESX being used (in my
case it's 5.0)?

Speaking with the VMWare admin, he's keen on having us try "PowerCLI"
instead, but looking at the code (http://vsential.com/2010/09/
datastore-size-reporting-via-powercli-script/), I doubt the results
will be any better. We're probably going to give it a go soon anyway,
but if that doesn't return the datastore readings either, then I'm
inclined to say that the readings just aren't available from the
VMWare side.

Also, could you clarify what you meant with "You can get any property
following the path names"? Sorry, I'm not terribly experienced with
Python and even less with VMWare's SDK API.

On May 28, 5:45 pm, Seba <argo...@gmail.com> wrote:
> Hey!
>  Sorry I couldn't reply before, I've been quite busy lately.
>
> You can used the performance manager over a wide range of managed
> entities, not only hosts. However there are some issue when retrieving

> performance data from datastores (seehttp://communities.vmware.com/thread/286747,http://communities.vmware.com/message/1623870#1623870)

[Tim Conrad]

powercli is pretty neat. It obviously uses power shell, which seems like a mixture of OO perl/python and bash shell scripting.  You end up doing queries and piping objects, which turns out to be amazingly powerful. But, it's a bit quirky, like many MS products.

Here's a script I wrote in a few minutes to get the datastores and create a CSV with that data. It doesn't do any error checking, but it works:  https://gist.github.com/2836910 

If you do anything with powercli, get powergui @ powergui.org. They have a nice editor that works really well. The package comes with a bunch of canned vm reports, as well. The documentation for powercli is really good with many examples of what you need. And with powergui, it's very similar to using any other IDE, where you can simply type '$ds." and you'll get a list of items to complete.

I found getting performance metrics back to be pretty slow and cpu intensive using pysphere. I ended up re-writing my project using vijava, which is considerably leaner. I wrote something to go to a vcenter and grab the stats and then shove them to a graphite for graphing of all of the stats. Here's a bitbucket link for that, it's not completed/fully packaged yet, though: https://bitbucket.org/timconradinc/vmstats/src

Tim

[PyFiend]

Oh wow, thanks for that! :)

I'll probably start digging around with PowerCLI in the next couple of
days and see how it goes, will definitely be referring to the info you
sent. I'll post back once any progress has been made.

Thanks again for the detailed info, appreciate it.

[PyFiend]

Quick update, tried PowerCLI 5, then 4, but still no luck unfortunately. I've checked with the VM admin, our user definitely has full admin rights in order to test.
It's the weirdest thing, it's really as if the datastore info just isn't available at all. We can get things like read and write average, but when it comes to capacity (used, free etc.) it just doesn't return anything at all.

At this stage I'm thinking whatever the problem is must be on the VMWare side of things, but whether there's a checkbox that needs ticking somewhere I have no idea. I'm going to try chasing this up further with VMWare and see if they maybe have something more on the subject, will post back any update once I receive anything.

Thanks again for the help with this, appreciate it.

On Wednesday, 30 May 2012 17:21:40 UTC+2, Tim Conrad wrote:

powercli is pretty neat. It obviously uses power shell, which seems like a mixture of OO perl/python and bash shell scripting.  You end up doing queries and piping objects, which turns out to be amazingly powerful. But, it's a bit quirky, like many MS products.

Here's a script I wrote in a few minutes to get the datastores and create a CSV with that data. It doesn't do any error checking, but it works:  https://gist.github.com/2836910 

If you do anything with powercli, get powergui @ powergui.org. They have a nice editor that works really well. The package comes with a bunch of canned vm reports, as well. The documentation for powercli is really good with many examples of what you need. And with powergui, it's very similar to using any other IDE, where you can simply type '$ds." and you'll get a list of items to complete.

I found getting performance metrics back to be pretty slow and cpu intensive using pysphere. I ended up re-writing my project using vijava, which is considerably leaner. I wrote something to go to a vcenter and grab the stats and then shove them to a graphite for graphing of all of the stats. Here's a bitbucket link for that, it's not completed/fully packaged yet, though: https://bitbucket.org/timconradinc/vmstats/src

Tim

[PyFiend]

Late update on this, but turns out the user we've been given to test didn't have the proper rights to query what we needed. My Python script is working great now, but just stuck with one last issue:

Is there perhaps some way I can print the Datacentres as well? My script currently looks as follows:

#!/usr/bin/env python

from pysphere import VIServer, VIProperty

def main():


    server = VIServer()
    server.connect("""HOST""", """USER""", """PASSWORD""")

    for ds_mor, name in server.get_datastores().items():
        props = VIProperty(server, ds_mor)

        if hasattr(props.summary, "uncommitted"):
            print "DATASTORE: "+str(name)+", Type: "+str(props.summary.type)+", Capacity: "+str(props.summary.capacity)+", Free space: "+str(props.summary.freeSpace)+", Uncommited: "+str(props.summary.uncommitted)
        else:
            print "DATASTORE: "+str(name)+", Type: "+str(props.summary.type)+", Capacity: "+str(props.summary.capacity)+", Free space: "+str(props.summary.freeSpace)+", Uncommited: N/A"

    server.disconnect()

if __name__ == '__main__':
    main()

The above produces something like this:

DATASTORE: itsimdhdat65-0257g5, Type: VMFS, Capacity: 573915004928, Free space: 142895742976, Uncommited: 0
DATASTORE: itsimdhdat67-0257g5, Type: VMFS, Capacity: 549487378432, Free space: 248160190464, Uncommited: 1073741824
DATASTORE: itsimdhdat68-0257g5, Type: VMFS, Capacity: 549487378432, Free space: 125776691200, Uncommited: 0
DATASTORE: itsimdhdat60-0257g5, Type: VMFS, Capacity: 573915004928, Free space: 433727733760, Uncommited: 0


What I'm hoping to get is something like this:

DATACENTRE: PlaceA, DATASTORE: itsimdhdat65-0257g5, Type: VMFS, Capacity: 573915004928, Free space: 142895742976, Uncommited: 0
DATACENTRE: PlaceA, DATASTORE: itsimdhdat67-0257g5, Type: VMFS, Capacity: 549487378432, Free space: 248160190464, Uncommited: 1073741824
DATACENTRE: PlaceB, DATASTORE: itsimdhdat68-0257g5, Type: VMFS, Capacity: 549487378432, Free space: 125776691200, Uncommited: 0
DATACENTRE: PlaceB, DATASTORE: itsimdhdat60-0257g5, Type: VMFS, Capacity: 573915004928, Free space: 433727733760, Uncommited: 0



Is there hopefully an easy-ish way to do this?


Cheers

[Seba]

This should do it

Regards,

Seba.

from pysphere import VIServer, VIProperty

def main():

    server = VIServer()

    server.connect("""HOST""", """USER""", """PASSWORD""")

    ds_by_dc = {}

    for dc_mor, dc_name in server.get_datacenters().items():

        p = VIProperty(server, dc_mor)

        for ds in p.datastore:

            ds_by_dc[ds._obj] = dc_name

    for ds_mor, name in server.get_datastores().items():

        props = VIProperty(server, ds_mor)

        if hasattr(props.summary, "uncommitted"):

            print "DATACENTRE: %s ,DATASTORE: %s, Type: %s, Capacity: %s, " \

                  "Free space: %s, Uncommited: %s" % (ds_by_dc[ds_mor], name, 

                    props.summary.type, props.summary.capacity, 

                    props.summary.freeSpace, props.summary.uncommitted)

        else:

            print "DATACENTRE: %s ,DATASTORE: %s, Type: %s, Capacity: %s, " \

                  "Free space: %s, Uncommited: N/A" % (ds_by_dc[ds_mor], name, 

                    props.summary.type, props.summary.capacity, 

                    props.summary.freeSpace)

    server.disconnect()

if __name__ == '__main__':

    main()

[PyFiend]

Hi Seba


This works absolutely beautifully, thanks for this.


Cheers