SMI-S Client Development - Need Inputs & Motivation

[Arun Anbalagan]

Dear Friends,

I have been pretty recently introduced to the SMI-S technology and currently in the process of understanding the same.

I work in a Data center and we are looking at the possibilities of using SMI-S to monitor and report various metrics related to the heterogeneous storage devices being used here. My understanding is that the storage vendors claiming to support SMI-S would provide the various APIs through their SMI-S provider, and we just need to develop a client application that can use these APIs to retrieve the metrics. Is my understanding correct?

I have taken up the task of development this client program, and exploring the options on how to proceed. I would be very grateful for any inputs on how should I get started. Primarily, I'm looking for an answer to the following questions:

- What are the prerequisites that I should have, before attempting any client code development?

- If I need to learn the various protocols, such as WEBM, CIM, etc., is there any good material available?

- How can I confirm whether SMI-S is capable of providing the metrics that I have been looking for? For example, Audit Security Logs, Firmware version, etc. I looked into the material at SNIA, but unfortunately, I couldn't easily find "the" one document that can provide a direct mapping to the interface that could fetch the information that I need. How can easily navigate to the list of classes and APIs to figure out which ones are the ones that I need?

- Are there any sample or prototype code available on client applications? And if so, where can I find them?  I guess, unless I get a chance to look at some samples, I might have a tough time achieving what I need.

- Any approximation on how much time would generally needed to develop such an SMI-S client application?

Thanks for all the help!!

Regards,

Arun.

[Gris Ge]

On Tue, Mar 07, 2017 at 06:54:08PM -0800, Arun Anbalagan wrote:
> - Are there any sample or prototype code available on client applications?
> And if so, where can I find them?  I guess, unless I get a chance to look at
> some samples, I might have a tough time achieving what I need.

Hi Arun Anbalagan,

Short sample for pywbem in python(I licensed them under public domain):

import pywbem
conn = pywbem.WBEMConnection('https://ip:5989',
('user', 'pass'), 'root/vendor_name_space', no_verification=True)
cim_pools = conn.EnumerateInstances('CIM_StoragePool')
print(cim_pools[0].items())

Long version:

https://github.com/libstorage/libstoragemgmt/tree/master/plugin/smispy
LGPL 2.1+ license. IANAL, but mere facts are in the public domain
even if those facts are embodied in expressive material that is covered by
copyright. Check with your lawyer before use these codes.

Extra notes based on my old experience on coding a NetApp REST API client and
above also SMI-S client: (sorry to be a buzz killer)

* It's not the one-for-all type of work. You still need to read vendor's
document and handling the differences between vendors' implementation of
SMI-S SPEC.

* IMHO, SMI-S performance is bad and coding is time consuming and
not fun due to the blur words in SMI-S standard SPEC.

Hope these helps.

Best regards.

BTW: This email is only my personal opinion and has NOTHING to do with
Red Hat.

--
Gris Ge

[Paul Robert Marino]

Also you can look at my project on github it's called lib-cim-perl
It's a pure Perl implementation and has a lot of practical documentation on how the queries functionally work that should apply to most implementations.

  Original Message  
From: f...@redhat.com
Sent: March 8, 2017 11:41 AM
To: aar...@gmail.com
Reply-to: smi-s-devel...@googlegroups.com
Cc: smi-s-devel...@googlegroups.com
Subject: Re: SMI-S Client Development - Need Inputs & Motivation

Hi Arun Anbalagan,

Long version:

Hope these helps.

Best regards.

--
Gris Ge

--
You received this message because you are subscribed to the Google Groups "SMI-S Developers Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to smi-s-developers-...@googlegroups.com.
To post to this group, send email to smi-s-devel...@googlegroups.com.
Visit this group at https://groups.google.com/group/smi-s-developers-group.
For more options, visit https://groups.google.com/d/optout.

[Jeffery K]

I will agree with Arun, it's not as "Generic" as you'd like it to be. I spent months developing a java application that would gather detailed performance data generically from SMI-S providers (par of my companies performance solution we sell) and even thought I made it as generic as i could, and used an external config/mapping file, to specify what metrics to get, so it would work with others, when I moved onto another vendor, I had to tweak some of the code, because certain assumptions didn't hold up between HP and EMC, per their use of the "Spec"

Jeffery Kuhn

Sightline Systems Corp. 

--
You received this message because you are subscribed to the Google Groups "SMI-S Developers Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to smi-s-developers-group+unsub...@googlegroups.com.
To post to this group, send email to smi-s-developers-group@googlegroups.com.

Visit this group at https://groups.google.com/group/smi-s-developers-group.
For more options, visit https://groups.google.com/d/optout.

--

Jeffery Kuhn
SightLine Systems Corp

[Rigger, James]

Hi Gris,

A couple of thoughts in response to your response since this is the first we have heard from you one these issues.

Performance is generally an issue associated with the implementation and can be compounded by the way in which information is retrieved from the implementation. For example, the client may ask for way more information than is necessary. Various profile views were created to facilitate the more common information that is retrieved and we see that folks are not implementing the use of views.

I assume the author community would more than welcome additional help in defining new views if they are needed and to understand the problems such that "blur" words could be refined to have more meaning.

The authors meet on Fridays at 10:00 Pacific of you could just send e-mail to the smitwg.

-----Original Message-----
From: smi-s-devel...@googlegroups.com [mailto:smi-s-devel...@googlegroups.com] On Behalf Of Gris Ge
Sent: Wednesday, March 08, 2017 8:54 AM
To: aar...@gmail.com
Cc: smi-s-devel...@googlegroups.com
Subject: Re: SMI-S Client Development - Need Inputs & Motivation

--
You received this message because you are subscribed to the Google Groups "SMI-S Developers Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to smi-s-developers-...@googlegroups.com.
To post to this group, send email to smi-s-devel...@googlegroups.com.

[Rigger, James]

Hi Jeffery,

 

We are glad to see that you are using SMI-S as part of your solution and I would like to speak with you more about it when we get a chance.  I see you are in Fairfax so I am assuming it is too late to speak with you today.

 

The performance profile has some particular challenges I that the profile had to be written to the lowest common denominator.  This is caused because there are various levels at which an implementation may be able to gather statistics just because the enterprise arrays have more capabilities than let’s say a low end array.  That being written there are ways to interrogate an implementation to determine what they do support so that your code can know if it should handle capabilities of a given array.

 

We hope you are looking at version 1.6.1 of the specification.  That is the newest and will be the most inclusive of the capabilities.  You can also look at the conformance testing results, www.snia.org/ctp/conformingproviers, to see of some of the particular functions are supported such as Performance Reporting Definition, Block IO Performance Reporting and Bulk retrieval of Performance Statistics.  If you do not see what you want then you  need to point your sales rep or partner rep and your customers to the page and ask why the array you are trying to work with does not support the function.  The vendors will not change unless the folks paying for the equipment ask for the changes.

 

Please feel free to send feedback to us as the goal is to improve the spec not just throw it out to the public and say good luck.

 

James

 

From: Jeffery K [mailto:jeffer...@sightlinesystems.com]
Sent: Wednesday, March 08, 2017 9:50 AM
To: smi-s-devel...@googlegroups.com
Cc: aar...@gmail.com
Subject: Re: SMI-S Client Development - Need Inputs & Motivation

 

I will agree with Arun, it's not as "Generic" as you'd like it to be. I spent months developing a java application that would gather detailed performance data generically from SMI-S providers (par of my companies performance solution we sell) and even thought I made it as generic as i could, and used an external config/mapping file, to specify what metrics to get, so it would work with others, when I moved onto another vendor, I had to tweak some of the code, because certain assumptions didn't hold up between HP and EMC, per their use of the "Spec"

To unsubscribe from this group and stop receiving emails from it, send an email to smi-s-developers-...@googlegroups.com.
To post to this group, send email to smi-s-devel...@googlegroups.com.

Visit this group at https://groups.google.com/group/smi-s-developers-group.
For more options, visit https://groups.google.com/d/optout.

 

--

Jeffery Kuhn
SightLine Systems Corp

[Arun Anbalagan]

Dear All,

Thank you very much for all your inputs. I can see that I have a long way to go, but the links and information that you have shared would hugely help me in making good progress. I'll keep working and keep you all posted on any issues or updates that I might come across during the development. Thanks again..

Regards,

Arun.

[Gris Ge]

On Wed, Mar 08, 2017 at 10:49:32PM +0000, Rigger, James wrote:
> Hi Gris,
Hi James,

>
> A couple of thoughts in response to your response since this is the first we
> have heard from you one these issues.
>
> Performance is generally an issue associated with the implementation and can
> be compounded by the way in which information is retrieved from the
> implementation. For example, the client may ask for way more information
> than is necessary. Various profile views were created to facilitate the

Yes. I do property filter, but that does not change the fact of bad
performance.
The bad performance I refer here is the complex workflow for retrieving
data.

Example on query disk info:

* Use profile register to check whether disk profile is supported.
* Use profile register to get vendor namespace.
* Enumerate CIM_DiskDrive.
* Disk size and sector size is in CIM_StorageExtent, I have to find
association of CIM_DiskDrive to it.
* To tell if disk is a spare disk, I need
CIM_StorageRedundancySet and CIM_IsSpare also.
* A lot other classes needed to tell the link speed and disk type.

It takes 10+ communication with SMI-S provider. That's why I call it bad
performance.
A good design would be 'disk-list-info' command and return all info you may
need for all disks.

> more common information that is retrieved and we see that folks are not
> implementing the use of views.
>
> I assume the author community would more than welcome additional help in
> defining new views if they are needed and to understand the problems such
> that "blur" words could be refined to have more meaning.

IMHO, the 'blur' cannot be fixed as SMI-S by design is to make storage array
vendors happy, not the client users, hence it has to be blur to allow every
storage array to fit in.
I have decided to move on and let my current SMIS code as it was.
Hopefully, swordfish could be better designed and implemented in the future.

Thank you for reaching out and your effort on SMI-S.

Best regard.

--
Gris Ge

[Jim Marshall]

Hi Arun,

 I'll take a stab at the questions not answered by Gris.

- What are the pre-reqs.

 Obviously need to know how to program in some language but other then that it's mainly reading the SMI-S specs to determine what information you need and how to obtain it.

- Need to learn protocols 

 You should never have to deal with the protocol, unless you want to. All available client libraries implement the protocol so you just simply make a call to get data and it does all the work.

- SMI-S capable of providing metrics you need

 See answer to last question below

- Sample code

 There are a myriad of client API's

 - Open Pegasus has a C/C++ based API

 - SBLIM has a Java based client (http://sblim.sourceforge.net/cim-client2-v2-doc/)

 - My company, WBEM Solutions, sells a Java based SDK (both SBLIM and our SDK use JSR48 so you should be able to switch between the two without code changes for the most part)

 - PyWBEM as Gris pointed out

 - I'm sure I missed a couple

- Does SMI-S support the metrics needed

 Each implementation is different so you would need to figure out if a 'profile' provides the data you need, you would then need to determine if the hardware you want to monitor supports that specific profile. For example the Block Services specification has a "Block Server Performance" profile.

You could also ask in this group what metrics you're looking and I'm sure someone in the group can tell you what profile provides it or if it's not available.

Good luck

Jim

[Paul Robert Marino]

This should be helpful with the queries
https://github.com/prmarino1/Lib-CIM-Perl/blob/master/LCP::Query.md#using-the-intrinsic-methods
T a lot of documentation there on exactly how CIM and SMI-S work and
even some example results with different options set.
The exact syntax is for the Perl API I wrote but the intrinsic methods
are strait our of DSP0200
http://www.dmtf.org/sites/default/files/standards/documents/DSP0200_1.4.0a.pdf
most of the WBEM servers only support the intrinsic methods at this time.
Particularly if you want to make vendor agnostic queries you will have
to look at what I call in the documentation "Query Modifiers" because
they tune how and if inheritances, references and instances of child
classes are returned which is needed to handle multiple vendor
implementations with the same query. The reason for this is vendors
are allowed to (and almost always do) make their own custom classes
with a different name that inherit from the standard ones, this is so
they can add their own custom data. this would seem like a problem but
if you use the right flags with enumerate, get, and references queries
you should see them when you query the standard ones as though you had
requested them directly.
I haven gotten around to documenting CQL yet sorry.

[Tony Asleson]

On 03/08/2017 09:27 PM, Gris Ge wrote:
> IMHO, the 'blur' cannot be fixed as SMI-S by design is to make storage array
> vendors happy, not the client users, hence it has to be blur to allow every
> storage array to fit in.

"SMI-S is a standard way to get to non-standard data", Dr. Alan Yoder

This is why there is more than 1 SMI-S driver in cinder.

-Tony

[Devchandra Meetei]

Hello Arun

I would rather starts with SLP discovery to find out the interop namespace from there. Once, we know the interop namespace, get  list of Registered_profiles and proceed from there.

You might need to have access to the profile specification. This will give a little generic way CIM Client application

[Arun Anbalagan]

Dear All,

Thanks so much..  I'm hopeful that I should be able to pick up some solid advises and inputs from here.

Can I just throw another question? Is there any document or web link that describes the step-by-step process involved in development an SMI-S client? While I'm pretty relieved that I don't have to master every protocol (WEBM, CIM, etc.), there are still some gray areas, such as how to establish the connection with a WEBM sever, and what follows.  I see that the specs in SNIA talk more about implementing the providers, but not exactly the clients (Am I wrong?). By any chance, are there any documented guides for dummies on SMI-S client development? Would be grateful for any light that you could shed. Thanks.

Regards,

Arun.

On Wednesday, March 8, 2017 at 12:13:54 PM UTC+9, Arun Anbalagan wrote: