Issues with self.instances and additional Puppet::Type in array...

[Gavin Williams]

Afternoon all

I've started working on converting a couple more of my NetApp network device providers to use a prefetch/flush model, as can see performance gains available, etc...

Anyways, I'm having issues with my netapp_qtree provider. It would appear that somehow, an additional Puppet::Type... row is getting into an array and breaking things...

Have created a gist here with the details, as the log file is quite long.
Also includes the instances and prefetch def's for my netapp_qtree provider...

As you can see on Line 337 of the log, the array contains 40 items, however on line 734 self.instances is trying to process item 41?!?!
What's also strange is that the item contents look like a Puppet Type (#<Puppet::Type::Netapp_qtree), whereas all the others in the array are NetApp specific items (#<NaElement:).

So, any ideas???

Cheers
Gavin

[Gavin Williams]

Morning all

Quick update... Looks like I managed to hack around the issue by adding the following:

...

        ap qtree_info
 
        # Check if it is a NaElement
        next unless qtree_info.respond_to?(:child_get_string)
 
        # Pull out the qtree name.
        name = qtree_info.child_get_string("qtree")
...

However this shows that I'm getting a total of 78 items processed, whereas the original array only contains 53 items...
The additional items being processed are all like: '#<Puppet::Type::Netapp_qtree::ProviderNetapp_qtree:'.
Have updated the gist with latest code and log file.

Would like to understand where these are coming from, and if it's something I'm doing incorrectly?

In the mean-time, following fixing that bug, the provider now seems to work as expected :)

Cheers
Gavin

[Gavin Williams]

Afternoon

Managed to find my issue.. Variable name re-use :(

Defined 'qtrees' as an empty array on line 3, and then populated it with a whole load of device output on line 18 :(

So have defined a 'qtree_instances' array on the outside to contain my output, with qtrees being used on the inside to hold the NetApp filer response... Latest code in Github.

Cheers
Gavin

[Luke Kanies]

Hi Gavin,

I'm glad to see you've sorted it out, sorry I didn't jump in before it was all resolved.

How has prefetching worked out for you, in terms of performance?

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 

Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | +1-615-594-8199

Join us at PuppetConf 2013, August 22-23 in San Francisco - http://bit.ly/pupconf13

[fatmcgav]

Luke

No worries.

For the 2 I've converted soo far, it's made quite a performance difference. Thankfully can pull back all the resources with one NetApp api call, so it's drop the resource time from 5+ seconds to sub 1 second.
However was a bit confused initially, as different providers seem to use different models... Some with just prefetch, some with just instances, some with both...

However I'm just about to start the more complex netapp_volume provider, which has 4+ properties that have different api calls on top of the base resource list call. This is by far the slowest provider out of them with an average of 20+ seconds per run.
So not sure of the best way to handle as yet.

Cheers
Gav

You received this message because you are subscribed to a topic in the Google Groups "Puppet Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to puppet-dev+...@googlegroups.com.

[Luke Kanies]

Hi Gavin,

Yeah, there isn't as much consistency in API as I would like.

The way it should work, I believe, is something like:

* The Type should support an 'instances' method that returns a complete list of all instances of that type that exist on the host itself.  This method should not require a catalog.

* The provider should support a 'prefetch' method, which accepts a list of resources from the catalog and updates them with the correct provider instance, with appropriate data already filled in to reflect the system state.

* Optionally, many providers implement (I think) their own 'instances' method, which returns all instances and doesn't require a catalog, to implement both of the above methods.

I just looked at the Provider base class, though, and it's not at all clear from this.  Part of the problem is that the system is smart enough to skip providers that don't support prefetch, and the way we denote 'doesn't support prefetch' is that it doesn't have the method.  Given that, I left the method off of the base class.  Because 'instances' isn't used for this kind of test, the base class ships with a stub method that just throws an error.

So yeah, all that's confusing, and could really use some additional work to make it easier to understand, and to bring some consistency.

[fatmcgav]

Luke

Cheers for that info, should prive useful...

Guess I could implement a method for each property that pulls back all that property values for all the volumes, and then match them up in prefetch.

Cheers
Gav

[Gavin Williams]

As a quick update, I've pushed my latest set of code tweaks up - https://github.com/fatmcgav/fatmcgav-netapp/commit/7a7d18bf39cdbb04a3b0b5192929ec6a857c0a5e

Need to tidy the code up a bit, but should be able to get the gist :) Have got a couple of defs which pull back the bulk of the information, and then a couple that need to be called on a volume by volume basis.
All gets put together in instances, and provider'd up in prefetch.

Comments welcome.

Cheers
Gavin

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 

Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | +1-615-594-8199

Join us at PuppetConf 2013, August 22-23 in San Francisco - http://bit.ly/pupconf13

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en.

To unsubscribe from this group and all its topics, send an email to puppet-dev+unsubscribe@googlegroups.com.

To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscribe@googlegroups.com.

To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

-- 

Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | +1-615-594-8199

Join us at PuppetConf 2013, August 22-23 in San Francisco - http://bit.ly/pupconf13

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-dev/74JW491YSAk/unsubscribe?hl=en.

To unsubscribe from this group and all its topics, send an email to puppet-dev+unsubscribe@googlegroups.com.

[Luke Kanies]

Looks good from a cursory glance.  Hopefully it's all working for you.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.

To post to this group, send email to puppe...@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[fatmcgav]

Yep, have been testing it today on my dev and production Puppet masters, and seems to be working as expected...

Have been very impressed with the performance improvements - Has dropped the netapp_volume resource time from 40+ seconds to <0.1 seconds. So MASSIVE improvements to be made from switching to the prefetch/flush style...

Cheers
Gav

To unsubscribe from this group and all its topics, send an email to puppet-dev+...@googlegroups.com.

[Luke Kanies]

That is awesome.  It's amazing what a bit of optimization will do for you.

[Gavin Williams]

Indeed... :)

Ok, next question for you/all...

How could I differentiate between what is a resource creation and a resource modification?

Have updated the netapp_user provider to prefetch/flush, however am struggling to differentiate between what is a resource creation, and what is a modification...
Code is here: https://github.com/fatmcgav/fatmcgav-netapp/commit/66092978f4182c5474a60011db99ee2e3e12e689

Thoughts?

Cheers
Gavin

[Luke Kanies]

Can you explain why you need to know the difference?

The framework should handle that for you.

[fatmcgav]

Luke

A user create requires a different API call to a modify... (useradmin-user-add vs useradmin-user-modify).

So I could leave the creation to 'create', however how do I then trigger a modify in flush when required?

Or should I treat any call into flush with a :present ensure as a modify?
Though I vaguely remember that causing a double-call on a resource create - Once from create, and then another from flush...

Cheers
Gavin

[Luke Kanies]

Ah.  You should copy the model in the user/group providers:  The providers have a 'create' method that uses useradd, for instance, and then a 'uid=' method that uses usermod.

Does that help?  Are you doing all of the actual work in the 'flush' method, and that's where you need to differentiate?

[fatmcgav]

I was looking at doing all the work in the 'flush' method, however have just been doing some more testing and think I've come up with an alternative...

Can run the user creation in the 'create' def, and then clear @property_hash so that it doesn't get invoked in 'flush'.

'Flush' then calls the user modification if required...

Seems to work quite well from my testing :)

Latest code is: https://github.com/fatmcgav/fatmcgav-netapp/blob/master/lib/puppet/provider/netapp_user/netapp_user.rb

Cheers
Gavin

[Luke Kanies]

Excellent.

[Nan Liu]

On Fri, Apr 5, 2013 at 3:33 AM, fatmcgav <fatm...@gmail.com> wrote:

I was looking at doing all the work in the 'flush' method, however have just been doing some more testing and think I've come up with an alternative...

Can run the user creation in the 'create' def, and then clear @property_hash so that it doesn't get invoked in 'flush'.

'Flush' then calls the user modification if required...

Seems to work quite well from my testing :)

Latest code is: https://github.com/fatmcgav/fatmcgav-netapp/blob/master/lib/puppet/provider/netapp_user/netapp_user.rb

There's an edge case that will fail for puppet resource, if you don't use the command then don't worry about this. Puppet resource depends on @property_hash, so clearing @property_hash after create will result in incorrect output when a resource is created. Currently I'm using something similar to this for flush:

def some_property=(val)

   @pending_change = true

   @property_hash['some_property'] = val

   # queue_change needs to be initialized and it's only necessary if you need to track what needs to be flushed.

   @queue_change['some_property'] = val

end

def flush

   if @pending_change

   ...

   end

end

Thanks,

Nan

[Gavin Williams]

Nan

Ah, ok.. Haven't used puppet resource as yet, however think it makes sense to try and support it where possible...

Will see what I can work up with your code :)

Cheers
Gavin