How to save a Puppet::Resource::Catalog object to disk for later application

[Trevor Vaughan]

Hi All,

I'm doing some experiments with extracted catalog snippets but seem to be having issues on saving the new catalog after the fact.

I have a Puppet::Resource::Catalog object that has content and relationships but when I attempt to save it using 'to_pson' I get an "undefined method `to_pson_data_hash`" for any given resource in the catalog.

As an aside, if I just run @resource.catalog.to_pson from within a provider, I end up with the same error.

Any help would be appreciated.

Thanks,

Trevor

--

Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699

-- This account not approved for unencrypted proprietary information --

[Adrien Thebo]

I've been working on a project with similar goals and ran into the same thing. The core of the issue is that there are two catalogs, the resource catalog and the RAL catalog. The resource catalog is what the Puppet compiler emits and contains Puppet::Resource instances; this is what's actually sent to the agent and what you'll see if you run `puppet master --compile`. The RAL catalog is generated when the catalog is actually being applied (https://github.com/puppetlabs/puppet/blob/master/lib/puppet/configurer.rb#L106), and converts Puppet::Resource instances to Puppet::Type instances. Puppet::Resource instances are meant to be serialized, but Puppet::Type instances aren't - I'm pretty sure that the error you're seeing is because Puppet is trying to serialize Puppet::Type instances that don't have the correct method defined.

There is a document in the Puppet source (https://github.com/puppetlabs/puppet/blob/master/docs/catalogs.md) that contains a bit more information about the different types of catalogs.

That being said there's a couple of ways of converting things around to a more easily serializable format.

One option is to individually convert Puppet::Type instances to Puppet::Resource instances as you find them; I've been doing them with something like this:

Puppet::Type.type(:service).instances.map { |res| res.to_resource }

Alternately, if you have an existing catalog and you want to convert and serialize that wholesale, you should be able to use `Puppet::Resource::Catalog#to_resource` (https://github.com/puppetlabs/puppet/blob/master/lib/puppet/resource/catalog.rb#L490) to do this. I haven't tested this out but it seems like it should work.

There are a couple of caveats in this - for instance things get a little bit screwy with some resource types such as files, but if you run into issues with too many parameters being generated/not enough parameters being generated, let me know.

If you make progress in this area let me know; since I'm doing some poking around in this area I would be interested in seeing what you find out as well.

--
You received this message because you are subscribed to the Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUnc589eJ3nujpF0G38%3D6hAHOMHVShU7xRU9KfzS%2BTjQg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Adrien Thebo | Puppet Labs

[Trevor Vaughan]

Hey Adrian,

Thanks for the response. I tried the Puppet::Resource::Catalog#to_resource method but ended up with an error on 'copy_as_resource' not found.

I also tried the 'to_resource' method on all of the resource in my catalog individually but that didn't seem to get me anywhere either.

Looking at the links that you posted, I suppose what I'm trying to do is actually save a RAL catalog and then reactivate it via the command line at a later point. Do you know if this is possible? I don't really need a compiler-level resource catalog since this won't be shipped off of the local system.

Thanks,

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALVJ9SKOK4HyZyOMYT%3DF31m7GxdL0jVvmYvKhzUAPBb5qhn0wg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

[Adrien Thebo]

The exact issues that you may run into depend on how the catalog was generated and what it's storing; in my scenario I'm using the RAL to collect Puppet::Type instances, I'm converting them to Puppet::Resource instances, and then inserting those into the catalog. Trying to individually convert resources in the catalog might be messy because you'll need to delete the type you're converting and then add the Resource instance; it's doable but there are some logistics around that.

Do you have code that you can provide to demonstrate what you're doing? Given a RAL catalog you should be able to get a resource catalog in some form, as for reactivating a catalog I'm not sure what you mean by "reactivate". And lastly, the "resource catalog" is effectively the only means of serializing a catalog right now; if you want to store a catalog you'll almost certainly have to convert your data to a resource catalog, or figure out how you can make Puppet::Type instances serializable.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoU7jp3SBnL2%3DOpa3abixQwPuUvWDgh6VEGRNntGjO8zsw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

Hey Adrian,

The code is pretty simple right now and is as follows:

### Type

module Puppet

  newtype(:catalog_test) do

    @doc = "Save a micro-catalog"

    newparam(:name) do

      isnamevar

      desc "Resource name to trigger off of"

    end

    newproperty(:resource_type) do

      desc "The resource type"

    end

  end

end

### Provider

Puppet::Type.type(:catalog_test).provide(:micro_catalog) do

  desc 'Save a micro-catalog'

  def resource_type

    my_catalog = Puppet::Resource::Catalog.new

    target_resource = @resource.catalog.resources.find{|x|

      (x.type == @resource[:resource_type].downcase.to_sym) &&

      (x.name == @resource[:name])

    }

    if target_resource

      @resource.catalog.relationship_graph.dependencies(target_resource).each do |res|

        my_catalog.add_resource(res)

      end

      my_catalog.add_resource(@resource)

      @resource.catalog.relationship_graph.dependents(target_resource).each do |res|

        my_catalog.add_resource(res)

      end

    end

# Tinker Point

require 'pry'

binding.pry

    return @resource[:resource_type]

  end

end

### Test File

file { '/tmp' : ensure => 'directory' }

file { '/tmp/foo1': content => 'blah' }

file { '/tmp/stack': ensure => 'directory' }

file { '/tmp/stack/test1': content => 'test' }

file { '/usr/test':

  content => 'test',

  require => File['/tmp/stack']

}

file { '/var/trigger':

  content => 'test',

  notify  => File['/tmp/stack/test1']

}

catalog_test { '/var/trigger':

  resource_type => 'file'

}

Thanks for the help,

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALVJ9SJ6OjCnWG5TVqOMNFJF8Jc3kXvhGP9wgubg2CwET1Fy%3DA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Adrien Thebo]

Hey Trevor,

I've been able to reproduce your problem and I see what's going on. Since this sort of post hoc catalog serialization isn't something that's done a lot (or at all) there are some sharp edges; I'm going to head into this code base with a big stick and I'm going to hit things really hard until they work. It might take me a few days but when I've got things a bit more sorted out I'll let you know what I've figured out.

Adrien

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoWBPxK5fSfB5CGFZWyWc_PN7htatdChp4S%3DbZ9PayEiSg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

Awesome! Thanks Adrien.

I've got some interesting plans for this if it works out and I appreciate you taking the time to figure out what's going on.

Trevor

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CALVJ9SJe2bkJT9W07kRO4YcOtPAm35BqMYFJVvWKxuVgQoq8xQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.