catalog-diff and create_resources not correct

[Johan De Wit]

Hi,

We wanted to use catalog diff to verify the old and new catalogs during conversionto hiera.  They should be both have the saem resources with the same paramters.
But every resource that is created in the 'new' version with create_resources(), gives a wrong catalog diff.

The resource is indeed in both catalogs, only the old one contains 2 automatic attributes (file: and line:).  I suspect this is the cause.

Anyone experienced simular result ?

In the meantime, I try to understand the catalog-diff code to see where this could be corrected.

Grts

johan

Catalog diff output :
--------------------------------------------------------------------------------
test_node                                                                 100.0%
--------------------------------------------------------------------------------
Total resources in old: 0
Total resources in new: 1
Only in new:
        notify[my_test_notify]
Catalag percentage added:       100.00
Catalog percentage removed:     0.00
Catalog percentage changed:     0.00
Added and removed resources:    +1 / 0
Node percentage:        100.0
Node differences:       1
--------------------------------------------------------------------------------
1 out of 1 nodes changed.                                                 100.0%
--------------------------------------------------------------------------------
Nodes with the most changes by percent changed:
1. test_node                                                             100.00%
Nodes with the most changes by differeces:
1. test_node                                                                  1false
 
The manifest new and catalog entry
 
node 'test_node' {
   $my_notify =  { 'my_test_notify' => { 'message' => 'this is a test notify',
                                         'name' => 'test_notify',
                                         'withpath' => 'true', }
                }
   create_resources(notify, $my_notify)
}
 
<snip>      {
        "parameters": {
          "withpath": "true",
          "message": "this is a test notify",
          "name": "test_notify"
        },
        "exported": false,
        "tags": [
          "notify",
          "my_test_notify",
          "node",
          "test_node",
          "class"
        ],
        "title": "my_test_notify",
        "type": "Notify"
      }
 
<snip>
 
The manifest OLD
 
node 'test_node' {
   notify { 'my_test_notify':
     message  => 'this is a test notify',
     name     => 'test_notify',
     withpath => true,
   }
}
 
<snip>
        "title": "test_node",
        "type": "Node"
      },
      {
        "parameters": {
          "withpath": true,
          "message": "this is a test notify",
          "name": "test_notify"
        },
        "exported": false,
        "line": 12,
        "file": "/etc/puppetlabs/puppet/environments/rese_old/manifests/nodes/test_node.pp",
        "tags": [
          "notify",
          "my_test_notify",
          "node",
          "test_node",
          "class"
        ],
        "title": "my_test_notify",
        "type": "Notify"
      }
<snip>

-- 
Johan De Wit

Open Source Consultant

Red Hat Certified Engineer              (805008667232363)
Puppet Certified Professional 2013/2014 (PCP0000006)
_________________________________________________________
 
Open-Future                 Phone     +32 (0)2/255 70 70
Zavelstraat 72              Fax       +32 (0)2/255 70 71
3071 KORTENBERG             Mobile    +32 (0)474/42 40 73
BELGIUM                     http://www.open-future.be
_________________________________________________________
 

Upcoming Events:

Zabbix Certified Specialist | http://www.open-future.be/zabbix-certified-professional-training-8th-till-9th-janaury

Zabbix Certified Professional | http://www.open-future.be/zabbix-certified-professional-training-8th-till-9th-janaury

Bacula Administrator 1 | http://www.open-future.be/bacula-administrator-i-training-13th-till-15th-january

Puppet Fundamentals | http://www.open-future.be/puppet-fundamentals-training-26th-till-28th-january

Puppet Architect | http://www.open-future.be/puppet-architect-training-29th-till-30th-january

Subscribe to our newsletter: http://eepurl.com/BUG8H

[Felix Frank]

On 12/23/2014 10:36 AM, Johan De Wit wrote:
> The resource is indeed in both catalogs, only the old one contains 2
> automatic attributes (file: and line:). I suspect this is the cause.

Hi Johan,

good call - I believe those are hints that the compiler adds, so that
the agent can give more helpful error output in case a resource fails
its validation checks.

Are you in a position to filter those from the "old" style catalogs
prior to diffing?

sed -i '/"line":|"file":/d' <catalog>

comes to mind :-)

HTH,
Felix

[Johan De Wit]

Hi Felix,

That is a good idea as quick fix. And why did I not think to that !! :)
Long term, such attributes should get excluded from the diff.

Time to delve into the ruby world :)

Grts

Johan

--
Johan De Wit

Open Source Consultant

Red Hat Certified Engineer (805008667232363)
Puppet Certified Professional 2013/2014 (PCP0000006)
_________________________________________________________

Open-Future Phone +32 (0)2/255 70 70
Zavelstraat 72 Fax +32 (0)2/255 70 71
3071 KORTENBERG Mobile +32 (0)474/42 40 73
BELGIUM http://www.open-future.be
_________________________________________________________

Next Events:
Zabbix Certified Training | http://www.open-future.be/zabbix-certified-specialist-training-5th-till-7th-january

[Johan De Wit]

On 24/12/14 00:50, Felix Frank wrote:

changing the old catalog does not work. The resources aren't seen
anymore by the catalog-diff program, both in the old and new catalog.

--
Johan De Wit

Open Source Consultant

Red Hat Certified Engineer (805008667232363)
Puppet Certified Professional 2013/2014 (PCP0000006)
_________________________________________________________

Open-Future Phone +32 (0)2/255 70 70
Zavelstraat 72 Fax +32 (0)2/255 70 71
3071 KORTENBERG Mobile +32 (0)474/42 40 73
BELGIUM http://www.open-future.be
_________________________________________________________

Next Events:
Zabbix Certified Training | http://www.open-future.be/zabbix-certified-specialist-training-5th-till-7th-january

[Alex Harvey]

Hi Johan,

I read your excellent blog post on this

http://johan.koewacht.net/puppet/2015/01/12/catalog-diff-create_resources.html

I have also run into the same issue.  Was this ever resolved?  

You wrote at the end,

But question now, should we patch the catalog-diff preprocessor.rb or puppet’s native create_resources.rb

Considering that the purpose of the catalog diff tool is to assist in migrations from earlier to later versions of Puppet, it's no use patching create_resources.rb in a newer version of Puppet.  I think this is really a bug in the catalog diff tool?

Thanks,

Alex

[Alex Harvey]

Actually looks like there's an open PR to fix this here

https://github.com/acidprime/puppet-catalog-diff/pull/28