Jira (PUP-9560) master compile errors on upper case class ref with unknown class where apply works

0 views
Skip to first unread message

Henrik Lindberg (JIRA)

unread,
May 22, 2019, 6:22:03 AM5/22/19
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
 
Puppet / Bug PUP-9560
master compile errors on upper case class ref with unknown class where apply works
Change By: Henrik Lindberg
Summary: Class references in Hiera data: parsing error? master compile errors on upper case class ref with unknown class where apply works
Component/s: Hiera & Lookup
Component/s: Compiler
This was initially reported as related to hiera and lookup, but when problem was reproduced it was triggered without hiera/lookup being involved. In essence a {{Class\[test]}} reference works when using apply, but not when compiling via the master - changing the {{test}} to {{Test}} makes it work on the master.

ORIGINAL
-----
 *Puppet Version: 6.0.5-1stretch*
*Puppet Server Version: 6.2.0-1stretch* 
*OS Name/Version: Debian 9*

I'm using the lookup function to create a file resource from Hiera data:

'/primary':
  ensure: 'directory'
  mode: '0755'
  owner: 'root'
  group: 'root'
  before: 'Class[nfs]'

This results in the following error when the agent runs on the node:

"Server Error: Invalid relationship:  File[/primary] \{ before => Class[nfs] }, because Class[nfs] doesn't seem to be in the catalog".

But here's the thing: if the first letter of the class _name_ is capitalized - to be clear, as in Class[Nfs] - the agent run proceeds smoothly.  

This is at odds with what happens if I create the file resource explicitly in a manifest.  In that case, it is _not_ necessary to capitalize the class name in the "before =>" attribute.

So on the face of it, it seems something is amiss in the Hiera lookup, or parsing, or ... something ...

(Incidentally, this happens no matter which class I'm referencing - it's not restricted to the "nfs" (derdanne-nfs module) class.)

Stephen Kenny

 
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Henrik Lindberg (JIRA)

unread,
May 22, 2019, 6:22:04 AM5/22/19
to puppe...@googlegroups.com
Henrik Lindberg updated an issue
Change By: Henrik Lindberg
Attachment: agent-run.debug

Stephen Kenny (JIRA)

unread,
May 22, 2019, 6:35:03 AM5/22/19
to puppe...@googlegroups.com
Stephen Kenny updated an issue
Change By: Stephen Kenny
Attachment: stacktrace.txt

Stephen Kenny (JIRA)

unread,
May 22, 2019, 6:35:03 AM5/22/19
to puppe...@googlegroups.com

Stephen Kenny (JIRA)

unread,
May 22, 2019, 6:35:03 AM5/22/19
to puppe...@googlegroups.com
Stephen Kenny commented on Bug PUP-9560
 
Re: master compile errors on upper case class ref with unknown class where apply works

Re case, yes it's exactly as you mention - the test class is defined in the file "test/manifests/init.pp" (where "test" is a subdirectory in our local modules directory), and everything is in lower case.

(Just in case it's relevant, our modules live in two separate directories.  We put local (home-grown) ones in /etc/puppetlabs/code/modules, while Forge modules live in /etc/puppetlabs/code/environments/production/modules.)

I've attached some stack trace information from the Puppet server's /var/log/puppetlabs/puppetserver.log file: stacktrace.txt

 

 

Henrik Lindberg (JIRA)

unread,
May 22, 2019, 1:08:09 PM5/22/19
to puppe...@googlegroups.com

Charlie Sharpsteen Have you seen this problem before?

Charlie Sharpsteen (JIRA)

unread,
May 22, 2019, 3:03:02 PM5/22/19
to puppe...@googlegroups.com

Nope, this is a new one to me.

Henrik Lindberg (JIRA)

unread,
May 22, 2019, 4:01:03 PM5/22/19
to puppe...@googlegroups.com

Charlie Sharpsteen thanks, I find it really strange that we haven't seen reports about this earlier since using an uppercase reference is not what people tend to use.

Josh Cooper (Jira)

unread,
Jan 27, 2021, 2:40:05 PM1/27/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9560

The error is triggered by the puppetdb terminus which explains why it doesn't happen when using apply (since you'd have to explicitly configure the apply application to use the puppetdb terminus via routes.yaml)

2019-05-22T11:25:54.959+01:00 ERROR [qtp1510301445-5198] [puppetserver] Puppet Invalid relationship: Notify[goes first] { before => Class[test] }, because Class[test] doesn't seem to be in the catalog
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/puppetdb.rb:407:in `block in synthesize_edges'
 
org/jruby/RubyArray.java:1801:in `each'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/puppetdb.rb:362:in `block in synthesize_edges'
 
org/jruby/RubyHash.java:1396:in `each'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/puppetdb.rb:360:in `block in synthesize_edges'
 
org/jruby/RubyArray.java:1801:in `each'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/puppetdb.rb:343:in `block in synthesize_edges'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:99:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/puppetdb.rb:341:in `block in synthesize_edges'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:99:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/puppetdb.rb:329:in `synthesize_edges'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/puppetdb.rb:69:in `block in munge_catalog'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:99:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/puppetdb.rb:52:in `munge_catalog'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/puppetdb.rb:13:in `block in save'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/puppetdb.rb:99:in `profile'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/puppetdb.rb:11:in `save'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/store_configs.rb:24:in `save'
 
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:200:in `find'
 
..

The puppetdb check was added in https://github.com/puppetlabs/puppetdb/pull/442 in response to create_resources passing through invalid resource references. It seems puppet is doing an capitalize/case-insensitive match during catalog evaluation and passing it through to the pdb terminus.
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Josh Cooper (Jira)

unread,
Jan 27, 2021, 3:28:05 PM1/27/21
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Attachment: stacktrace.txt

Josh Cooper (Jira)

unread,
Jan 27, 2021, 4:06:04 PM1/27/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9560
 
Re: master compile errors on upper case class ref with unknown class where apply works

This is pretty easy to reproduce when puppetdb is installed:

# mkdir -p /etc/puppetlabs/code/environments/test/manifests
 
# cat > /etc/puppetlabs/code/environments/test/manifests/site.pp <<END
 
notify { 'in site.pp':
 
  before => 'Class[test]',
 
}
 
include test
 
END
 
# mkdir -p /etc/puppetlabs/code/environments/test/modules/test/manifests/
 
# cat /etc/puppetlabs/code/environments/test/modules/test/manifests/init.pp <<END
 
class test {}
 
END
 
# chown -R puppet:puppet /etc/puppetlabs/code/environments/test
 
# puppet agent -t --environment test
 
Info: Using configured environment 'test'
 
Info: Retrieving pluginfacts
 
Info: Retrieving plugin
 
Info: Retrieving locales
 
Info: Loading facts
 
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Invalid relationship: Notify[in site.pp] { before => Class[test] }, because Class[test] doesn't seem to be in the catalog

The problem is the line:

before => 'Class[test]',

Changing that to any of the following fixes the issue:

before => Class['test'],
 
before => 'Class[Test]',
 
before => Class['Test'],

The difference between the bad and good cases are these lines in the serialized catalog:

[root@candid-fluting ~]# diff bad.txt good.txt
 
68c68
 
<         "before": "Class[test]"
 
---
 
>         "before": "Class[Test]"

Should the compiler accept the "Class[test]" reference?

Josh Cooper (Jira)

unread,
Jan 27, 2021, 8:06:05 PM1/27/21
to puppe...@googlegroups.com
Josh Cooper commented on Bug PUP-9560

So this looks like a catalog normalization issue only when processing class references and using "Class[name]". Note the difference in before parameters for the bad and good notify resources below:

class a {}
 
include a
 
notify { 'bad':
 
 before => 'Class[a]'
 
}
 
notify { 'good':
 
 before => Class['a']
 
}

In the generated catalog, Notify[good] references Class[A], but Notify[bad] references Class[a]. However, only Class[A] exists in the catalog:

 {"type"=>"Class", "title"=>"A", "tags"=>["class", "a"], "exported"=>false},
 
   {"type"=>"Notify",
 
    "title"=>"bad",
 
    ...
 
    "parameters"=>{"before"=>"Class[a]"}},
 
   {"type"=>"Notify",
 
    "title"=>"good",
 
    ...
 
    "parameters"=>{"before"=>"Class[A]"}}],

Puppet's compiler/relationship validator doesn't have problems with mismatched class titles, because it calls Catalog#resource to lookup the class resource and that method accepts both cases:

(byebug) catalog.resource("Class[a]")
 
Class[A]{:name=>"A"}
 
(byebug) catalog.resource("Class[A]")
 
Class[A]{:name=>"A"}

But the puppetdb terminus doesn't use that same logic to resolve edges, so it thinks Class[a] is missing.

The same problem doesn't occur for resources however:

notify { 'first': }
 
notify { 'second':
 
  before => 'Notify[first]'
 
}
 
notify { 'third':
 
  before => Notify['second']
 
}

The compiler produces a catalog whose before parameters reference lowercased resource titles:

 {"type"=>"Notify",
 
    "title"=>"first",
 
    ...
 
   {"type"=>"Notify",
 
    "title"=>"second",
 
    ...
 
    "parameters"=>{"before"=>"Notify[first]"}},
 
   {"type"=>"Notify",
 
    "title"=>"third",
 
    ...
 
    "parameters"=>{"before"=>"Notify[second]"}}],

I'm inclined to close won't fix since there is a simple way to handle this (use Class['nfs']). Thoughts Henrik Lindberg Ben Ford?

Henrik Lindberg (Jira)

unread,
Jan 29, 2021, 10:51:04 AM1/29/21
to puppe...@googlegroups.com

I agree with Josh Cooper, the correct way to reference the class is by using the lower class name in this case. The upper cased name is actually a reference to the data type - so when writing Class[A] that should really mean the meta type of the class named "a". With classes, since their instances are singletons it does not make a difference if statements are made about the specific "a" instance or all instances since there can be only one.

In the puppet language the expression Class[A] gives an error since it does not accept a type reference to the type A. Ideally the same kind of reference in string form "Class[A]" should be an error. The puppet language accepts Class["A"] since it is specified that class names are case independant (and lower case in normal form), and now the class name is a string so it downcases it.

There are a bunch of other problems with the parser that parses resource references (the classic being resources with square brackets in their name). If that is ever sanitized it would be worth to clean up this case mismatch as well.

OTOH, puppet db could do a case insensitive compare.

David McTavish (Jira)

unread,
Mar 8, 2022, 3:20:02 PM3/8/22
to puppe...@googlegroups.com
David McTavish updated an issue
 
Change By: David McTavish
Priority: Normal Low
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages