Puppet server failing on invalid UTF-8 facts

[gio...@optimiseweb.co.uk]

Hello,

I'm having some issues after upgrading to Puppet 4.

Basically on a puppet agent run I get this error:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: invalid byte sequence in UTF-8

I tracked it down, and found this HTTP request to be the cause of it

94 4.641797000 192.168.3.45 192.168.3.99 HTTP 1836    POST /puppet/v3/catalog/nuctest HTTP/1.1  (application/x-www-form-urlencoded)

96 4.669936000 192.168.3.99 192.168.3.45 HTTP 311     HTTP/1.1 400 Bad Request  (text/plain)

In that POST request, I decoded the "facts" item to find a few fields which in fact are not valid (I hope the characters won't be altered when posting this topic)

{
  "name":"nuctest",
  "values":{
    [...]
    "chassisassettag":"���������������������������������",
    "dmi":{
      "bios":{ [...] },
      "board":{ [...] },
      "chassis":{
        "asset_tag":"���������������������������������",
        "type":"Desktop"
      },
      "manufacturer":"���������������������������������",
      "product":{
        "name":"���������������������������������",
        "serial_number":"���������������������������������",
        "uuid":"80071049-A653-E211-99B0-C03FD564AF0A"
      }
    },
    "manufacturer":"���������������������������������",
    "productname":"���������������������������������",
    [...]
  }
}

These facts, based on what I read here https://github.com/puppetlabs/facter/blob/master/lib/schema/facter.yaml , are copied from some files in /sys/class/dmi

I tried to look at some of them

[root@nuctest ~]# cat /sys/class/dmi/id/chassis_asset_tag
���������������������������������

[root@nuctest ~]# cat /sys/class/dmi/id/product_serial
���������������������������������

[root@nuctest ~]# cat /sys/class/dmi/id/product_name
���������������������������������

[root@nuctest ~]# cat /sys/class/dmi/id/product_name
���������������������������������

[root@nuctest ~]# cat /sys/class/dmi/id/board_vendor
Intel Corporation

As you can see, not all of them are made up of odd characters, but I have no idea why some are...

This is where puppetserver returns the error while parsing those strings:

2015-07-09 11:31:46,908 ERROR [puppet-server] Puppet invalid byte sequence in UTF-8

org/jruby/RubyString.java:6202:in `count'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych/visitors/yaml_tree.rb:248:in `binary?'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych/visitors/yaml_tree.rb:262:in `visit_String'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych/visitors/yaml_tree.rb:128:in `accept'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych/visitors/yaml_tree.rb:324:in `visit_Hash'

org/jruby/RubyHash.java:1341:in `each'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych/visitors/yaml_tree.rb:322:in `visit_Hash'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych/visitors/yaml_tree.rb:128:in `accept'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych/visitors/yaml_tree.rb:467:in `emit_coder'

org/jruby/RubyHash.java:1341:in `each'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych/visitors/yaml_tree.rb:465:in `emit_coder'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych/visitors/yaml_tree.rb:450:in `dump_coder'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych/visitors/yaml_tree.rb:126:in `accept'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych/visitors/yaml_tree.rb:92:in `push'

/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/shared/psych.rb:244:in `dump'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/yaml.rb:21:in `dump'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:380:in `replace_file'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/yaml.rb:20:in `dump'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/yaml.rb:30:in `save'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:276:in `save'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/node/facts.rb:21:in `save'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:39:in `extract_facts_from_request'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:20:in `extract_facts_from_request'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:45:in `find'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:134:in `do_find'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `call'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:223:in `override'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `process'

org/jruby/RubyArray.java:1613:in `each'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:60:in `process'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in `process'

file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppet-server-lib/puppet/server/master.rb:39:in `handleRequest'

Puppet$$Server$$Master_279047714.gen:13:in `handleRequest'

request_handler_core.clj:274:in `invoke'

request_handler_service.clj:14:in `handle_request'

request_handler.clj:3:in `invoke'

request_handler.clj:3:in `invoke'

core.clj:626:in `invoke'

core.clj:2468:in `doInvoke'

master_core.clj:47:in `invoke'

ring.clj:22:in `invoke'

ring.clj:13:in `invoke'

comidi.clj:267:in `invoke'

ringutils.clj:106:in `invoke'

ringutils.clj:62:in `invoke'

ringutils.clj:68:in `invoke'

ringutils.clj:118:in `invoke'

jetty9_core.clj:408:in `invoke'

As mentioned, this machine was successfully managed by Puppet 3.7 before (I went for a clean install, so there might be some difference in the configuration, even though I used the same kickstart file...)

The OS is CentOS 6.5, kernel 2.6.32-431.29.2.el6.i686

The machine is an Intel NUC DN2820FYKH (I did upgrade the BIOS before reinstalling the OS, and I guess that might have interfered)

I suppose there must be a fix to have those file properly working, which I am looking for. However, is there anything I could do so that puppet does not fail on this, but rather just "ignore" the corrupted facts?

Thanks,

Giorgio

[Federico Agnelli]

Hi Giorgio,

I have the same problem with a puppet client on Intel NUC5i5RYH.
All other clients (on different hardware) with the same puppet version work fine.

My puppet version for both agent and server is 4.2.0. Version 4.1.0 was ok.

Did you find a fix?

Thanks,
Federico

[Peter Huene]

On Monday, July 20, 2015 at 1:16:07 PM UTC-7, Federico Agnelli wrote:

Hi Giorgio,

I have the same problem with a puppet client on Intel NUC5i5RYH.
All other clients (on different hardware) with the same puppet version work fine.

My puppet version for both agent and server is 4.2.0. Version 4.1.0 was ok.

It looks like Facter is expecting the contents of those files to be UTF-8, which may not be the correct thing to do.  Could you install/run the dmiencode utility and paste the output?  Also, cat the same files Giorgio did and attach as a file (the characters are coming through as invalid characters in Google Groups).  Thanks!

[Peter Huene]

Correction: the utility is *dmidecode*.

[Federico Agnelli]

Hi Peter,

attached you can find the output of dmidecode and another file with the output of the cat command.

Thanks,
Federico

[Peter Huene]

On Wed, Jul 22, 2015 at 3:01 AM, Federico Agnelli <fagn...@gmail.com> wrote:

Hi Peter,

attached you can find the output of dmidecode and another file with the output of the cat command.

It looks like, for whatever reason, those values were set by the manufacturer to 0xFF 0xFF 0xFF ...

dmidecode outputs a series of dots to represent these invalid characters.  I've opened https://tickets.puppetlabs.com/browse/FACT-1140 to track this issue so that Facter fixes it in a similar way.

I should have a fix into the stable branch soon, but unfortunately it won't make it into the about-to-be-released Facter 3.0.2.

Thanks!

 

Thanks,
Federico


Il giorno lunedì 20 luglio 2015 22:24:18 UTC+2, Peter Huene ha scritto:

On Monday, July 20, 2015 at 1:23:20 PM UTC-7, Peter Huene wrote:

On Monday, July 20, 2015 at 1:16:07 PM UTC-7, Federico Agnelli wrote:

Hi Giorgio,

I have the same problem with a puppet client on Intel NUC5i5RYH.
All other clients (on different hardware) with the same puppet version work fine.

My puppet version for both agent and server is 4.2.0. Version 4.1.0 was ok.

It looks like Facter is expecting the contents of those files to be UTF-8, which may not be the correct thing to do.  Could you install/run the dmiencode utility and paste the output?  Also, cat the same files Giorgio did and attach as a file (the characters are coming through as invalid characters in Google Groups).  Thanks!

Correction: the utility is *dmidecode*.

 

 

Did you find a fix?

Thanks,
Federico

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/164d75be-37df-47aa-a341-a093cda95cdb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Join us at PuppetConf 2015, October 5-9 in Portland, OR - www.2015.puppetconf.com 

Register early to save 40%!

[Federico Agnelli]

Thanks to you Peter!
Bye,
Fede