[Puppet - Bug #15062] puppet fails if template contains invalid utf-8

[tic...@puppetlabs.com]

Issue #15062 has been updated by Mathieu Arnold.

Hi,

I’ve been testing ruby 1.9.3 on a test server, and I’m getting this issue too.

Saying that “everything has to be UTF-8” is a nice thing, but in the real world, it’s not happening any time soon. I need to be able to distribute non UTF-8 files, I even have a couple of files containing delimited but unescaped binary values. Enforcing UTF-8 for .pp files should not, I think, harm anyone, even if I don’t see a real good reason for, but enforcing it for template files is IMHO a bad idea, puppet should see those files as a bunch of bytes, that it has to ensure is present, but anything more is a bad idea.

Regards,

---

Bug #15062: puppet fails if template contains invalid utf-8

• Author: Chris Price
• Status: Needs Decision
• Priority: Normal
• Assignee: eric sorenson
• Category: templates
• Target version:
• Affected Puppet version: 2.7.16
• Keywords: character encoding binary utf8
• Branch:

If you attempt to use a file resource with a ‘content’ parameter pointing at a template, and the template contains binary content, you may get an error like this:

Error: Failed to apply catalog: Parameter content failed: Munging failed for value ...
invalid byte sequence in UTF-8

I’ve reproduced the failure in 2.7.16 and 3.x, though the error messages differ slightly between the two (and also depending on whether you repro via ‘apply’ or via master/agent run).

I’m attaching the binary file that I’m using to repro. Save it into a directory structure like this:

modules/mymod/templates/mytemplate.erb

Add the “modules” directory to your module path and then you can repro with the following manifest:

file { "/tmp/myfile":
    mode => 755,
    content => template("mymod/mytemplate.erb"),
}

Note that if you use the ‘source’ parameter rather than the ‘content’ parameter (and avoid calling the template function), the manifest can be applied successfully; so the issue is when bringing in binary data as a string.

---

You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://projects.puppetlabs.com/my/account

[tic...@puppetlabs.com]

Issue #15062 has been updated by Josh Cooper.

Mathieu Arnold wrote:

puppet should see those files as a bunch of bytes

That won’t work for templates, because Puppet needs to read the bytes, convert to a ruby string, and invoke ERB.new(str,...). So an encoding needs to be specified somewhere. We could assume UTF-8, and/or provide a mechanism for the caller to specify the encoding, as Deepak suggested.

This isn’t an issue for the source parameter, because in that case we just perform a binary copy.

[tic...@puppetlabs.com]

Issue #15062 has been updated by Mariusz Gronczewski.

At this point even having sensible default would help, as if for some reason environment isnt “right” puppet just fails with cryptic (for new user) message.

Could encoding just be made parameter to template ? like template(‘some/file.erb’,‘ISO-8859-2’) ? Then default could be UTF8 but few files needing specific encodunf could be easily fixed

[tic...@puppetlabs.com]

Issue #15062 has been updated by Mathieu Arnold.

Any news on that front ?

[tic...@puppetlabs.com]

Issue #15062 has been updated by Mathieu Arnold.

• Priority changed from Normal to High

---

Bug #15062: puppet fails if template contains invalid utf-8

• Author: Chris Price
• Status: Needs Decision

• Priority: High

• Assignee: eric sorenson
• Category: templates
• Target version:
• Affected Puppet version: 2.7.16
• Keywords: character encoding binary utf8
• Branch: