Jira (PUP-6447) Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-6447 Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows Change By: Ethan Brown Fix Version/s: PUP 4.6.x Add Comment

This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9)

[Ethan Brown (JIRA)]

Ethan Brown created an issue

Puppet / PUP-6447 Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Issue Type: Bug Assignee: Unassigned Components: Windows Created: 2016/06/27 4:10 PM Labels: windows i18n utf-8 Priority: Normal Reporter: Ethan Brown UTF-8 files produced on Windows often contain a BOM. Puppet now reads most files as utf-8, but doesn't allow for them to have a leading BOM. This can be a problem for end users on Windows, where they might need to go through workarounds to produce content without a BOM. An easy solution is to modify any calls specifying utf-8 when opening files for read, to instead specify bom|utf-8. Ruby will handle the BOM correctly in this case.

Add Comment

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-6447 Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Change By: Ethan Brown

UTF-8 files produced on Windows often contain a BOM.  Puppet now reads most files as utf-8, but doesn't allow for them to have a leading BOM.

This can be a problem for end users on Windows, where they might need to go through workarounds to produce content without a BOM.   For instance, PowerShell can write files as UTF-8 with the {{Out-File}} cmdlet, specifying {{-Encoding UTF8}} - but the files are written with a BOM, which ends up blowing up Ruby if we try to read the files in Puppet.   An easy solution is to modify any calls specifying {{utf-8}} when opening files for read, to instead specify {{bom|utf-8}}.  Ruby will handle the BOM correctly in this case.

Add Comment

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-6447 Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows Change By: Ethan Brown

UTF-8 files produced on Windows often contain a BOM.  Puppet now reads most files as utf-8, but doesn't allow for them to have a leading BOM.

This can be a problem for end users on Windows, where they might need to go through workarounds to produce content without a BOM.  For instance, PowerShell can write files as UTF-8 with the {{Out-File}} cmdlet, specifying {{-Encoding UTF8}} - but the files are written with a BOM, which ends up blowing up Ruby if we try to read the files in Puppet  (under certain circumstances) .

An easy solution is to modify any calls specifying {{utf-8}} when opening files for read, to instead specify {{bom|utf-8}}.  Ruby will handle the BOM correctly in this case.

Note that the Ruby {{YAML.load_file}} API already handles the BOM situation correctly, so {{YAML}} files will generally be excluded from this problem.

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-6447

Re: Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows Language specification needs to be updated for this.

Add Comment

[Craig Gomes (JIRA)]

Craig Gomes updated an issue

Puppet / PUP-6447

Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Change By: Craig Gomes Sprint: Windows 2016-08-10

Add Comment

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-6447 Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Change By: Ethan Brown Sprint: Windows 2016-09-21

Add Comment

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-6447 Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Change By: Ethan Brown Fix Version/s: PUP 4.6.z Fix Version/s: PUP 4.8.0

Add Comment

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-6447 Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Change By: Ethan Brown Sprint: Windows 2016-09-21

Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-6447 Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Change By: Ethan Brown Team: Agent & Platform

Add Comment

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-6447 Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Change By: Ethan Brown Fix Version/s: PUP 4.8.0

Add Comment

[Erick Banks (JIRA)]

Erick Banks commented on PUP-6447

Re: Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows Encountered this again in Windows UTF-8 testing. Saved DSC manifest "UTF-8 w/ BOM" and errored out with: Error: Could not parse for environment production: Syntax error at '∩' at C:/Use rs/Administrator/Desktop/blah.pp:1:1 on node x9bshj31wdcfefy.delivery.puppetlabs .net Saved to "UTF-8" and executed same manifest successfully.

Add Comment

[Jeremy Adams (JIRA)]

Jeremy Adams commented on PUP-6447

Re: Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Recently when doing an integration in javascript that wrote a facter fact in JSON, I encountered this issue. Facter would not tolerate UTF-8 with a BOM for the structured JSON fact. var command = "New-Item -path '" + factsDir + "' -type directory -force \n" + "$MyPath = '" + factsFile + "' \n" + "$MyFile = " + oneLineFacts + " \n" + "$Utf8NoBomEncoding = New-Object System.Text.UTF8Encoding($False) \n" + "[System.IO.File]::WriteAllLines($MyPath, $MyFile, $Utf8NoBomEncoding)";

Add Comment

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-6447

Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Change By: Ethan Brown Labels: i18n  triaged  utf-8 windows

Add Comment

[Ethan Brown (JIRA)]

Ethan Brown updated an issue

Puppet / PUP-6447 Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Change By: Ethan Brown Priority: Normal Major

Add Comment

This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)

[Ethan Brown (JIRA)]

Ethan Brown commented on PUP-6447

Re: Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows Note that this issue has specifically been raised recently around using ERB templates for PowerShell files in MODULES-1996. The call to Puppet::FileSystem.read_preserve_line_endings doesn't use the form of read that allows BOMs when reading files - see https://github.com/puppetlabs/puppet/blob/5b8386ce9edf944ceee8328526dbb9f238c1403a/lib/puppet/file_system/windows.rb#L109 and https://github.com/puppetlabs/puppet/blob/5b8386ce9edf944ceee8328526dbb9f238c1403a/lib/puppet/file_system/file_impl.rb#L80 It should be straightforward enough to tactically replace that one problem area (in a separate ticket).

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-6447

Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Change By: Josh Cooper Team: Coremunity Night's Watch

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-6447 Allow UTF8 files to have a leading BOM to be more tolerant of files produced on Windows

Change By: Josh Cooper Epic Link: PUP-6719

Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)