Jira (FACT-3099) Caching UTF8 values results in "\\xCC" from ASCII-8BIT to UTF-8

33 views
Skip to first unread message

Steve Traylen (Jira)

unread,
Nov 18, 2021, 10:27:16 AM11/18/21
to puppe...@googlegroups.com
Steve Traylen created an issue
 
Facter / Bug FACT-3099
Caching UTF8 values results in "\\xCC" from ASCII-8BIT to UTF-8
Issue Type: Bug Bug
Affects Versions: FACT 4.2.5
Assignee: Unassigned
Components: Facter 4
Created: 2021/11/18 7:26 AM
Priority: Normal Normal
Reporter: Steve Traylen

This is with the ec2_userdata fact but I expect true for any fact where the value is UTF-8 and the fact is cached.

Userdata is utf-8

curl --silent  http://169.254.169.254/openstack/latest/user_data | grep --color='auto' -P "[^[:ascii:]]" 
  - [ vdb, /build1, "xfs", "nofail̦", "0", "0" ]

When this is cached via a configuration of 

facts: {
 
  ttls: [
 
      {
 
          "EC2": "7 days"
 
      }
 
 
 
  ]
 
}

Then this results in a debug of

facter --debug ec2_userdata
 
 
 
[2021-11-18 16:13:40.964313 ] DEBUG Facter::Util::Resolvers::Http - Request to http://169.254.169.254/latest/api/token failed with error code 404 
[2021-11-18 16:13:42.090980 ] DEBUG Facter::CacheManager - EC2 facts cache file expired, missing or is corrupt                                    
[2021-11-18 16:13:42.091039 ] DEBUG Facter::CacheManager - caching values for EC2 facts                                                           
Traceback (most recent call last):                                                                                                                
        15: from /opt/puppetlabs/puppet/bin/facter:10:in `<main>'                                                                                 
        14: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/cli/cli_launcher.rb:23:in `start'                                   
        13: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'                                       
        12: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'                                         
        11: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'                        
        10: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'                                       
         9: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/cli/cli.rb:124:in `query'                                           
         8: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.rb:454:in `to_user_output'                                                    
         7: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.rb:521:in `resolve_facts_for_user_query'                                      
         6: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/core/fact_manager.rb:29:in `resolve_facts'                          
         5: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/core/cache_manager.rb:38:in `cache_facts'                           
         4: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/core/cache_manager.rb:166:in `write_cache'                          
         3: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/core/cache_manager.rb:166:in `each'
 
         2: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/core/cache_manager.rb:175:in `block in write_cache'
 
         1: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/json/common.rb:286:in `pretty_generate'
 
/opt/puppetlabs/puppet/lib/ruby/2.7.0/json/common.rb:286:in `generate': "\\xCC" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)

Note that if I remove the cache configuration completly then everything is fine and
the facter value correctly contains UTF-8.

```
facter ec2_userdata | grep --color='auto' -P "[^[:ascii:]]"

  • [ vdb, /build1, "xfs", "nofail̦", "0", "0" ]
    ```
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Steve Traylen (Jira)

unread,
Nov 18, 2021, 10:28:03 AM11/18/21
to puppe...@googlegroups.com
Steve Traylen updated an issue
Change By: Steve Traylen
This is with the ec2_userdata fact but I expect true for any fact where the value is UTF-8 and the fact is cached.

Userdata is utf-8

{noformat}

curl --silent  http://169.254.169.254/openstack/latest/user_data | grep --color='auto' -P "[^[:ascii:]]"
  - [ vdb, /build1, "xfs", "nofail̦", "0", "0" ]
{noformat}


When this is cached via a configuration of

{noformat}

facts: {
  ttls: [
      {
          "EC2": "7 days"
      }

  ]
}
{noformat}


Then this results in  a debug of

{noformat}

facter --debug ec2_userdata

[2021-11-18 16:13:40.964313 ] DEBUG Facter::Util::Resolvers::Http - Request to http://169.254.169.254/latest/api/token failed with error code 404
[2021-11-18 16:13:42.090980 ] DEBUG Facter::CacheManager - EC2 facts cache file expired, missing or is corrupt                    
[2021-11-18 16:13:42.091039 ] DEBUG Facter::CacheManager - caching values for EC2 facts                    
Traceback (most recent call last):                    
        15: from /opt/puppetlabs/puppet/bin/facter:10:in `<main>'                    
        14: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/cli/cli_launcher.rb:23:in `start'                    
        13: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'                    
        12: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'                    
        11: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'                    
        10: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'                    
         9: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/cli/cli.rb:124:in `query'                    
         8: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.rb:454:in `to_user_output'                    
         7: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.rb:521:in `resolve_facts_for_user_query'                    
         6: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/core/fact_manager.rb:29:in `resolve_facts'                    
         5: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/core/cache_manager.rb:38:in `cache_facts'                    
         4: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/core/cache_manager.rb:166:in `write_cache'                    
         3: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/core/cache_manager.rb:166:in `each'
         2: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/core/cache_manager.rb:175:in `block in write_cache'
         1: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/json/common.rb:286:in `pretty_generate'
/opt/puppetlabs/puppet/lib/ruby/2.7.0/json/common.rb:286:in `generate': "\\xCC" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
{noformat}


Note that if I remove the cache configuration completly then everything is fine and
the facter value correctly contains UTF-8.

``` {noformat}
facter ec2_userdata | grep --color='auto' -P "[^[:ascii:]]"
  - [ vdb, /build1, "xfs", "nofail̦", "0", "0" ]
``` {noformat}



Steve Traylen (Jira)

unread,
Nov 18, 2021, 10:39:02 AM11/18/21
to puppe...@googlegroups.com
Steve Traylen commented on Bug FACT-3099
 
Re: Caching UTF8 values results in "\\xCC" from ASCII-8BIT to UTF-8

Should have added that with facter 3.14.15 the cache is created correctly as UTF-8.

```
file /opt/puppetlabs/facter/cache/cached_facts/EC2
/opt/puppetlabs/facter/cache/cached_facts/EC2: UTF-8 Unicode text, with very long lines
```

and the returned value from cache is UTF-8.

Ciprian Badescu (Jira)

unread,
Nov 29, 2021, 10:18:01 AM11/29/21
to puppe...@googlegroups.com
Ciprian Badescu updated an issue
 
Change By: Ciprian Badescu
Team: Night's Watch

Ciprian Badescu (Jira)

unread,
Nov 29, 2021, 10:19:02 AM11/29/21
to puppe...@googlegroups.com
Ciprian Badescu updated an issue
Change By: Ciprian Badescu
Labels: needs_repro

Dorin Pleava (Jira)

unread,
Dec 3, 2021, 7:22:02 AM12/3/21
to puppe...@googlegroups.com
Dorin Pleava commented on Bug FACT-3099
 
Re: Caching UTF8 values results in "\\xCC" from ASCII-8BIT to UTF-8

Managed to reproduce this with a configured ec2 instance that had ec2_userdata set to some UTF-8 values.
I did not reproduce this with a custom fact that returns UTF-8 data.

Also this ""
xCC" from ASCII-8BIT to UTF-8" happens when outputting combination if ASCII and UTF-8 facts. 

$ facter ec2_userdata
 
- [ vdb, /build1, "xfs", "nofail̦", "0", "0" ]
 
$ facter uptime
 
0:49 hours
 
$ facter uptime ec2_userdata
 
Traceback (most recent call last):
 
	13: from /opt/puppetlabs/puppet/bin/facter:10:in `<main>'
 
	12: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/cli/cli_launcher.rb:23:in `start'
 
	11: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'
 
	10: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
 
	 9: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
 
	 8: from /opt/puppetlabs/puppet/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
 
	 7: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/cli/cli.rb:124:in `query'
 
	 6: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.rb:458:in `to_user_output'
 
	 5: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/formatters/legacy_fact_formatter.rb:13:in `format'
 
	 4: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/formatters/legacy_fact_formatter.rb:39:in `format_for_multiple_user_queries'
 
	 3: from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter/framework/formatters/legacy_fact_formatter.rb:63:in `hash_to_facter_format'
 
	 2: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/json/common.rb:286:in `pretty_generate'
 
	 1: from /opt/puppetlabs/puppet/lib/ruby/2.7.0/json/common.rb:286:in `generate'
 
/opt/puppetlabs/puppet/lib/ruby/2.7.0/json/common.rb:286:in `to_json': "\\xCC" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)

Ciprian Badescu (Jira)

unread,
Dec 6, 2021, 3:33:03 AM12/6/21
to puppe...@googlegroups.com
Ciprian Badescu updated an issue
Change By: Ciprian Badescu
Sprint: NW - 2021-12-17

Ciprian Badescu (Jira)

unread,
Dec 6, 2021, 4:14:03 AM12/6/21
to puppe...@googlegroups.com
Ciprian Badescu updated an issue
Change By: Ciprian Badescu
Story Points: 2

Dorin Pleava (Jira)

unread,
Dec 6, 2021, 9:44:02 AM12/6/21
to puppe...@googlegroups.com
Dorin Pleava assigned an issue to Dorin Pleava
Change By: Dorin Pleava
Assignee: Dorin Pleava
Reply all
Reply to author
Forward
0 new messages