Jira (FACT-1354) Facter ec2_userdata handles binary data incorrectly

6 views
Skip to first unread message

Michael Smith (JIRA)

unread,
Mar 1, 2016, 1:52:02 PM3/1/16
to puppe...@googlegroups.com
Michael Smith created an issue
 
Facter / Bug FACT-1354
Facter ec2_userdata handles binary data incorrectly
Issue Type: Bug Bug
Affects Versions: FACT 3.1.4, FACT 3.0.0
Assignee: Unassigned
Created: 2016/03/01 10:51 AM
Priority: Normal Normal
Reporter: Michael Smith

On an ec2 instance with binary userdata specified (as in gzipped data), facter ec2_userdata emits extra bytes at the end. This means you can't pipe the data to gunzip and unzip it.

For comparison, try curl http://169.254.169.254/latest/user-data/ | gunzip.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc)
Atlassian logo

Michael Smith (JIRA)

unread,
Mar 8, 2016, 1:49:03 PM3/8/16
to puppe...@googlegroups.com
Michael Smith updated an issue
Change By: Michael Smith
Story Points: 3
Sprint: Client Triage
Scrum Team: Client Platform

Kylo Ginsberg (JIRA)

unread,
Apr 13, 2016, 6:54:14 PM4/13/16
to puppe...@googlegroups.com
Kylo Ginsberg updated an issue
Change By: Kylo Ginsberg
Sprint: Client  Triage  2016-05-18 (Freeze)
This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9)
Atlassian logo

Steve Barlow (JIRA)

unread,
Apr 27, 2016, 6:31:03 PM4/27/16
to puppe...@googlegroups.com
Steve Barlow updated an issue
Change By: Steve Barlow
Sprint: Client 2016-05-18 (Freeze)

Kylo Ginsberg (JIRA)

unread,
Apr 27, 2016, 6:33:04 PM4/27/16
to puppe...@googlegroups.com
Kylo Ginsberg commented on Bug FACT-1354
 
Re: Facter ec2_userdata handles binary data incorrectly

FACT-1024 was the 2.x version of this ticket.

Michael Smith (JIRA)

unread,
Feb 7, 2017, 2:47:02 PM2/7/17
to puppe...@googlegroups.com
Michael Smith updated an issue
 
Change By: Michael Smith
Labels: maintenance
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Raphaël Pinson (JIRA)

unread,
Feb 23, 2017, 4:07:02 AM2/23/17
to puppe...@googlegroups.com
Raphaël Pinson commented on Bug FACT-1354
 
Re: Facter ec2_userdata handles binary data incorrectly

For the record, if you don't care about the ec2_userdata fact, you can simply blank it with a file in /etc/puppetlabs/facter/facter.d/ that contains ec2_userdata=""

Maggie Dreyer (JIRA)

unread,
May 17, 2017, 2:15:05 PM5/17/17
to puppe...@googlegroups.com

Maggie Dreyer (JIRA)

unread,
May 17, 2017, 2:15:07 PM5/17/17
to puppe...@googlegroups.com

Maggie Dreyer (JIRA)

unread,
May 17, 2017, 2:15:09 PM5/17/17
to puppe...@googlegroups.com
Maggie Dreyer updated an issue
Change By: Maggie Dreyer
Labels: maintenance  triaged

Branan Riley (JIRA)

unread,
May 17, 2017, 2:17:07 PM5/17/17
to puppe...@googlegroups.com

Branan Riley (JIRA)

unread,
May 17, 2017, 2:17:07 PM5/17/17
to puppe...@googlegroups.com
Branan Riley updated an issue
Change By: Branan Riley
Labels: maintenance  triaged

Moses Mendoza (JIRA)

unread,
May 18, 2017, 1:45:55 PM5/18/17
to puppe...@googlegroups.com

Josh Cooper (JIRA)

unread,
Aug 3, 2017, 2:00:03 PM8/3/17
to puppe...@googlegroups.com

Travis Fields (JIRA)

unread,
Mar 8, 2018, 7:40:02 PM3/8/18
to puppe...@googlegroups.com

Any movement on this ticket, it is biting us a bit now. Would prefer to not go down the revert to PSON or fact blocking route.

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Branan Riley (JIRA)

unread,
Mar 21, 2018, 6:11:04 PM3/21/18
to puppe...@googlegroups.com
Branan Riley updated an issue
 
Change By: Branan Riley
Labels: cloud ec2 triaged

Travis Fields (JIRA)

unread,
Feb 25, 2019, 8:12:04 PM2/25/19
to puppe...@googlegroups.com
 
Re: Facter ec2_userdata handles binary data incorrectly

Perhaps base64 encode the string first and than decode out the other side

Travis Fields (JIRA)

unread,
Feb 25, 2019, 8:12:04 PM2/25/19
to puppe...@googlegroups.com

Joshua Keiser (JIRA)

unread,
Feb 27, 2019, 4:19:04 PM2/27/19
to puppe...@googlegroups.com
Joshua Keiser updated an issue
 
Change By: Joshua Keiser
CS Priority: Needs Priority

Branan Riley (JIRA)

unread,
Feb 27, 2019, 4:49:03 PM2/27/19
to puppe...@googlegroups.com
Branan Riley commented on Bug FACT-1354
 
Re: Facter ec2_userdata handles binary data incorrectly

With the changes in Puppet 5 and 6 to ensure we're using utf-8 everywhere, reporting non-utf8 data from EC2 metadata is gonna be even more complicated than it was when this ticket was filed. We pretty much can't send that data to the server now, so we probably shouldn't even be collecting it.

The ideal solution would be to plumb Puppet's datatypes down into facter, so we could report the userdata as binary. The next best solutions are to base64 encode it unconditionally, or just stop reporting it entirely. The first is basically just wishful thinking, and the second two are both breaking changes (though certainly more correct than the current behavior).

The compromise would be to conditionally omit or base64-encode the fact if it's not utf-8. This has some potentially gross impacts for certain users (for example, latin-1 text could look like utf-8 sometimes, when it's only using ASCII). But I think that'd only happen for cases that currently require a PSON or fact-blocking workaround, so those users wouldn't be any worse off. We could then switch to always encoding (or always omitting the fact) in Facter 4.

Travis Fields (JIRA)

unread,
Feb 28, 2019, 2:28:05 PM2/28/19
to puppe...@googlegroups.com

That is great for Facter 4 however. This is broken now and the only way to get around is disable ALL EC2 group facts or to ensure we don't use GZIP but would block us from having larger UserData scripts due to EC2 limits. What is the near term fix on this?

Thomas Kishel (JIRA)

unread,
May 9, 2019, 7:28:04 PM5/9/19
to puppe...@googlegroups.com

Might there be a way to provide guidance to detect the extra bytes and strip them?

Jarret Lavallee (JIRA)

unread,
May 16, 2019, 6:56:02 PM5/16/19
to puppe...@googlegroups.com
Jarret Lavallee updated an issue
 
Change By: Jarret Lavallee
CS Priority: Needs Priority Reviewed

Erik Swanson (Jira)

unread,
May 4, 2021, 1:39:03 AM5/4/21
to puppe...@googlegroups.com
Erik Swanson commented on Bug FACT-1354
 
Re: Facter ec2_userdata handles binary data incorrectly

I'm not sure why, but I wasn't able to get it working in `/etc/puppetlabs/facter/facter.d`.

However, I just fixed my fleet by using SSM to run this everywhere:

cat > /opt/puppetlabs/facter/facts.d/ec2_userdata.sh <<'EOF'
#!/bin/sh
echo 'ec2_userdata=""'
EOF
chmod 0555 /opt/puppetlabs/facter/facts.d/ec2_userdata.sh

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages