Jira (FACT-3117) Azure metadata not fetching tags

28 views
Skip to first unread message

Alvin Rodis (Jira)

unread,
Apr 27, 2022, 6:08:02 AM4/27/22
to puppe...@googlegroups.com
Jordi Garcia created an issue
 
Facter / Bug FACT-3117
Azure metadata not fetching tags
Issue Type: Bug Bug
Assignee: Unassigned
Created: 2022/04/27 3:07 AM
Priority: Normal Normal
Reporter: Jordi Garcia

After migration from 2019.8.9 to 2021.5.0 the customer removed `keirans/azuremetadata` module and started to use default puppet for fetching az_metadata.

After the upgrade the agent runs are coming across the following error:

* Please see Zendesk Support tab for further comments and attachments.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)
Atlassian logo

Alvin Rodis (Jira)

unread,
Apr 27, 2022, 6:08:03 AM4/27/22
to puppe...@googlegroups.com
Alvin Rodis updated an issue
Change By: Alvin Rodis
Zendesk Ticket Count: 1
Zendesk Ticket IDs: 48158

Alvin Rodis (Jira)

unread,
Apr 27, 2022, 6:08:03 AM4/27/22
to puppe...@googlegroups.com
Alvin Rodis updated an issue
Change By: Alvin Rodis
Labels: jira_escalated

Jordi Garcia (Jira)

unread,
Apr 27, 2022, 6:11:02 AM4/27/22
to puppe...@googlegroups.com
Jordi Garcia updated an issue
Change By: Jordi Garcia
After migration from 2019.8.9 to 2021.5.0 the customer removed `keirans/azuremetadata` module and started to use default puppet for fetching az_metadata.

After the upgrade the agent runs are coming across the following error:

 
{code:java}
~ Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Data Provider type mismatch: Got String when a hash-like object was expected to access value using 'environment' from key 'facts.az_metadata.compute.tags.environment' on node{code}
 

 

The customer has implemented the following code in `hiera.yaml`:

 
---
version: 5
 
defaults:
  datadir: data
  data_hash: yaml_data
 
hierarchy:
  - name: "Per-node data (eyaml version)"
    globs:
      - "per_role/%\{facts.az_metadata.compute.tags.environment}/%\{trusted.extensions.pp_role}.yaml"
      - "per_region/%\{facts.az_metadata.compute.location}/%\{facts.az_metadata.compute.tags.environment}/%\{facts.az_metadata.compute.tags.application}/
* Please see Zendesk Support tab for further comments and attachments . ~ yaml"
      - "per_region/%\{facts.az_metadata.compute.location}/%\{facts.az_metadata.compute.tags.service}/%\{facts.az_metadata.compute.tags.application}/*.yaml"
      - "secrets/%\{facts.az_metadata.compute.tags.environment}/*.yaml"
      - "secrets/*.yaml"      
      - "per_environment/%\{facts.az_metadata.compute.tags.environment}/%\{facts.az_metadata.compute.tags.application}/*.yaml"
      - "per_environment/%\{facts.az_metadata.compute.tags.environment}/%\{facts.az_metadata.compute.tags.environment}.yaml"
      - "per_service/%\{facts.az_metadata.compute.tags.service}/%\{facts.az_metadata.compute.tags.application}/%\{facts.az_metadata.compute.location}/%\{facts.az_metadata.compute.tags.environment}.yaml"
      - "per_service/%\{facts.az_metadata.compute.tags.service}/%\{facts.az_metadata.compute.tags.application}/%\{facts.az_metadata.compute.tags.environment}.yaml"
      - "per_service/%\{facts.az_metadata.compute.tags.service}/%\{facts.az_metadata.compute.tags.environment}.yaml"
      - "per_os/%\{facts.os.name}.yaml"
 
 

Jordi Garcia (Jira)

unread,
Apr 27, 2022, 6:12:01 AM4/27/22
to puppe...@googlegroups.com
Jordi Garcia updated an issue
After migration from 2019.8.9 to 2021.5.0 the customer removed `keirans/azuremetadata` module and started to use default puppet for fetching az_metadata.

After the upgrade the agent runs are coming across the following error:

 
{code:java}

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Data Provider type mismatch: Got String when a hash-like object was expected to access value using 'environment' from key 'facts.az_metadata.compute.tags.environment' on node{code}
 

 

The customer has implemented the following code in `hiera.yaml`:

 
---
version: 5
 
defaults:
  datadir: data
  data_hash: yaml_data
 
hierarchy:
  - name: "Per-node data (eyaml version)"
    globs:
      - "per_role/%\{facts.az_metadata.compute.tags.environment}/%\{trusted.extensions.pp_role}.yaml"
      - "per_region/%\{facts.az_metadata.compute.location}/%\{facts.az_metadata.compute.tags.environment}/%\{facts.az_metadata.compute.tags.application}/*.yaml"

      - "per_region/%\{facts.az_metadata.compute.location}/%\{facts.az_metadata.compute.tags.service}/%\{facts.az_metadata.compute.tags.application}/*.yaml"
      - "secrets/%\{facts.az_metadata.compute.tags.environment}/*.yaml"
      - "secrets/*.yaml"      
      - "per_environment/%\{facts.az_metadata.compute.tags.environment}/%\{facts.az_metadata.compute.tags.application}/*.yaml"
      - "per_environment/%\{facts.az_metadata.compute.tags.environment}/%\{facts.az_metadata.compute.tags.environment}.yaml"
      - "per_service/%\{facts.az_metadata.compute.tags.service}/%\{facts.az_metadata.compute.tags.application}/%\{facts.az_metadata.compute.location}/%\{facts.az_metadata.compute.tags.environment}.yaml"
      - "per_service/%\{facts.az_metadata.compute.tags.service}/%\{facts.az_metadata.compute.tags.application}/%\{facts.az_metadata.compute.tags.environment}.yaml"
      - "per_service/%\{facts.az_metadata.compute.tags.service}/%\{facts.az_metadata.compute.tags.environment}.yaml"
      - "per_os/%\{facts.os.name}.yaml"
 
 

Nirupama Mantha (Jira)

unread,
Apr 27, 2022, 11:55:02 AM4/27/22
to puppe...@googlegroups.com
Nirupama Mantha updated an issue
Change By: Nirupama Mantha
Acceptance Criteria: Can retrieve tags from azure metadata facts and use that in a Hiera configuration

Nirupama Mantha (Jira)

unread,
Apr 27, 2022, 11:56:02 AM4/27/22
to puppe...@googlegroups.com
Nirupama Mantha updated an issue
Change By: Nirupama Mantha
Story Points: 3

Nirupama Mantha (Jira)

unread,
Apr 27, 2022, 11:57:10 AM4/27/22
to puppe...@googlegroups.com
Nirupama Mantha updated an issue
Change By: Nirupama Mantha
Fix Version/s: FACT 4.2.10

Michael Hashizume (Jira)

unread,
Apr 27, 2022, 12:04:02 PM4/27/22
to puppe...@googlegroups.com
Michael Hashizume updated an issue
Change By: Michael Hashizume
Sprint: Phoenix 2022-05-11
Team: Phoenix

Josh Cooper (Jira)

unread,
Apr 27, 2022, 1:50:02 PM4/27/22
to puppe...@googlegroups.com
Josh Cooper commented on Bug FACT-3117
 
Re: Azure metadata not fetching tags

Ah facter 3 and 4 use slightly different API versions

https://github.com/keirans/azuremetadata/blob/f641e94340771b1873c0cbca27b035f278399493/lib/facter/azure_metadata.rb#L20

https://github.com/puppetlabs/facter/blob/3e5715ac88fd8cf2a0fe69a57d7f2b62f461bc21/lib/facter/resolvers/az.rb#L8

Also the Azure API docs say "The tags field is a string with the tags delimited by semicolons." so I don't believe tags have ever been collected in the way the customer is wanting. See
https://docs.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service?tabs=linux#sample-3-get-vm-tags

There is a "tagsList" option make JSON parsing easier.

Josh Cooper (Jira)

unread,
Apr 27, 2022, 6:31:01 PM4/27/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Fix Version/s: FACT 4.2.10

Josh Cooper (Jira)

unread,
Apr 27, 2022, 6:31:02 PM4/27/22
to puppe...@googlegroups.com
Josh Cooper commented on Bug FACT-3117
 
Re: Azure metadata not fetching tags

An earlier version of the 3rd party module used to call the 2018-04-02 version of the API and convert the "tags" as reported by Azure into a Hash:

https://github.com/keirans/azuremetadata/blob/02e18bff84783f11f15cc8f94c0f734fca950fe6/lib/facter/azure_metadata.rb#L22-L23

The module was modified to call a newer version of the API and the tag parsing wasn't preserved. See
https://github.com/keirans/azuremetadata/commit/1e5668fba79195a2904dd8ee64391cb208d7ba1f#diff-7c55f6f5e9b2f257c6ea24ca9e84d192f9a2689045474d0ec14f45cf2bf7df85L19-L24 The author explicitly states that this in intentional and 0.2.0 is a breaking change. See https://github.com/keirans/azuremetadata#api-version-pinned-to-2020-06-01

You should use the tagsList value instead which is native functionaly. As a result, 0.2.0 is a breaking change for previous users.
This native approach also gracefully handles special characters in tag keys and values unlike previous versions of the module.

For example, from the Azure docs: "The tags field is a string with the tags delimited by semicolons. This output can be a problem if semicolons are used in the tags themselves."

Facter 4 does not claim to maintain compatibility with the old version of the module, so the customer should either use the old version of the module or create a separate custom fact to convert the "az_metadata.compute.tagsList" Array into a Hash.

Josh Cooper (Jira)

unread,
Apr 29, 2022, 1:10:03 PM4/29/22
to puppe...@googlegroups.com
Josh Cooper commented on Bug FACT-3117

Summarizing comments here. TL;DR the 3rd party module used to transform the Azure tags metadata into the "az_metadata" structured fact, but the module was modified in 0.2.0 to no longer do that. Facter 4 does the same as 0.2.0 of the module. In other words, it returns the Azure metadata verbatim.

Here's the older version of the module https://github.com/keirans/azuremetadata/blob/02e18bff84783f11f15cc8f94c0f734fca950fe6/lib/facter/azure_metadata.rb#L22-L23
And the change to the module where tag transformation was removed: https://github.com/keirans/azuremetadata/commit/1e5668fba79195a2904dd8ee64391cb208d7ba1f#diff-7c55f6f5e9b2f257c6ea24ca9e84d192f9a2689045474d0ec14f45cf2bf7df85L19-L24
The author explicitly states the change is intentional and 0.2.0 is a breaking change. See https://github.com/keirans/azuremetadata#api-version-pinned-to-2020-06-01

The reason for the change is because (as mentioned in the Azure docs) "The tags field is a string with the tags delimited by semicolons. This output can be a problem if semicolons are used in the tags themselves."
https://docs.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service?tabs=linux#sample-3-get-vm-tags

Facter 4 is behaving as expected, so I'm going to close this as won't fix.

Reply all
Reply to author
Forward
0 new messages