Jira (PDB-2634) PQL & Hashes: Provide hash projection capability for facts & resource params

36 views
Skip to first unread message

Kenneth Barber (JIRA)

unread,
Apr 19, 2016, 2:16:07 PM4/19/16
to puppe...@googlegroups.com
Kenneth Barber created an issue
 
PuppetDB / Task PDB-2634
PQL & Hashes: Provide hash projection capability for facts & resource params
Issue Type: Task Task
Assignee: Unassigned
Created: 2016/04/19 11:15 AM
Priority: Normal Normal
Reporter: Kenneth Barber

Alongside PDB-2632, we should provide a mechanism for projection leaf values inside a hash, for example on the new `inventory` and existing `resources` endpoint we have facts, trusted and parameters. The proposal is to use the dotted notation to express this projection in PQL (and some adequate way in AST also).
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9)
Atlassian logo

Kenneth Barber (JIRA)

unread,
Apr 19, 2016, 2:17:05 PM4/19/16
to puppe...@googlegroups.com
Kenneth Barber updated an issue
Change By: Kenneth Barber
Sprint: Hopper

Ryan Senior (JIRA)

unread,
May 23, 2016, 4:39:04 PM5/23/16
to puppe...@googlegroups.com
Ryan Senior updated an issue
Change By: Ryan Senior
Sprint: Hopper

Wyatt Alt (JIRA)

unread,
Jun 28, 2016, 7:26:03 PM6/28/16
to puppe...@googlegroups.com
Wyatt Alt updated an issue
Change By: Wyatt Alt
Story Points: 3 5

Wyatt Alt (JIRA)

unread,
Mar 28, 2017, 7:37:03 PM3/28/17
to puppe...@googlegroups.com
Wyatt Alt commented on Task PDB-2634
 
Re: PQL & Hashes: Provide hash projection capability for facts & resource params

This becomes way easier with the jsonb_object_agg function introduced in 9.5, so I'm blocking it on PE-17748
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Wyatt Alt (JIRA)

unread,
Apr 26, 2017, 6:36:02 PM4/26/17
to puppe...@googlegroups.com
Wyatt Alt updated an issue
Change By: Wyatt Alt
Alongside PDB-2632, we should provide a mechanism for projection leaf values inside a hash, for example on the new `inventory` and existing `resources` endpoint we have facts, trusted and parameters. The proposal is to use the dotted notation to express this projection in PQL (and some adequate way in AST also).


The deliverable here is that this kind of thing should work:

{code}
inventory[facts.uptime]{ facts.operatingsystem = "foo" }
{code}

Nate McCurdy (JIRA)

unread,
Jul 27, 2017, 8:28:02 PM7/27/17
to puppe...@googlegroups.com
Nate McCurdy commented on Task PDB-2634
 
Re: PQL & Hashes: Provide hash projection capability for facts & resource params

A PQL query that I intuitively thought I could do is this:

puppet query 'facts[certname,value] { name = "os.family" }'

When that didn't work, I tried the next thing that I intuitively thought should work:

puppet query 'inventory[certname, facts.os.family]{ }'

I was bummed when I found that wasn't the case. I'm a big +1 to this feature existing!

Nick Walker (JIRA)

unread,
Aug 10, 2017, 5:02:06 PM8/10/17
to puppe...@googlegroups.com
Nick Walker commented on Task PDB-2634

I was asked for "all hostnames of RHEL 7 systems in the production environment" and after lots of iteration it came out to be:

puppet query 'facts[value]{name = "fqdn" and certname in inventory[certname]{facts.os.family = "RedHat" and facts.os.release.major = "7"}}'

which is certainly much more confusing that the proposed which would come out to:

puppet query 'inventory[facts.fqdn]{facts.osfamily = "RedHat" and facts.os.release.major = "7"}'

Owen Rodabaugh (JIRA)

unread,
Dec 5, 2017, 11:47:03 AM12/5/17
to puppe...@googlegroups.com
Owen Rodabaugh updated an issue
 
Change By: Owen Rodabaugh
CS Priority: Normal
CS Impact: Improvements to make PQL easier to user/more easily understood would be a very good thing given it's the primary way to query data from the system as well to select nodes to run tasks against. 

PQL is central to using Puppet/PE and usability improvements should be prioritized.
CS Severity: 3 - Serious
CS Business Value: 4 - $$$$$
CS Frequency: 3 - 25-50% of Customers
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
Atlassian logo

Owen Rodabaugh (JIRA)

unread,
Feb 27, 2018, 5:16:03 PM2/27/18
to puppe...@googlegroups.com
Owen Rodabaugh updated an issue
Change By: Owen Rodabaugh
CS Priority: Normal Needs Priority
This message was sent by Atlassian JIRA (v7.5.1#75006-sha1:7df2574)
Atlassian logo

Reid Vandewiele (JIRA)

unread,
Feb 27, 2018, 6:28:05 PM2/27/18
to puppe...@googlegroups.com
Reid Vandewiele commented on Task PDB-2634
 
Re: PQL & Hashes: Provide hash projection capability for facts & resource params 

inventory[facts.sshrsakey, facts.sshecdsakey] { facts.os.family = "RedHat" and facts.os.release.major = "7" }

 

Seems like a pretty reasonable thing to query for, right?

This is honestly the closest I think I can get today.

#!/bin/bash
 
 
 
query='
 
  facts[certname,name,value] {
 
    (
 
      name = "sshrsakey" or
 
      name = "sshecdsakey"
 
    )
 
    and inventory {
 
      facts.os.family = "RedHat" and
 
      facts.os.release.major = "7"
 
    }
 
  }
 
'
 
 
 
puppet query "$query" | /opt/puppetlabs/puppet/bin/ruby -rdeep_merge -rjson -e '
 
  puts JSON.pretty_generate(
 
    JSON.parse(STDIN.read).inject({}) { |h,e|
 
      h.deep_merge!({ e["certname"] => {e["name"] => e["value"]} })
 
    }
 
  )
 
'

I don't know how to get the data I want in a useable way without requiring ugly, manual transformation.

Owen Rodabaugh (JIRA)

unread,
Mar 1, 2018, 7:28:04 PM3/1/18
to puppe...@googlegroups.com
Owen Rodabaugh updated an issue
Change By: Owen Rodabaugh
Issue Type: Task Improvement

Owen Rodabaugh (JIRA)

unread,
Mar 1, 2018, 7:34:03 PM3/1/18
to puppe...@googlegroups.com
Owen Rodabaugh updated an issue
Change By: Owen Rodabaugh
CS Priority: Needs Priority Major
CS Impact:
Improvements to make PQL easier to user/more easily understood would be a very good thing given it's the primary way to query data from the system as well to select nodes to run tasks against.
This use case is likely the most common one for extracting node data.

PQL is central to using Puppet/PE and usability improvements should be prioritized.


Without this consultants and customers are forced to do a lot of post-processing of the full dataset to pick out the handful of fields they want as seen in Reid's example below. Common scenarios that are impacted like this are integrations with CMDBs and ticketing systems(e.g. Remedy) where customers want data grabbed regularly.
CS Business Value: 4 5 - $$$$$ $

Reid Vandewiele (JIRA)

unread,
Jul 6, 2018, 1:26:03 PM7/6/18
to puppe...@googlegroups.com
Reid Vandewiele updated an issue
Change By: Reid Vandewiele
Alongside PDB-2632, we should provide a mechanism for projection leaf values inside a hash, for example on the new `inventory` and existing `resources` endpoint we have facts, trusted and parameters. The proposal is to use the dotted notation to express this projection in PQL (and some adequate way in AST also).

The deliverable here is that this kind of thing should work:

{code}
inventory[facts.uptime]{ facts.operatingsystem = "foo" }
{code}


{code}
inventory[certname, facts.sshrsakey, facts.sshecdsakey] {

  facts.os.family = "RedHat" and
  facts.os.release.major = "7"
}
{code}
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Bill Diamond (JIRA)

unread,
Apr 22, 2019, 9:01:09 AM4/22/19
to puppe...@googlegroups.com
Bill Diamond commented on Improvement PDB-2634
 
Re: PQL & Hashes: Provide hash projection capability for facts & resource params

I am enthusiastic about this as it's relevant for the compliance use cases. The government team needs ongoing support for PG-SQL 9 until EDB or a vendor takes the open source PG-SQL 11 product through DISA STIG which has this as a native feature.  

Robert Roland (JIRA)

unread,
Apr 22, 2019, 4:32:05 PM4/22/19
to puppe...@googlegroups.com
Robert Roland updated an issue
 
Change By: Robert Roland
Labels: query-engine-improvements

Austin Blatt (JIRA)

unread,
Aug 26, 2019, 8:34:08 PM8/26/19
to puppe...@googlegroups.com
Austin Blatt assigned an issue to Austin Blatt
Change By: Austin Blatt
Assignee: Austin Blatt

Austin Blatt (JIRA)

unread,
Aug 26, 2019, 8:53:03 PM8/26/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Acceptance Criteria: In scope, format <endpoint> - (<parameter>, ....)

{{inventory}} - ({{facts}}, {{trusted}})
{{resources}} - ({{parameters}})

Out of scope, format <endpoint> - (<parameter>, ....)

{{factsets}} - ({{facts)}}

Austin Blatt (JIRA)

unread,
Aug 26, 2019, 8:56:04 PM8/26/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Fix Version/s: PDB 6.6.0

Austin Blatt (JIRA)

unread,
Aug 27, 2019, 12:07:05 PM8/27/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Acceptance Criteria: Fact projection for the following endpoints/parameters with AST and PQL support.

In scope, format <endpoint> - (<parameter>, ....)

{{inventory}} - ({{facts}}, {{trusted}})
{{resources}} - ({{parameters}})

Out of scope, format <endpoint> - (<parameter>, ....)

{{factsets}} - ({{facts)}}

Austin Blatt (JIRA)

unread,
Aug 27, 2019, 12:25:08 PM8/27/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Acceptance Criteria: Fact json projection for the following endpoints/parameters with AST and PQL support.


In scope, format <endpoint> - (<parameter>, ....)

{{inventory}} - ({{facts}}, {{trusted}})
{{resources}} - ({{parameters}})

Out of scope, format <endpoint> - (<parameter>, ....)

{{factsets}} - ({{facts)}}

Austin Blatt (JIRA)

unread,
Aug 28, 2019, 6:34:05 PM8/28/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Issue Type: Improvement New Feature

Austin Blatt (JIRA)

unread,
Sep 13, 2019, 1:11:07 PM9/13/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Fix Version/s: PDB 6.6.0

Austin Blatt (JIRA)

unread,
Sep 13, 2019, 1:11:08 PM9/13/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Fix Version/s: PDB 6.7.0

Austin Blatt (JIRA)

unread,
Sep 20, 2019, 2:52:06 PM9/20/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Release Notes Summary: This still needs release notes for the new feature. See the documentation in the commit for usage docs.

This is merging with a couple known issues,
1. a space is required after any dotted field (ie. inventory[facts.os.family ] {})
2. any dotted path has a maximum 63 characters
Release Notes: New Feature

Austin Blatt (JIRA)

unread,
Sep 27, 2019, 4:50:03 PM9/27/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Release Notes Summary:
This still needs release notes for the new feature. See the documentation in the commit for usage docs.
Known issue
This is merging with a couple known issues,
1. a space is required after any dotted field (ie. inventory[facts.os.family ] {})
 2. any dotted path has a maximum 63 characters

Austin Blatt (JIRA)

unread,
Sep 27, 2019, 4:51:03 PM9/27/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Release Notes Summary:
Known issue
2 The inventory and resource endpoints now support using dot notation for projections . any dotted path has a maximum 63 characters This will allow you to pick specific

Austin Blatt (JIRA)

unread,
Sep 27, 2019, 4:54:04 PM9/27/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Release Notes Summary: The inventory and resource endpoints now support using dot notation for projections. This will allow you to pick specific a specific fact or resource parameter to return instead of the entire json for facts or parameters.

Austin Blatt (JIRA)

unread,
Sep 27, 2019, 4:56:05 PM9/27/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Release Notes Summary: The inventory and resource endpoints now support using dot notation for projections. This will allow you to pick specific a specific fact or resource parameter to return instead of the entire json for facts or parameters. You can find examples of using dot notation in [PQL](./api/query/examples-pql.markdown) and [AST](./api/query/v4/ast.markdown)

Austin Blatt (JIRA)

unread,
Sep 27, 2019, 4:56:05 PM9/27/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Release Notes Summary:
The inventory and resource endpoints now support using dot notation for projections. This will allow you to pick specific a specific fact or resource parameter to return instead of the entire json for facts or parameters. You can find examples of using dot notation in [PQL](./api/query/examples-pql.markdown) and [AST](./api/query/v4/ast.markdown)


Known issue

Austin Blatt (JIRA)

unread,
Sep 27, 2019, 4:57:05 PM9/27/19
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Release Notes Summary:
The inventory and resource endpoints now support using dot notation for projections. This will allow you to pick specific a specific fact or resource parameter to return instead of the entire json for facts or parameters. You can find examples of using dot notation in [PQL](./api/query/examples-pql.markdown) and [AST](./api/query/v4/ast.markdown)

Known issue - PDB-4521

Heston Hoffman (JIRA)

unread,
Sep 30, 2019, 1:59:05 PM9/30/19
to puppe...@googlegroups.com
Heston Hoffman updated an issue
Change By: Heston Hoffman
Labels: query-engine-improvements resolved-issue-added

Claudia Petty (Jira)

unread,
Jun 21, 2023, 10:56:06 AM6/21/23
to puppe...@googlegroups.com
Claudia Petty updated an issue
Change By: Claudia Petty
Labels: new-feature query-engine-improvements resolved-issue-added
This message was sent by Atlassian Jira (v8.20.21#820021-sha1:38274c8)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages