Jira (PDB-3171) query compilation is very slow for queries with giant arrays

[Wyatt Alt (JIRA)]

Wyatt Alt created an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays Issue Type: Bug Assignee: Unassigned Attachments: query.json Created: 2016/11/04 10:07 AM Priority: Normal Reporter: Wyatt Alt Nick Lewis alerted me that queries like the attached take a long time (30s+) to compile. I did some crude sampling of stack dumps and it looks like (unsurprisingly) we're spending most of that time in pre-order-visit. We should do some targeted optimization of query compilation – even with 10k nodes in the array, 30s seems way excessive. The problem occurs with large numbers of "or" clauses as well, so there's no particular reason to think this issue is array-specific. Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Russell Mull (JIRA)]

Russell Mull updated an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Russell Mull Sprint: Hopper

Add Comment

[Karen Van der Veer (JIRA)]

Karen Van der Veer updated an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Karen Van der Veer Team: Systems Engineering

Add Comment

[Rob Browning (JIRA)]

Rob Browning assigned an issue to Rob Browning

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Rob Browning Assignee: Rob Browning

Add Comment

[Russell Mull (JIRA)]

Russell Mull updated an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Russell Mull Sprint: Hopper

Add Comment

[Margaret Lee (Jira)]

Margaret Lee updated an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Margaret Lee Epic Link: PE-31891

Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)

[Austin Blatt (Jira)]

Austin Blatt updated an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Austin Blatt Team: PuppetDB Skeletor

Add Comment

This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)

[Austin Blatt (Jira)]

Austin Blatt assigned an issue to Nick Lewis

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Austin Blatt Assignee: Rob Browning Nick Lewis

Add Comment

[Austin Blatt (Jira)]

Austin Blatt updated an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Austin Blatt Story Points: 1

Add Comment

[Cas Donoghue (Jira)]

Cas Donoghue updated an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Cas Donoghue Sprint: Skeletor 03/15/2023

Add Comment

[Austin Blatt (Jira)]

Austin Blatt commented on PDB-3171

Re: query compilation is very slow for queries with giant arrays This has successfully promoted to 2021.7 and main.

Add Comment

[Austin Blatt (Jira)]

Austin Blatt commented on PDB-3171

Re: query compilation is very slow for queries with giant arrays

Promoted into main as 2023.1.0-rc1-92-g7108a7f, tests look green except some ones that have been persistently red for weeks. Promoted into 2021.7.x as 2021.7.3-rc1-77-g6c56868, waiting to see results of pe-integration tests, which are still running

Add Comment

[Austin Blatt (Jira)]

Austin Blatt updated an issue

PuppetDB / PDB-3171

query compilation is very slow for queries with giant arrays

Change By: Austin Blatt Fix Version/s: PDB 7.12.2 Fix Version/s: PDB 8.0.0

Add Comment

[Austin Blatt (Jira)]

Austin Blatt commented on PDB-3171

Re: query compilation is very slow for queries with giant arrays Same set of old failures on main, so I'll say that's "good enough". Unfortunately, due to the state of internal CI, that's likely the best we can get right now.

Add Comment

[Austin Blatt (Jira)]

Austin Blatt updated an issue

PuppetDB / PDB-3171

query compilation is very slow for queries with giant arrays

Change By: Austin Blatt Release Notes: Bug Fix

Add Comment

[Austin Blatt (Jira)]

Austin Blatt updated an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Austin Blatt Release Notes Summary: Improved speed of queries that filter based on giant arrays.

Add Comment

[Ingrida Cazers (Jira)]

Ingrida Cazers commented on PDB-3171

Re: query compilation is very slow for queries with giant arrays Nick Lewis I'm planning to document this enhancement in PE 2023.1 and PE 2021.7.3, scheduled for release in April. I have two questions: 1. Is the following description correct? _*Improved performance when querying PuppetDB * This enhancement helps to improve performance for PuppetDB queries involving thousands of nodes. _ 2. Is this considered an improvement in PE, to be documented in the PE release notes, or should it be documented in the PuppetDB release notes? https://www.puppet.com/docs/puppetdb/7/release_notes.html?_ga=2.135361270.633001831.1678722544-1884905924.1648157504&_gac=1.52716316.1678735900.EAIaIQobChMItKWMttLZ_QIVvBKtBh2nZgD7EAAYASAEEgJkPvD_BwE&_gl=1*1rkuc9i*_ga*MTg4NDkwNTkyNC4xNjQ4MTU3NTA0*_ga_7PSYLBBJPT*MTY3ODk5OTczNi4zNjAuMS4xNjc5MDAwMjg3LjAuMC4w

Add Comment

[Ingrida Cazers (Jira)]

Ingrida Cazers commented on PDB-3171

Re: query compilation is very slow for queries with giant arrays

In a Slack conversation today, Nick and I agreed on the following text: Improved performance when querying PuppetDB This enhancement helps to improve performance for PuppetDB queries that contain large arrays, for example, if many nodes are enumerated or many terms are joined by a single "and" or "or" element.

Add Comment

[Ingrida Cazers (Jira)]

Ingrida Cazers commented on PDB-3171

Re: query compilation is very slow for queries with giant arrays

I will plan to document this update in the PE release notes, and will leave it to the PuppetDB develoment team to document it in the PuppetDB release notes.

Add Comment

[Jonathan Newman (Jira)]

Jonathan Newman updated an issue

PuppetDB / PDB-3171

query compilation is very slow for queries with giant arrays

Change By: Jonathan Newman Fix Version/s: PDB 7.12.2 Fix Version/s: PDB 7.13.0

Add Comment

[Charmaine Pritchett (Jira)]

Charmaine Pritchett updated an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Charmaine Pritchett Labels: jira_escalated query-engine-improvements

Add Comment

[Charmaine Pritchett (Jira)]

Charmaine Pritchett updated an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Charmaine Pritchett Zendesk Ticket Count: 1 Zendesk Ticket IDs: 51143

Add Comment

[Ingrida Cazers (Jira)]

Ingrida Cazers updated an issue

PuppetDB / PDB-3171 query compilation is very slow for queries with giant arrays

Change By: Ingrida Cazers Labels: docs docs-reviewed jira_escalated query-engine-improvements

Add Comment