Jira (BOLT-1528) Bolt returning json data for many nodes causing memory contention and very slow bolt task runs.

9 views
Skip to first unread message

Austin Boyd (JIRA)

unread,
Nov 4, 2019, 12:42:04 PM11/4/19
to puppe...@googlegroups.com
Austin Boyd updated an issue
 
Puppet Task Runner / Improvement BOLT-1528
Bolt returning json data for many nodes causing memory contention and very slow bolt task runs.
Change By: Austin Boyd
Labels: jira_escalated
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Austin Boyd (JIRA)

unread,
Nov 4, 2019, 12:42:04 PM11/4/19
to puppe...@googlegroups.com

Austin Boyd (JIRA)

unread,
Nov 4, 2019, 12:42:05 PM11/4/19
to puppe...@googlegroups.com
Chris Webster created an issue
Issue Type: Improvement Improvement
Assignee: Unassigned
Created: 2019/11/04 9:41 AM
Priority: Normal Normal
Reporter: Chris Webster

Bolt returning json data for many nodes can cause the local system to use an excessive amount of memory. This does not cause the bolt run to fail but causes the run to stall and take much longer to complete the more nodes being executed on. We have requested that they verify this theory by monitoring memory usage during a bolt task run but have not received a response to that request.

As seen with this ticket a local run of the Powershell script completes in 30 seconds. Running with bolt on 40 servers it takes 20 to 30 minutes and can result in inconsistent errors. Adding more agents only makes the issue whorse.

Lowing concurrency from 100 to 25 did not have any effect on performance as reported by the client.

----- original message -----
Hello Jose,

We haven't heard from you in a while and wanted to check in to see if you needed any additional help. In the meantime, I'm setting this issue to resolved; but if you'd like to return to working on this, you're welcome to reply at https://support.puppetlabs.com/requests/36583 to re-open the issue.

Thank you,
Chris

* Please see Zendesk Support tab for further comments and attachments.

Chris Webster (JIRA)

unread,
Nov 4, 2019, 12:47:03 PM11/4/19
to puppe...@googlegroups.com
Chris Webster updated an issue
Change By: Chris Webster
Bolt returning json data for many nodes can cause the local system to use an excessive amount of memory. This does not cause the bolt run to fail but causes the run to stall and take much longer to complete the more nodes being executed on. We have requested that they verify this theory  by monitoring memory usage during a bolt task run but have not received a response to that request.

As seen with this ticket a local run of the Powershell script completes in 30 seconds. Running with bolt on 40 servers it takes 20 to 30 minutes and can result in inconsistent  errors. Adding more agents only makes the issue whorse.



Lowing concurrency from 100 to 25 did not have any effect on performance as reported by the client.

----- original message -----  
Hello Jose,

We haven't heard from you in a while and wanted to check in to see if you needed any additional help. In the meantime, I'm setting this issue to resolved; but if you'd like to return to working on this, you're welcome to reply at  https://support.puppetlabs.com/requests/36583 to re-open the issue.

Thank you,
Chris

~* Please see Zendesk Support tab for further comments and attachments.~

Chris Webster (JIRA)

unread,
Nov 4, 2019, 12:49:03 PM11/4/19
to puppe...@googlegroups.com

Chris Webster (JIRA)

unread,
Nov 4, 2019, 12:55:02 PM11/4/19
to puppe...@googlegroups.com
Chris Webster updated an issue
Bolt returning json JSON data for many nodes can cause the local system to use an excessive amount of memory. This does not cause the bolt Bolt run to fail but causes the run to stall and take much longer to complete the more nodes being executed on. We have requested that they verify this theory by monitoring memory usage during a bolt task run but have not received a response to that request.

As seen with this ticket a local run of the Powershell script completes in 30 seconds. Running with
bolt Bolt on 40 servers it takes 20 to 30 minutes and can result in inconsistent errors. Adding more agents only makes the issue whorse worse .


Lowing concurrency from 100 to 25 did not have any effect on performance as reported by the client.

 

~* Please see Zendesk Support tab for further comments and attachments.~
Reply all
Reply to author
Forward
0 new messages