Jira (BOLT-1301) multi-threaded/parallel execution of plans within a plan

8 views
Skip to first unread message

Nathan Giuliani (JIRA)

unread,
May 12, 2019, 7:39:05 PM5/12/19
to puppe...@googlegroups.com
Nathan Giuliani created an issue
 
Puppet Task Runner / New Feature BOLT-1301
multi-threaded/parallel execution of plans within a plan
Issue Type: New Feature New Feature
Assignee: Unassigned
Created: 2019/05/12 4:38 PM
Priority: Normal Normal
Reporter: Nathan Giuliani

Provide the ability to execute plans across multiple nodes in parallel. For example, consider the following scenario that relates to operating system patching:

  • Plan module::patch_and_reboot has the following steps:
    1. Perform some pre-patching tasks
    2. Install OS updates/patches
    3. Reboot the node and wait for it to return
    4. Perform some post-patching tasks

When executing the above plan on multiple nodes (likely from within from another plan that handles pre/post steps for the entire server list, result handling, etc.), the steps are serialized into a single, synchronous task sequence. If one node takes an hour for step 2 (install updates), but the rest only take 5 minutes, the remaining steps for all other nodes will wait for the single node to complete step 2 before moving on.

Workarounds are to code more logic into each of the steps (which are ultimately bolt tasks - scripts), or use an external script to call multiple instances of bolt executing the plan.

It would be ideal if this is something bolt could handle itself - either through the ability for run_plan to multi-thread plan execution, or through the implementation of multi-threading/background jobs in a similar fashion to Python threading or PowerShell runspaces.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Alex Dreyer (JIRA)

unread,
May 13, 2019, 2:17:03 PM5/13/19
to puppe...@googlegroups.com
Alex Dreyer commented on New Feature BOLT-1301
 
Re: multi-threaded/parallel execution of plans within a plan

We've discussed a few use cases for parallel execution in bolt and come to the conclusion that bolt needs a few different solutions tailored to specific use cases for parallelism. To clarify here the request here is to run the same series of steps across a single set of targets moving on to a subsequent step on one target as soon as the first one is finished?

Nathan Giuliani (JIRA)

unread,
May 14, 2019, 2:23:02 AM5/14/19
to puppe...@googlegroups.com

Yes, what you've described is how I see it. Using the example above with a plan module::patch_and_reboot, I'd expect to be able to call it from another plan as follows:

  • pan module:patch_lots_of_servers is run on 10 nodes, which does the following:
    1. Perform some global pre-patching steps like taking vm snapshots, and emailing users that patching is starting
    2. run_plan module::patch_and_reboot on all 10 nodes
      • i'd like puppet to run 10 parallel processes here, each executing the plan on an individual node
      • each node will work through the steps in that plan independently, but execution of this 'top-level' plan won't continue until all 10 parallel processes have been completed
    3. Perform some global post-patching steps like removing vm snapshots, and emailing users that patching has finished

I did try searching for an existing issue here and didn't come across BOLT-176, but I believe that it is a very similar request, it's just that this one relates to execution of a plan, not a task (which already happens in parallel across nodes based on the concurrency parameter).

Yasmin Rajabi (JIRA)

unread,
Aug 19, 2019, 6:05:03 PM8/19/19
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages