Jira (BOLT-1400) Add 'run_command' endpoint to bolt-server

0 views
Skip to first unread message

Casey Williams (JIRA)

unread,
Jun 17, 2019, 3:39:02 PM6/17/19
to puppe...@googlegroups.com
Casey Williams created an issue
 
Puppet Task Runner / Task BOLT-1400
Add 'run_command' endpoint to bolt-server
Issue Type: Task Task
Assignee: Unassigned
Created: 2019/06/17 12:38 PM
Priority: Normal Normal
Reporter: Casey Williams

Background

When told to run a task on some nodes, a PE master typically contacts the nodes over the PCP protocol. It sends a formatted request over PCP to the pxp-agent service running on the nodes, directing them to run the task locally. But if the nodes don't have a pxp-agent service running, the PE master must contact them via SSH or WinRM instead. For these cases, the PE master runs a "pe-bolt-server" service, which is a sinatra application that waits for post requests to /ssh/run_task or /winrm/run_task, then runs the task via an instance of the bolt executor, just like bolt does when you run a task via the CLI. Basically, it's a thin REST API wrapper around normal bolt "task run" operations that the PE master can use when there's no PXP agent to talk to over PCP.

In PE Kearney, "task run" is the only supported action over PCP, so it follows that the only endpoints in the bolt-server REST API are for "task run". For other typical bolt actions ("command run", "file upload", "script run"), PCP operations are handled by wrapping any non-task action in an ephemeral task in order to use the "task run" endpoints.

PCP-867 and ORCH-2304 describe process of enabling "command run" over PCP on the agent and server. This ticket describes the new endpoint on bolt-server that will support "command run" when there is no PCP transport available.

Requirements

Changes will be made to the bolt server app:

  • The app follows the json schema specification. Add a description of the JSON schema for data passed to the new endpoints here. This schema should match the one defined for the command action in PCP-867, that is:

    {
        "type": "object",
        "properties": {
            "command": {
                "type": "string"
            }
        }
    }
    

  • Add new POST endpoints in the transport sinatra app here to support running a command via SSH and WinRM, using the new JSON schema.

Testing

  • Write RSpec tests for the changes to bolt here.
  • Acceptance/integration tests are also maintained from the orchestrator side: add new integration test(s) for the interaction of the orchestrator and bolt-server while running a command here.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Casey Williams (JIRA)

unread,
Jun 17, 2019, 3:41:02 PM6/17/19
to puppe...@googlegroups.com

Casey Williams (JIRA)

unread,
Jun 17, 2019, 3:41:03 PM6/17/19
to puppe...@googlegroups.com

Casey Williams (JIRA)

unread,
Jun 17, 2019, 3:42:02 PM6/17/19
to puppe...@googlegroups.com
Casey Williams updated an issue
*Background*


When told to run a task on some nodes, a PE master typically contacts the nodes over the PCP protocol. It sends a formatted request over PCP to the pxp-agent service running on the nodes, directing them to run the task locally. But if the nodes don't have a pxp-agent service running, the PE master must contact them via SSH or WinRM instead. For these cases, the PE master runs a "pe-bolt-server" service, which is a sinatra application that waits for post requests to /ssh/run_task or /winrm/run_task, then runs the task via an instance of the bolt executor, just like bolt does when you run a task via the CLI. Basically, it's a thin REST API wrapper around normal bolt "task run" operations that the PE master can use when there's no PXP agent to talk to over PCP.

In PE Kearney, "task run" is the only supported action over PCP, so it follows that the only endpoints in the bolt-server REST API are for "task run". For other typical bolt actions ("command run", "file upload", "script run"), PCP operations are handled by wrapping any non-task action in an ephemeral task in order to use the "task run" endpoints.

PCP-867 and ORCH-2304 describe process of enabling "command run" without a task wrapper over PCP on the agent and server. This ticket describes the new endpoint on bolt-server that will support "command run" when there is no PCP transport available.

*Requirements*

Changes will be made to the [bolt server app|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server]:

* The app follows the [json schema specification|https://json-schema.org/specification.html]. Add a description of the JSON schema for data passed to the new endpoints [here|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server/schemas]. This schema should match the one defined for the command action in PCP-867, that is:
{code}{

    "type": "object",
    "properties": {
        "command": {
            "type": "string"
        }
    }
}
{code}

* Add new POST endpoints in the transport sinatra app [here|https://github.com/puppetlabs/bolt/blob/master/lib/bolt_server/transport_app.rb] to support running a command via SSH and WinRM, using the new JSON schema.

*Testing*

* Write RSpec tests for the changes to bolt [here|https://github.com/puppetlabs/bolt/tree/master/spec/bolt_server].
* Acceptance/integration tests are also maintained from the orchestrator side: add new integration test(s) for the interaction of the orchestrator and bolt-server while running a command [here|https://github.com/puppetlabs/orchestrator/blob/7a145bc93b33a1de075c95812ba39ed7eb198cc5/test/puppetlabs/orchestrator/integration/bolt_server.clj].

Sean McDonald (JIRA)

unread,
Jun 17, 2019, 3:56:03 PM6/17/19
to puppe...@googlegroups.com
Sean McDonald updated an issue
Change By: Sean McDonald
Sprint: Skeletor Kanban 20190619

Sean McDonald (JIRA)

unread,
Jun 17, 2019, 3:56:03 PM6/17/19
to puppe...@googlegroups.com

Casey Williams (JIRA)

unread,
Jun 17, 2019, 4:20:03 PM6/17/19
to puppe...@googlegroups.com
Casey Williams updated an issue
Change By: Casey Williams
*Background*

When told to run a task on some nodes, a PE master typically contacts the nodes over the PCP protocol. It sends a formatted request over PCP to the pxp-agent service running on the nodes, directing them to run the task locally. But if the nodes don't have a pxp-agent service running, the PE master must contact them via SSH or WinRM instead. For these cases, the PE master runs a "pe-bolt-server" service, which is a sinatra application that waits for post requests to /ssh/run_task or /winrm/run_task, then runs the task via an instance of the bolt executor, just like bolt does when you run a task via the CLI. Basically, it's a thin REST API wrapper around normal bolt "task run" operations that the PE master can use when there's no PXP agent to talk to over PCP.

In PE Kearney, "task run" is the only supported action over PCP, so it follows that the only endpoints in the bolt-server REST API are for "task run". For other typical bolt actions ("command run", "file upload", "script run"), PCP operations are handled by wrapping any non-task action in an ephemeral task in order to use the "task run" endpoints.

PCP-867 and ORCH-2304 describe process of enabling "command run" without a task wrapper over PCP on the agent and server. This ticket describes the new endpoint on bolt-server that will support "command run" when there is no PCP transport available.

*Requirements*

Changes will be made to the [bolt server app|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server]:

* The app follows the [json schema specification|https://json-schema.org/specification.html]. Add a description of the JSON schema for data passed to the new endpoints [here|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server/schemas]. This schema should match the one defined for the command action in PCP-867, that is:
{code}{
    "type": "object",
    "properties": {
        "command": {
            "type": "string"
        }
    }
}
{code}

* Add new POST endpoints in the transport sinatra app [here|https://github.com/puppetlabs/bolt/blob/master/lib/bolt_server/transport_app.rb] to support running a command via SSH and WinRM, using the new JSON schema.

*Testing*

* Write RSpec tests for the changes to bolt [here|https://github.com/puppetlabs/bolt/tree/master/spec/bolt_server].
* Acceptance/integration tests are also maintained from on the orchestrator side : add new integration test(s) exist for the interaction of the orchestrator and bolt-server while running a command this code [here|https://github.com/puppetlabs/orchestrator/blob/ 7a145bc93b33a1de075c95812ba39ed7eb198cc5 lovejoy /test/puppetlabs/orchestrator/integration/bolt_server.clj] , but updating them is *out of scope* for this ticket . *Do* run a quick manual test that the API endpoints work as expected on a running PE master, but making changes orchestrator to make use of the endpoints should be done as part of an ORCH ticket.

Casey Williams (JIRA)

unread,
Jun 17, 2019, 5:06:03 PM6/17/19
to puppe...@googlegroups.com
Casey Williams updated an issue
*Background*

When told to run a task on some nodes, a PE master typically contacts the nodes over the PCP protocol. It sends a formatted request over PCP to the pxp-agent service running on the nodes, directing them to run the task locally. But if the nodes don't have a pxp-agent service running, the PE master must contact them via SSH or WinRM instead. For these cases, the PE master runs a "pe-bolt-server" service, which is a sinatra application that waits for post requests to /ssh/run_task or /winrm/run_task, then runs the task via an instance of the bolt executor, just like bolt does when you run a task via the CLI. Basically, it's a thin REST API wrapper around normal bolt "task run" operations that the PE master can use when there's no PXP agent to talk to over PCP.

In PE Kearney, "task run" is the only supported action over PCP, so it follows that the only endpoints in the bolt-server REST API are for "task run". For other typical bolt actions ("command run", "file upload", "script run"), PCP operations are handled by wrapping any non-task action in an ephemeral task in order to use the "task run" endpoints.

PCP-867 and ORCH-2304 describe process of enabling "command run" without a task wrapper over PCP on the agent and server. This ticket describes the new endpoint on bolt-server that will support "command run" when there is no PCP transport available.

*Requirements*

Changes will be made to the [bolt server app|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server]:
* The app follows the [json schema specification|https://json-schema.org/specification.html]. Add a description of the JSON schema for data passed to the new endpoints [here|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server/schemas]. This schema should match the one defined for the command action in PCP-867, that is:
{code}{
    "type": "object",
    "properties": {
        "command": {
            "type": "string"
        }
    }
}
{code}

* Add new POST endpoints in the transport sinatra app [here|https://github.com/puppetlabs/bolt/blob/master/lib/bolt_server/transport_app.rb] to support running a command via SSH and WinRM, using the new JSON schema.
* Document the new endpoints in the [developer-docs|https://github.com/puppetlabs/bolt/blob/master/developer-docs/].

*Testing*
* Write RSpec tests for the changes to bolt [here|https://github.com/puppetlabs/bolt/tree/master/spec/bolt_server].
* Acceptance/integration tests on the orchestrator side exist for this code [here|https://github.com/puppetlabs/orchestrator/blob/lovejoy/test/puppetlabs/orchestrator/integration/bolt_server.clj], but updating them is *out of scope* for this ticket. *Do* run a quick manual test that the API endpoints work as expected on a running PE master, but making changes orchestrator to make use of the endpoints should be done as part of an ORCH ticket.

Geoff Nichols (JIRA)

unread,
Jun 19, 2019, 2:19:04 PM6/19/19
to puppe...@googlegroups.com
Geoff Nichols updated an issue
Change By: Geoff Nichols
Sprint: Skeletor Kanban 20190619 , Skeletor Kanban 20190703

Sean McDonald (JIRA)

unread,
Jul 3, 2019, 2:46:03 PM7/3/19
to puppe...@googlegroups.com
Sean McDonald assigned an issue to Unassigned
Change By: Sean McDonald
Assignee: Casey Williams

Branan Riley (JIRA)

unread,
Jul 3, 2019, 2:50:06 PM7/3/19
to puppe...@googlegroups.com
Branan Riley updated an issue
Change By: Branan Riley
Sprint: Skeletor Kanban 20190619, Skeletor Kanban 20190703 , Skeletor Kanban 20190717

Geoff Nichols (JIRA)

unread,
Jul 17, 2019, 2:13:06 PM7/17/19
to puppe...@googlegroups.com
Geoff Nichols updated an issue
Change By: Geoff Nichols
Sprint: Skeletor Kanban 20190619, Skeletor Kanban 20190703, Skeletor Kanban 20190717 , Skeletor Kanban 20190731

Branan Riley (JIRA)

unread,
Jul 31, 2019, 2:07:05 PM7/31/19
to puppe...@googlegroups.com
Branan Riley updated an issue
Change By: Branan Riley
Sprint: Skeletor Kanban 20190619, Skeletor Kanban 20190703, Skeletor Kanban 20190717, Skeletor Kanban 20190731 , Skeletor Kanban 20190814

Scott McClellan (JIRA)

unread,
Jul 31, 2019, 2:49:02 PM7/31/19
to puppe...@googlegroups.com

Branan Riley (JIRA)

unread,
Aug 14, 2019, 7:39:05 PM8/14/19
to puppe...@googlegroups.com
Branan Riley updated an issue
Change By: Branan Riley
Sprint: Skeletor Kanban 20190619, Skeletor Kanban 20190703, Skeletor Kanban 20190717, Skeletor Kanban 20190731, Skeletor Kanban 20190814 , Skeletor Kanban 20190828

Scott McClellan (JIRA)

unread,
Aug 20, 2019, 2:38:03 AM8/20/19
to puppe...@googlegroups.com

Chris Cowell (JIRA)

unread,
Aug 22, 2019, 11:50:03 PM8/22/19
to puppe...@googlegroups.com

Geoff Nichols (JIRA)

unread,
Aug 28, 2019, 8:37:03 PM8/28/19
to puppe...@googlegroups.com
Geoff Nichols updated an issue
Change By: Geoff Nichols
Labels: closed-github-move docs_reviewed run_command

Geoff Nichols (JIRA)

unread,
Aug 28, 2019, 8:39:04 PM8/28/19
to puppe...@googlegroups.com
Geoff Nichols commented on Task BOLT-1400
 
Re: Add 'run_command' endpoint to bolt-server

Scott McClellan - looks like this was automatically closed - what's the correct ticket status here?

Geoff Nichols (JIRA)

unread,
Aug 28, 2019, 8:43:02 PM8/28/19
to puppe...@googlegroups.com
Geoff Nichols updated an issue
Change By: Geoff Nichols
Sprint: Skeletor Kanban 20190619, Skeletor Kanban 20190703, Skeletor Kanban 20190717, Skeletor Kanban 20190731, Skeletor Kanban 20190814, Skeletor Kanban 20190828 , Skeletor Kanban 20190911

Geoff Nichols (JIRA)

unread,
Aug 29, 2019, 8:18:03 PM8/29/19
to puppe...@googlegroups.com
Geoff Nichols updated an issue
Change By: Geoff Nichols
Labels: closed-github-move docs_reviewed run_command

Geoff Nichols (JIRA)

unread,
Aug 29, 2019, 8:18:04 PM8/29/19
to puppe...@googlegroups.com

Geoff Nichols (JIRA)

unread,
Aug 29, 2019, 8:21:02 PM8/29/19
to puppe...@googlegroups.com
Geoff Nichols updated an issue
Change By: Geoff Nichols
Fix Version/s: BOLT 1.29.0
Reply all
Reply to author
Forward
0 new messages