Jira (BOLT-1401) Add 'upload_file' endpoint to bolt-server

0 views
Skip to first unread message

Casey Williams (JIRA)

unread,
Jun 17, 2019, 3:45:03 PM6/17/19
to puppe...@googlegroups.com
Casey Williams created an issue
 
Puppet Task Runner / Task BOLT-1401
Add 'upload_file' endpoint to bolt-server
Issue Type: Task Task
Assignee: Unassigned
Created: 2019/06/17 12:44 PM
Priority: Normal Normal
Reporter: Casey Williams

Background

When told to run a task on some nodes, a PE master typically contacts the nodes over the PCP protocol. It sends a formatted request over PCP to the pxp-agent service running on the nodes, directing them to run the task locally. But if the nodes don't have a pxp-agent service running, the PE master must contact them via SSH or WinRM instead. For these cases, the PE master runs a "pe-bolt-server" service, which is a sinatra application that waits for post requests to /ssh/run_task or /winrm/run_task, then runs the task via an instance of the bolt executor, just like bolt does when you run a task via the CLI. Basically, it's a thin REST API wrapper around normal bolt "task run" operations that the PE master can use when there's no PXP agent to talk to over PCP.

In PE Kearney, "task run" is the only supported action over PCP, so it follows that the only endpoints in the bolt-server REST API are for "task run". For other typical bolt actions ("command run", "file upload", "script run"), PCP operations are handled by wrapping any non-task action in an ephemeral task in order to use the "task run" endpoints.

PCP-868 and ORCH-2305 describe process of enabling "upload file" without a task wrapper over PCP on the agent and server. This ticket describes the new endpoint on bolt-server that will support "upload file" when there is no PCP transport available.

Requirements

Changes will be made to the bolt server app:

  • The app follows the json schema specification. Add a description of the JSON schema for data passed to the new endpoints here. This schema should match the one defined for the command action in PCP-868.
  • Add new POST endpoints in the transport sinatra app here to support uploading a file via SSH and WinRM, using the new JSON schema.

Testing

  • Write RSpec tests for the changes to bolt here.
  • Acceptance/integration tests are also maintained from the orchestrator side: add new integration test(s) for the interaction of the orchestrator and bolt-server while uploading a file here.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Casey Williams (JIRA)

unread,
Jun 17, 2019, 3:52:02 PM6/17/19
to puppe...@googlegroups.com
Casey Williams updated an issue
Change By: Casey Williams
*Background*


When told to run a task on some nodes, a PE master typically contacts the nodes over the PCP protocol. It sends a formatted request over PCP to the pxp-agent service running on the nodes, directing them to run the task locally. But if the nodes don't have a pxp-agent service running, the PE master must contact them via SSH or WinRM instead. For these cases, the PE master runs a "pe-bolt-server" service, which is a sinatra application that waits for post requests to /ssh/run_task or /winrm/run_task, then runs the task via an instance of the bolt executor, just like bolt does when you run a task via the CLI. Basically, it's a thin REST API wrapper around normal bolt "task run" operations that the PE master can use when there's no PXP agent to talk to over PCP.

In PE Kearney, "task run" is the only supported action over PCP, so it follows that the only endpoints in the bolt-server REST API are for "task run". For other typical bolt actions ("command run", "file upload", "script run"), PCP operations are handled by wrapping any non-task action in an ephemeral task in order to use the "task run" endpoints.

PCP-868 and ORCH-2305 describe process of enabling "upload file" without a task wrapper over PCP on the agent and server. This ticket describes the new endpoint on bolt-server that will support "upload file" when there is no PCP transport available.

*Requirements*

Changes will be made to the [bolt server app|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server]:

* The app follows the [json schema specification|https://json-schema.org/specification.html]. Add a description of the JSON schema for data passed to the new endpoints [here|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server/schemas]. This schema should match the one defined for the
command "file upload" action in PCP-868.

* Add new POST endpoints in the transport sinatra app [here|https://github.com/puppetlabs/bolt/blob/master/lib/bolt_server/transport_app.rb] to support uploading a file via SSH and WinRM, using the new JSON schema.

*Testing*

* Write RSpec tests for the changes to bolt [here|https://github.com/puppetlabs/bolt/tree/master/spec/bolt_server].
* Acceptance/integration tests are also maintained from the orchestrator side: add new integration test(s) for the interaction of the orchestrator and bolt-server while uploading a file [here|https://github.com/puppetlabs/orchestrator/blob/lovejoy/test/puppetlabs/orchestrator/integration/bolt_server.clj].

Casey Williams (JIRA)

unread,
Jun 17, 2019, 4:21:03 PM6/17/19
to puppe...@googlegroups.com
Casey Williams updated an issue
*Background*

When told to run a task on some nodes, a PE master typically contacts the nodes over the PCP protocol. It sends a formatted request over PCP to the pxp-agent service running on the nodes, directing them to run the task locally. But if the nodes don't have a pxp-agent service running, the PE master must contact them via SSH or WinRM instead. For these cases, the PE master runs a "pe-bolt-server" service, which is a sinatra application that waits for post requests to /ssh/run_task or /winrm/run_task, then runs the task via an instance of the bolt executor, just like bolt does when you run a task via the CLI. Basically, it's a thin REST API wrapper around normal bolt "task run" operations that the PE master can use when there's no PXP agent to talk to over PCP.

In PE Kearney, "task run" is the only supported action over PCP, so it follows that the only endpoints in the bolt-server REST API are for "task run". For other typical bolt actions ("command run", "file upload", "script run"), PCP operations are handled by wrapping any non-task action in an ephemeral task in order to use the "task run" endpoints.

PCP-868 and ORCH-2305 describe process of enabling "upload file" without a task wrapper over PCP on the agent and server. This ticket describes the new endpoint on bolt-server that will support "upload file" when there is no PCP transport available.

*Requirements*

Changes will be made to the [bolt server app|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server]:

* The app follows the [json schema specification|https://json-schema.org/specification.html]. Add a description of the JSON schema for data passed to the new endpoints [here|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server/schemas]. This schema should match the one defined for the "file upload" action in PCP-868.


* Add new POST endpoints in the transport sinatra app [here|https://github.com/puppetlabs/bolt/blob/master/lib/bolt_server/transport_app.rb] to support uploading a file via SSH and WinRM, using the new JSON schema.

*Testing*

* Write RSpec tests for the changes to bolt [here|https://github.com/puppetlabs/bolt/tree/master/spec/bolt_server].
* Acceptance/integration tests are also maintained from on the orchestrator side : add new integration test(s) exist for the interaction of the orchestrator and bolt-server while uploading a file this code [here|https://github.com/puppetlabs/orchestrator/blob/lovejoy/test/puppetlabs/orchestrator/integration/bolt_server.clj] , but updating them is *out of scope* for this ticket . *Do* run a quick manual test that the API endpoints work as expected on a running PE master, but making changes orchestrator to make use of the endpoints should be done as part of an ORCH ticket.

Casey Williams (JIRA)

unread,
Jun 17, 2019, 5:06:02 PM6/17/19
to puppe...@googlegroups.com
Casey Williams updated an issue
*Background*

When told to run a task on some nodes, a PE master typically contacts the nodes over the PCP protocol. It sends a formatted request over PCP to the pxp-agent service running on the nodes, directing them to run the task locally. But if the nodes don't have a pxp-agent service running, the PE master must contact them via SSH or WinRM instead. For these cases, the PE master runs a "pe-bolt-server" service, which is a sinatra application that waits for post requests to /ssh/run_task or /winrm/run_task, then runs the task via an instance of the bolt executor, just like bolt does when you run a task via the CLI. Basically, it's a thin REST API wrapper around normal bolt "task run" operations that the PE master can use when there's no PXP agent to talk to over PCP.

In PE Kearney, "task run" is the only supported action over PCP, so it follows that the only endpoints in the bolt-server REST API are for "task run". For other typical bolt actions ("command run", "file upload", "script run"), PCP operations are handled by wrapping any non-task action in an ephemeral task in order to use the "task run" endpoints.

PCP-868 and ORCH-2305 describe process of enabling "upload file" without a task wrapper over PCP on the agent and server. This ticket describes the new endpoint on bolt-server that will support "upload file" when there is no PCP transport available.

*Requirements*

Changes will be made to the [bolt server app|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server]:

* The app follows the [json schema specification|https://json-schema.org/specification.html]. Add a description of the JSON schema for data passed to the new endpoints [here|https://github.com/puppetlabs/bolt/tree/master/lib/bolt_server/schemas]. This schema should match the one defined for the "file upload" action in PCP-868.

* Add new POST endpoints in the transport sinatra app [here|https://github.com/puppetlabs/bolt/blob/master/lib/bolt_server/transport_app.rb] to support uploading a file via SSH and WinRM, using the new JSON schema.
* Document the new endpoints in the [developer-docs|https://github.com/puppetlabs/bolt/blob/master/developer-docs/].

*Testing*

* Write RSpec tests for the changes to bolt [here|https://github.com/puppetlabs/bolt/tree/master/spec/bolt_server].
* Acceptance/integration tests on the orchestrator side exist for this code [here|https://github.com/puppetlabs/orchestrator/blob/lovejoy/test/puppetlabs/orchestrator/integration/bolt_server.clj], but updating them is *out of scope* for this ticket. *Do* run a quick manual test that the API endpoints work as expected on a running PE master, but making changes orchestrator to make use of the endpoints should be done as part of an ORCH ticket.

Sean McDonald (JIRA)

unread,
Aug 8, 2019, 7:00:03 PM8/8/19
to puppe...@googlegroups.com
Sean McDonald updated an issue
Change By: Sean McDonald
Sprint: Skeletor Kanban 20190814

Branan Riley (JIRA)

unread,
Aug 14, 2019, 7:39:05 PM8/14/19
to puppe...@googlegroups.com
Branan Riley updated an issue
Change By: Branan Riley
Sprint: Skeletor Kanban 20190814 , Skeletor Kanban 20190828

Scott McClellan (JIRA)

unread,
Aug 26, 2019, 1:47:03 PM8/26/19
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages