Jira (BOLT-57) Download a file

3 views
Skip to first unread message

Cody Herriges (JIRA)

unread,
Oct 24, 2017, 4:26:02 PM10/24/17
to puppe...@googlegroups.com
Cody Herriges commented on New Feature BOLT-57
 
Re: Download a file

I'd think there is two different modes of operation a person would want to leverage; a situation where the downloading of the file is just an transitive process and one where I actually wish the file to remain once a plan is complete. In the first case I'd expect a file to either be downloaded to a temporary location with the return value of the function to be the path to the file or the file just literally base64 encoded into a variable. Storing gigabytes of data as a string is probably not the best idea but was likely the way I was going to work around not having this functionality for the time being. I'd also expect that in the first mode of operation, all files downloaded would be automatically cleaned up at the end of plan execution.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)
Atlassian logo

Henrik Lindberg (JIRA)

unread,
Oct 25, 2017, 9:03:03 AM10/25/17
to puppe...@googlegroups.com
Henrik Lindberg commented on New Feature BOLT-57
 
Re: Download a file

Auto cleanup sounds good. A user that wants to keep a file could always run a "copy file" local task to persiste the file past the end of the plan.

Alex Dreyer (JIRA)

unread,
Nov 14, 2017, 12:49:03 PM11/14/17
to puppe...@googlegroups.com
Alex Dreyer updated an issue
 
Puppet Task Runner / New Feature BOLT-57
Download a file
Change By: Alex Dreyer
From [~alex]:

{quote}
I've run into the opposite problem in trying to deploy forge as well. I need to build an artifact on a target and then copy it back to the controller to deploy elsewhere.
{quote}

This ticket is to add a download file
 from target nodes to the bolt controller  option to the CLI. It probably only makes sense to download from a single node to the controller.

See also the ability to upload files TASKS-51.
This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)
Atlassian logo

Alex Dreyer (JIRA)

unread,
Nov 14, 2017, 12:51:03 PM11/14/17
to puppe...@googlegroups.com
Alex Dreyer commented on New Feature BOLT-57
 
Re: Download a file

Can this be called "download" since it's the opposite of upload. Would it make more sense to change upload to copy_to and call this copy_from?

What is the syntax for the function?
What is the syntax for the

{bolt file download}

command?
How should it deal with multiple targets?

Henrik Lindberg (JIRA)

unread,
Nov 14, 2017, 1:17:03 PM11/14/17
to puppe...@googlegroups.com
Henrik Lindberg commented on New Feature BOLT-57
 
Re: Download a file

How about mimicking rcp and have a remote_copy(source, target) where the side being remote must be given as a Target and the side that is a local file is a path? Proposing this because people are probably familiar with rcp, and cp and that this gets rid of the somewhat ambiguous "to"/"from" or "up"/"down".

Here is an example. Note that we encode the path to the file as an option in the Target.

To copy from the remote system to the local bolt controller:

remote_copy(Target('kermit.com', path => '/tmp/buildresult.zip'), "${resultdir}/buildresult.zip")

To copy the other way:

remote_copy("${resultdir}/buildresult.zip", Target('kermit.com', path => '/tmp/buildresult.zip'))

Henrik Lindberg (JIRA)

unread,
Nov 14, 2017, 1:40:04 PM11/14/17
to puppe...@googlegroups.com
 
Re: Download a file

To support multiple operations source and target could be arrays - such that:

  • an array in source or target must have the same kind - i.e. Array[Target] or Array[String].
  • if target is Array[Target] (i.e. to remotes):
    • a (non array) single source is copied to each remote host at the specified path per target
    • an array source - is an error (as there is no way to specify different filenames per different targets)
  • if source is Array[Target] (i.e. from remotes)
    • The target must be a (non array) directory and each copied file is stored in a directory <givendir>/<targethost> with the remote (leaf) filename
    • An array target is an error

Example - copy 'foo.txt' to two hosts:

$targets = [
  Target('kermit.com', path => '/tmp/foo.txt'),
  Target('gonzo.com', path => '/tmp/foo.txt')
]
remote_copy('/tmp/foo.txt', $targets)

Get files from two hosts and store in '/tmp/collected':

$targets = [
  Target('kermit.com', path => '/tmp/foo.txt'),
  Target('gonzo.com', path => '/tmp/foo.txt')
]
remote_copy($targets,  '/tmp/collected')

This would result in the files /tmp/collected/kermit.com/foo.txt, and /tmp/collected/kermit.com/foo.txt being available on the bolt controller node.

Cody Herriges (JIRA)

unread,
Nov 14, 2017, 2:43:04 PM11/14/17
to puppe...@googlegroups.com
Cody Herriges commented on New Feature BOLT-57
 
Re: Download a file

I don't take any issue with the idea of following the "cp" semantics, think it is a pretty solid idea. I am a bit uneasy about associating anything with the "r*" family of unix commands though, even is the underlying implementation doesn't use them. Means we're constantly having to remind people that this is actually SSH and more like "scp" than "rcp".

What's the CLI equivalent of the suggested Puppet code?

Peter Souter (JIRA)

unread,
Nov 14, 2017, 3:03:03 PM11/14/17
to puppe...@googlegroups.com
Peter Souter commented on New Feature BOLT-57
 
Re: Download a file

If we are going to go for an scp like semantic, it would be good if we could do things like wild-card matching, as that would help out with https://tickets.puppetlabs.com/browse/BOLT-191

David Kramer (JIRA)

unread,
Nov 15, 2017, 12:05:09 PM11/15/17
to puppe...@googlegroups.com
David Kramer updated an issue
Change By: David Kramer
Sprint: Tasks ELP (1.0) Bolt Ready for Grooming

Henrik Lindberg (JIRA)

unread,
Nov 15, 2017, 2:01:04 PM11/15/17
to puppe...@googlegroups.com
Henrik Lindberg commented on New Feature BOLT-57
 
Re: Download a file

Using scp semantics is certainly fine - it, in contrast to rcp also allows remote to remote copying, if we want that, my proposal/idea above needs to be somewhat modified.

David Kramer (JIRA)

unread,
Nov 16, 2017, 4:47:05 PM11/16/17
to puppe...@googlegroups.com
David Kramer updated an issue
Change By: David Kramer
Sprint: Bolt  Ready for Grooming  One Point Oh

Alex Dreyer (JIRA)

unread,
Dec 20, 2017, 12:30:03 PM12/20/17
to puppe...@googlegroups.com
Alex Dreyer updated an issue
Change By: Alex Dreyer
Sprint: Bolt  One Point Oh  Irving

David Kramer (JIRA)

unread,
Jan 23, 2018, 12:23:03 PM1/23/18
to puppe...@googlegroups.com
David Kramer updated an issue
Change By: David Kramer
Sprint: Bolt Irving Discover/Design

Alex Dreyer (JIRA)

unread,
Feb 27, 2018, 6:46:03 PM2/27/18
to puppe...@googlegroups.com
Alex Dreyer updated an issue
Change By: Alex Dreyer
Sprint: Discover/Design
This message was sent by Atlassian JIRA (v7.5.1#75006-sha1:7df2574)
Atlassian logo

Michael Smith (JIRA)

unread,
Feb 27, 2018, 6:48:03 PM2/27/18
to puppe...@googlegroups.com
Michael Smith assigned an issue to Michael Smith
Change By: Michael Smith
Assignee: Michael Smith

Michael Smith (JIRA)

unread,
Mar 2, 2018, 6:19:03 PM3/2/18
to puppe...@googlegroups.com
Michael Smith commented on New Feature BOLT-57
 
Re: Download a file

I like remote_copy as a flexible interface. Underneath, the Transport API would still need to operate with upload/download semantics as not all transports can inter-operate. If you're transferring between things that support scp-semantics (like SSH and SSH) we could have an override implementation for remote_copy.

To support a remote_copy on the CLI, we would want to have something similar to scp's syntax for declaring local and remote files. URIs already have a path component that we're not using, which would make sense here. So you would have commands like

bolt file copy local_file.txt ssh://user:password@pluto:22/tmp/local_file.txt

There are some quirks to the above that seem sensible:

  • an unadorned file is a "local" file, implicitly mapping to a URI like local:///local_file.txt
  • I chose file copy because that better fits our model of object action. That suggests we should use file_copy for the function rather than remote_copy.
  • it should otherwise follow the semantics Henrik outlined: Array[Target] -> Target, Target -> Array[Target], Target -> Target are all supported but not array to array.

A with_path helper on Target would be nice so you can pass $target.with_path('/tmp/foo.txt'), or

get_targets($nodes).map |$t| { $t.with_path('/tmp/foo.txt') }

I think adding directories as a source in BOLT-191 would fit onto this cleanly (copy the directory recursively instead of a single file, require that the destination not exist).

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Michael Smith (JIRA)

unread,
Mar 2, 2018, 6:28:03 PM3/2/18
to puppe...@googlegroups.com
Michael Smith commented on New Feature BOLT-57
 
Re: Download a file

Ah, rereading a little I see Henrik wasn't proposing scp-semantics, just a new way of formulating upload and download as a single function with consistent semantics.

Michael Smith (JIRA)

unread,
Mar 2, 2018, 7:09:02 PM3/2/18
to puppe...@googlegroups.com
Michael Smith commented on New Feature BOLT-57
 
Re: Download a file

One thing the CLI format I suggested makes weird is working with an inventory. I would suggest we have a separate format like group:/tmp/local_file that we use for groups, where the path is added to each target in the group. Group names are constrained enough that group/tmp/local_file would be unambiguous, but it just looks weird.

Henrik Lindberg (JIRA)

unread,
Mar 4, 2018, 5:57:03 AM3/4/18
to puppe...@googlegroups.com
Henrik Lindberg commented on New Feature BOLT-57
 
Re: Download a file

Michael Smith What I proposed earlier was for rcp semantics (no copy from one remote to another). I like the idea of file_copy function with scp semantics, since you can then copy a file from/to any - including a copy on the local host between two paths. If scp can be used to perform an operation it could be used otherwise down/up-load needs to be performed.

What I proposed was not "just copy" it also included semantics that files from remotes to local where copied to a location per target so that they do not step on each other (collection of "same file" from multiple remotes). Maybe that is a special case deserving its own function like collect_files which is implemented in terms of copy_file where copy_file just copies to/from as given by targets/paths? (Same thing is naturally achievable by passing options to copy_file).

Michael Smith (JIRA)

unread,
Mar 9, 2018, 5:08:04 PM3/9/18
to puppe...@googlegroups.com
Michael Smith assigned an issue to Unassigned
 
Puppet Task Runner / New Feature BOLT-57
Download a file
Change By: Michael Smith
Assignee: Michael Smith

Randell Pelak (JIRA)

unread,
Jun 28, 2018, 6:02:52 PM6/28/18
to puppe...@googlegroups.com
Randell Pelak commented on New Feature BOLT-57
 
Re: Download a file

was asked to put my use case in here.
For ref arch setup, we expect to be running on a neutral box and setting up a ref arch. But we need to allow it to be done in stages for various reasons. Once we have uploaded the pe.conf to the master in the early stages, if the user stops, and then later runs the later stages, we need to get the pe.conf from the master so that we are sure we are using the latest copy. We may have to modify it and send it back as well, so we need the full contents of the file to avoid losing any helpful comments that anyone/anything may have added.

Charlie Sharpsteen (JIRA)

unread,
Jun 28, 2018, 6:39:04 PM6/28/18
to puppe...@googlegroups.com
Charlie Sharpsteen commented on New Feature BOLT-57
 
Re: Download a file

Randell Pelak For a small text file like pe.conf you could probably execute a task that just echoes the file content to stdout. Then swap it for a download function when that gets created.

Randell Pelak (JIRA)

unread,
Jun 28, 2018, 6:42:03 PM6/28/18
to puppe...@googlegroups.com
Randell Pelak commented on New Feature BOLT-57
 
Re: Download a file

Charlie Sharpsteen yeah, that would be plan A. But if we have a fancy download feature. We aren't at the point we are implementing that part just yet. Our first rev will probably just assume it hasn't changed since we put it there. But it isn't too far out now that we will want to verify that assumption...

Trevor Vaughan (JIRA)

unread,
Jul 16, 2018, 2:08:03 PM7/16/18
to puppe...@googlegroups.com
Trevor Vaughan commented on New Feature BOLT-57
 
Re: Download a file

Here's my use case. Feel free to steal the code with attribution.

https://github.com/simp/rubygem-simp-beaker-helpers/blob/master/lib/simp/beaker_helpers.rb#L31-L71

Case 1: I'm pointing at something in docker on my local host -> Use docker cp because fast
Case 2: I have rsync on both ends -> Use it because fast
Otherwise: Fall back to ssh

Alex Dreyer (JIRA)

unread,
Jan 9, 2019, 5:15:04 PM1/9/19
to puppe...@googlegroups.com
Alex Dreyer commented on New Feature BOLT-57
 
Re: Download a file

Another user has requested "file copy" rather then just file_download.

Joshua Partlow (JIRA)

unread,
Jun 12, 2019, 6:00:04 PM6/12/19
to puppe...@googlegroups.com
Joshua Partlow commented on New Feature BOLT-57
 
Re: Download a file

Ran into this ticket as well; need to synchronizing /etc/puppetlabs/enterprise between infrastructure nodes in a bolt plan run from a separate controller node.

Reply all
Reply to author
Forward
0 new messages