Copy files between 2 minions

[Sebastien Douche]

Hi here.
nowadays we use the scp (or rsync over ssh) command to copy files but
the idea is to use ZeroMQ in Salt. How to add this functionality? Do
you think it's a crazy idea?

The goal is to manage all infrastructure tasks with Salt.


Thanks in advance for your help.


PS: kudos for the documentation, it's a pleasure to learn Salt with it.


--
Sebastien Douche <sdo...@gmail.com>
Twitter: @sdouche / G+: +sdouche

[Clint Savage]

Sebastien,

It really sounds like this documentation isn't being found. I wonder
if there is a better set of terms to help. However, here are a couple
ways to do what you are suggesting.

Have a look at http://docs.saltstack.org/en/latest/ref/modules/all/salt.modules.publish.html.
You may be interested in
http://docs.saltstack.org/en/latest/topics/event/index.html

I hope that helps.

Cheers,

Clint

[Sebastien Douche]

On Fri, Aug 24, 2012 at 11:13 AM, Clint Savage <her...@gmail.com> wrote:
> It really sounds like this documentation isn't being found.

Uh?!

> Have a look at http://docs.saltstack.org/en/latest/ref/modules/all/salt.modules.publish.html.

I don't understand how the publish module can help me. What module to
use to copy files? Can you write a working example?

[Joseph Hall]

Unless somebody has a correction to my understanding, I have bad news for you.

ZeroMQ is set up as a communication layer between master and minion.
Unfortunately in this regard, it is not a communication layer between
minion and minion; they would have to relay their communication
through the master.

Were this possible, extensive use of it to copy files would of course
create a high load on the master, and would probably bottleneck pretty
hard on the master, very quickly.

However, while the master is allowed to push files to minions, it is
not currently possible to copy a file from a minion back to the
master. I think it would be entirely possible to write a module that
could do this (and a salt-runner to help copy it back to a different
minion), but there are pretty serious security implications with
allowing a minion to push files (or any arbitrary data) to the master.

Short answer: salt may be able to initiate the scp with a passwordless
key (I do this routinely between my minions), but it should not be
used as the actual transport mechanism.

On Fri, Aug 24, 2012 at 2:53 AM, Sebastien Douche <sdo...@gmail.com> wrote:

--
"In order to create, you have to have the willingness, the desire to
be challenged, to be learning." -- Ferran Adria (speaking at Harvard,
2011)

[Thomas S Hatch]

I have been debating on a number of ways to add this capability. There are a lot of questions about how it can be done in a clean way without there being a security issue, I also have been debating about making it a system like a distributed data distribution system, I would like to make Salt able to sole the problem of distributing files in an efficient way to all minions without having 10,000 minions ask for a full, large fine from the master alone.

[David Boucha]

On Aug 25, 2012 10:07 AM, "Thomas S Hatch" <that...@gmail.com> wrote:
>
> I have been debating on a number of ways to add this capability. There are a lot of questions about how it can be done in a clean way without there being a security issue, I also have been debating about making it a system like a distributed data distribution system, I would like to make Salt able to sole the problem of distributing files in an efficient way to all minions without having 10,000 minions ask for a full, large fine from the master alone.
>
>

Hmm, as far as distributing a large file to 10,000+ minions, maybe we could somehow have the master be a bittorrent tracker. we wouldn't want to make everyone install a bittorrent client on every minion, but it would be an interesting option for those who might find that useful

[Thomas S Hatch]

On Sat, Aug 25, 2012 at 11:04 AM, David Boucha <bou...@gmail.com> wrote:

On Aug 25, 2012 10:07 AM, "Thomas S Hatch" <that...@gmail.com> wrote:
>
> I have been debating on a number of ways to add this capability. There are a lot of questions about how it can be done in a clean way without there being a security issue, I also have been debating about making it a system like a distributed data distribution system, I would like to make Salt able to sole the problem of distributing files in an efficient way to all minions without having 10,000 minions ask for a full, large fine from the master alone.
>
>

Hmm, as far as distributing a large file to 10,000+ minions, maybe we could somehow have the master be a bittorrent tracker. we wouldn't want to make everyone install a bittorrent client on every minion, but it would be an interesting option for those who might find that useful

This is a little ways out, but as you may know I have spent some time with distributed file systems. I want to see if I can make a stateless zeromq distributed file deployment system that can be directy integrated into the minions. But like I said, it is a little ways down the road still :)

[Les Mikesell]

That sounds extreme enough to be impossible - or nearly so. How
about something simple like a router/vpn-like facility that could pass
rsync connections through without knowing addressing ahead of time and
perhaps have access control based on groups? Or maybe just push
everything through the master, but with control of groups and some
related tmp space delegated to specific minions. That is, make it
easy for different people to manage different sets of minions,
generally by having their own minion that can act as a staging box for
deployments.

--
Les Mikesell
lesmi...@gmail.com

[Thomas S Hatch]

Yes, there are a lot of possible solutions. Right now there are a few blockers here, so I don't think that we will get to the solution for a little while, and hopefully I have a bigger perspective on everything by then

[Sebastien Douche]

On Fri, Aug 24, 2012 at 3:44 PM, Joseph Hall <perl...@gmail.com> wrote:
> Unless somebody has a correction to my understanding, I have bad news for you.
>
> ZeroMQ is set up as a communication layer between master and minion.
> Unfortunately in this regard, it is not a communication layer between
> minion and minion; they would have to relay their communication
> through the master.

Same understanding here. Thanks Joseph.

[Sebastien Douche]

On Mon, Aug 27, 2012 at 5:04 PM, Thomas S Hatch <that...@gmail.com> wrote:

Hi Thomas

Oh, a teaser ;).

[Thomas S Hatch]

Heh, there is a lot going on right now, so I am a little behind on development, but catching up!

[Sebastien Douche]

On Tue, Aug 28, 2012 at 9:46 PM, Thomas S Hatch <that...@gmail.com> wrote:
> Heh, there is a lot going on right now, so I am a little behind on
> development, but catching up!

Don't worry, it's a joke :).

[Robert Parker]

Has this functionality been addressed yet?  I'd like the ability to force a recursive file/directory copy between two minions (over ssh or 0mq, but encrypted in any case) without first manually copying it to the master's file root.

[Seth House]

Take a look at the newly added minionfs:

http://docs.saltstack.com/topics/tutorials/minionfs.html

> --
> You received this message because you are subscribed to the Google Groups
> "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to salt-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[J C Lawrence]

This seems like both a Bad Idea and a slippery slope.  Salt isn't, or shouldn't be, a core tool for distributing large file volumes.  Salt is, or should be, a Command and Control tool rather than a core file distribution system.  In-line with that Command & Control focus, Salt would tell other systems to go get large filesets from yet other systems, not to actually do the shovelling of file bits itself over its own systems and transports...

Methinks rsync is the Right Tool for your problem, and having salt orchestrate rsync invocations seems an Excellent Thing.

-- JCL seems to do rather a lot of SSH key shuffling and rsync invocation with Salt

--

[Les Mikesell]

On Mon, Mar 17, 2014 at 12:54 PM, J C Lawrence <cl...@kanga.nu> wrote:
> This seems like both a Bad Idea and a slippery slope. Salt isn't, or
> shouldn't be, a core tool for distributing large file volumes. Salt is, or
> should be, a Command and Control tool rather than a core file distribution
> system. In-line with that Command & Control focus, Salt would tell other
> systems to go get large filesets from yet other systems, not to actually do
> the shovelling of file bits itself over its own systems and transports...

Have to disagree... Establishing a secure communications channel
where the minions only need the address of the master is much of the
beauty of salt. If you can't use that and have to set up a different
topology to actually do anything, what was the point? You might as
well just ssh everything in the first place.

> Methinks rsync is the Right Tool for your problem, and having salt
> orchestrate rsync invocations seems an Excellent Thing.

I do agree that rsync does things right. Or maybe svn update from a
central repository. But, then you need a generic way to connect them
with the the salt transport as a multiplexor.

--
Les Mikesell
lesmi...@gmail.com