How to share single large file using salt ?

[Ajinkya Joshi]

Hello

I want to copy one large installation tar file (file size is around 1.3 GB) on all minions .

How to share this large file using salt ??

[Charles Baker]

You might start with file.copy: 

http://docs.saltstack.com/en/latest/ref/states/all/salt.states.file.html#salt.states.file.copy

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Charles H. Baker
864.990.1297

Knowing is not enough; we must apply. Willing is not enough; we must do. Bruce Lee

[Stephen Spencer]

+1 for file.copy

Keep a close eye on your salt-master when you're testing this.  Depending on the speed of your network, RAM allocated to the salt-master and the speed of your storage device(s) you might find it beneficial to put files that large on a host running nginx or some such and just use salt  --async to call curl/fetch/wget for transferring it to the minions.  Offloading large transfers onto a web server will make sure your salt-master's resources aren't overly dedicated to distribution of a single file and the --async option will provide a JID string that can be used to query the jobs cache to check on the status of the transfer.

salt-run jobs.print_job <JID string>

-S

You know, I used to think it was awful that life was so unfair. Then I thought, wouldn't it be much worse if life were fair, and all the terrible things that happen to us come because we actually deserve them? So, now I take great comfort in the general hostility and unfairness of the universe.

[Markus Kramer]

I am customizing peer-to-peer (BitTorrent) for that purpose
Salt will provide metadata (*.torrent). and in place of a tracker, seed adresses.
Further reading at https://github.com/markuskramerIgitt/Magni

[Stephen Spencer]

Nice!  You may well have this covered already, but there are some good blog posts out there regarding optimization of bittorrent on a LAN.  There are several protocol optimization features designed to make life good over the interwebs but are somewhat problematic on high-speed low-latency networks.  I seem to remember that those articles came by way of the teams at (maybe) Facebook, Linkedin and Twitter.

-S

--

[Markus Kramer]

Hi Spencer,
If you can remember where you have found these blogs I would be glad  to read them.
I am aware that i have to optimize, that this is a complex task, an I have not found interweb optimizaton report.

Twitter and Facebook are p2peering only within data centres.
Twitter (which called its system "murder") uses exactly one "upload" peer, so it is a daisy chain of peers (see image).

I need to distribute to (mobile) user clients on a LAN/intranet (including Wifi and VPN), so the Twitter/Facebook setup is not exactly my use case.

Best regards,
Markus

[Stephen Spencer]

Unfortunately not.  I think I found them, read them and then let them pass into internet archivery a year-and-a-half ago.

-S

[Markus Kramer]

Ah, that place :-)

Anyhow, I will be finetuning earliest in March and then make a little note here.