Pros/Cons of SIMP RSYNC vs Puppet File function to sync large files/folders

[Brian S]

Hello-

I have a question concerning why it might be better to use the SIMP RSYNC module to sync files/folders vs the built-in puppet function File.  For SA I need to sync multiple large tar.gz files, that are fairly static, but also config files that are static across the puppet controlled nodes.  I was thinking of using the puppet file function for the config files and rsync for the large tar.gz files, but figured it might be best to keep them aligned if possible.

I assume the default use of stunnel with RSYNC leads to a better security posture for syncing files than the native File function built within puppet?  But is there anything else besides this as to why I should use one over the other?  Are there performance benefits between the two?

Thanks-

Brian

[Trevor Vaughan]

Hi Brian,

The rsync native type was created not for better security but because the Puppet native file resource is abysmal is managing large numbers of files and/or large files.

The large number of files issue comes down to the resource load on your system due to performing a checksum on, and creating a resource in the graph for, each file that will be transferred.

The large file issue is a combination of the checksum issue (takes a long time and is resource intensive) and the way Puppet transfers files. Rsync transfers in small differential chunks while Puppet always serializes (huge bloat) and ships the entire file (large network load and serialization/deserialization effort).

So, that's the back story and why we use it for our kickstart images, virus definitions, etc...

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "SIMP Q&A Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/b3f2284e-79ce-4eae-ae43-72e6027f7c99%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

[Brian S]

Awesome this is exactly the backstory I was looking for!

-Brian

On Tuesday, February 14, 2017 at 11:09:57 AM UTC-5, Trevor Vaughan wrote:

Hi Brian,

The rsync native type was created not for better security but because the Puppet native file resource is abysmal is managing large numbers of files and/or large files.

The large number of files issue comes down to the resource load on your system due to performing a checksum on, and creating a resource in the graph for, each file that will be transferred.

The large file issue is a combination of the checksum issue (takes a long time and is resource intensive) and the way Puppet transfers files. Rsync transfers in small differential chunks while Puppet always serializes (huge bloat) and ships the entire file (large network load and serialization/deserialization effort).

So, that's the back story and why we use it for our kickstart images, virus definitions, etc...

Thanks,

Trevor

On Tue, Feb 14, 2017 at 9:59 AM, Brian S <brians...@gmail.com> wrote:

Hello-

I have a question concerning why it might be better to use the SIMP RSYNC module to sync files/folders vs the built-in puppet function File.  For SA I need to sync multiple large tar.gz files, that are fairly static, but also config files that are static across the puppet controlled nodes.  I was thinking of using the puppet file function for the config files and rsync for the large tar.gz files, but figured it might be best to keep them aligned if possible.

I assume the default use of stunnel with RSYNC leads to a better security posture for syncing files than the native File function built within puppet?  But is there anything else besides this as to why I should use one over the other?  Are there performance benefits between the two?

Thanks-

Brian

--
You received this message because you are subscribed to the Google Groups "SIMP Q&A Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to simp+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/b3f2284e-79ce-4eae-ae43-72e6027f7c99%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

Good to hear it! We're adding it to the Docs now.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/378f4509-a98a-45ab-8127-310b129b3367%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.