Asynchronous file transfer through puppet ????

[sanjiv singh]

   
Hi All,
         I  have configured puppet with passenger with LDAP as configuration store.. Using puppet for deploying application different machines. Application deployment consist of pushing numerous templates, flat files(JARs,ZIPs,WARs etc) and various system/application level configuration changes.

         I have written puppet manifest accordingly.Here is sample for pushing two ZIPs.
   #--------------------------------------------------------
    file { "/home/user/application1.zip":
    group => 'app',
    source => "puppet://puppet/${environment}/application1.zip",
    }
    file { "/home/user/application2.zip":
    group => 'app',
    source => "puppet://puppet/${environment}/application2.zip",
    }
    #-------------------------------------------------------
    what am i questioning here is that puppet daemon execution is single threaded.It processes each resource one-by-one( based on dependencies) .In this case FILE resource. It transfers file Zips one-by-one irrespective of dependency.

    Is there any way, we can configure puppet/passenger so that it can transfer files asynchronously. If its does not have dependency,puppet daemon spawning thread for asynchronous file transfer ?

    In my environment, I need to transfer number of files. More files means more delay in puppet daemon execution.

    I am trying to achieve some performance gain over application deployment.

Regards,
Sanjiv Singh
Software Engineer (iLabs)
Impetus Infotech (India) Pvt. Ltd.

  
   

    

[jcbollinger]

On Apr 11, 4:05 am, sanjiv singh <sanjiv.is...@gmail.com> wrote:
> Hi All,

>          I  have configured *puppet with passenger with LDAP as
> configuration store*.. Using puppet for deploying application different

> machines. Application deployment consist of pushing numerous templates,
> flat files(JARs,ZIPs,WARs etc) and various system/application level
> configuration changes.
>
>          I have written puppet manifest accordingly.Here is sample for
> pushing two ZIPs.
>    #--------------------------------------------------------
>     file { "/home/user/application1.zip":
>     group => 'app',
>     source => "puppet://puppet/${environment}/application1.zip",
>     }
>     file { "/home/user/application2.zip":
>     group => 'app',
>     source => "puppet://puppet/${environment}/application2.zip",
>     }
>     #-------------------------------------------------------
>     what am i questioning here is that puppet daemon execution is single
> threaded.It processes each resource one-by-one( based on dependencies) .In
> this case FILE resource. It transfers file Zips one-by-one irrespective of
> dependency.
>
>     Is there any way, we can configure puppet/passenger so that it can
> transfer files asynchronously. If its does not have dependency,puppet
> daemon spawning thread for asynchronous file transfer ?
>
>     In my environment, I need to transfer number of files. More files means
> more delay in puppet daemon execution.
>
>     I am trying to achieve some performance gain over application
> deployment.

For application deployment, it is usually much better to build
packages suitable for the target's native packaging system (RPMs, for
example), put them in a local repository, and use Package resources to
install them. This is likely to perform much better than will using
File resources to transfer many files.

In any event, unless you're talking about large numbers of small
files, you are probably saturating your bandwidth already. Parallel
file transfer will not help you in that case, and it might even
introduce enough additional I/O overhead to slow you down overall. In
any case, it would likely reduce the maximum number of clients each
master could support.

If you don't want to build packages, then a better bet for speeding up
your runs is to tweak the 'checksum' property and maybe the 'backup'
property of your File resources. For the checksum, 'md5lite' should
be somewhat faster than the default ('md5'), and 'mtime' should be
substantially faster. Setting the 'backup' parameter to 'false'
should also speed things a bit. Note, however, that changing these
settings from their defaults gains speed by reducing integrity
protection.


John