Module for lsyncd - multiple syncs to be configured, but only one config file

[Ugo Bellavance]

Hi,

I wrote my first puppet module and it is for lsyncd.  I know that there is already a module for that but I think it was written to manage lsyncd and csync2 syncs and I don't really understand everything it it so I can't really use it or learn from it.  I have been able to create a simple module, below is the code from init.pp (I know, I should modularize it so that init.pp calls ::install and ::config, but I'll do that once the module works as I want).  It is actually working, but only allows for one sync to be configured.  I'd like it to be able to configure more than one sync, but since all the configs for lsyncd is in one file, I wouldn't know how to do it unless I add some variables like $method, $source2, $dest2, $desthost2, $append2 and $method, $source3, $dest3, $desthost3, $append3, $method, $sourcex, $destx, $desthostx, $appendx. I think someone here would know how to do it.

Code for init.pp:

class lsyncd ($method, $source, $dest, $desthost, $append) {

package { 'lsyncd':

        ensure => installed,

    }

    file { 'lsyncd.conf':

        ensure  => present,

        path    => '/etc/lsyncd.conf',

        owner   => root,

        group   => root,

        mode    => '0644',

        content => template('lsyncd/lsyncd.conf.erb')

    }

    service { 'lsyncd':

        ensure      => running,

        enable      => true,

        hasrestart  => true,

        hasstatus   => true,

        require     => [Package['lsyncd'], File['lsyncd.conf']],

        subscribe   =>

            File[

                'lsyncd.conf'

            ],

    }

}

This is the code for the erb template (feel free to let me know if I you find errors):

settings {

   logfile    = "/var/log/lsyncd.log",

   statusFile = "/tmp/lsyncd.status",

   inotifyMode = "CloseWrite or Modify",

   delay       = "1"

}

sync {

   <%= @method %>,

   source     = "<%= source %>",

   host       = "<%= desthost %>",

   targetdir  = "<%= dest %>",

<% if @append -%>

   rsync      = {

      _extra = {"--append"}

   }

<% end -%>

}

Here is how I'd like to have my config file:

settings {

   logfile    = "/var/log/lsyncd.log",

   statusFile = "/tmp/lsyncd.status",

   inotifyMode = "CloseWrite or Modify",

   delay       = "1"

}

sync {

   default.rsyncssh,

   source     = "/var/www/test/src",

   host       = "www.example.com",

   targetdir  = "/var/www/test/dst",

   rsync      = {

      _extra = {"--append"}

   }

}

sync {

   default.rsyncssh,

   source     = "/var/www/",

   host       = "www.example.com",

   targetdir  = "/var/www/test/dst1",

   rsync      = {

      _extra = {"--append"}

   }

}

sync {

   default.rsyncssh,

   source     = "/var/log/",

   host       = "www.example.com",

   targetdir  = "/var/www/test/dst2",

   rsync      = {

      _extra = {"--append"}

   }

}

Thanks,

[Poil]

Hi,

You can use hiera (here a yaml)

sync:
  instance1:
    source: "toto"
    host: "titi
 instance2:
    source: "toto"
    host: "titi

And in your template juste loop on it
<% @sync.each_pair do |name, value| -%>
<%= name %>
   source     = "<%= value['source'] %>",
   host       = "<%= value['host'] %>"
<% end %>

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/a63e58f0-1da9-40ef-bb4c-b5e99f3b0365%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jakov Sosic]

On 05/15/2014 09:25 PM, Ugo Bellavance wrote:
> Hi,
>
> I wrote my first puppet module and it is for lsyncd. I know that there
> is already a module for that but I think it was written to manage lsyncd
> and csync2 syncs and I don't really understand everything it it so I
> can't really use it or learn from it. I have been able to create a
> simple module, below is the code from init.pp (I know, I should
> modularize it so that init.pp calls ::install and ::config, but I'll do
> that once the module works as I want). It is actually working, but only
> allows for one sync to be configured. I'd like it to be able to
> configure more than one sync, but since all the configs for lsyncd is in
> one file, I wouldn't know how to do it unless I add some variables
> like $method, $source2, $dest2, $desthost2, $append2 and $method,
> $source3, $dest3, $desthost3, $append3, $method, $sourcex, $destx,
> $desthostx, $appendx. I think someone here would know how to do it.
>

I would suggest you to take a look at concat module.

You can basically split your single config file into multiple parts, and
write a defined type for sync sections. For example:

# smb.conf
concat { 'lsyncd.conf':

path => '/etc/lsyncd.conf',
owner => root,
group => root,
mode => '0644',
}

::concat::fragment { 'lsyncd.conf:settings':
target => 'lsyncd.conf',
content => template('lsyncd/settings.conf.erb'),
order => '100',
}

And write your own defined type, for example:

define lsycnd::sync (
$method,
$source,
$host,
$targetdir,
) {

include ::lsyncd

::concat::fragment { "lsyncd.conf:sync:${title}":
target => 'lsyncd.conf',
content => template('lsyncd/sync.conf.erb'),
order => '200',
}

}

Now, your sync.conf.erb would look like:

[Ugo Bellavance]

On Thursday, May 15, 2014 5:45:57 PM UTC-4, Jakov Sosic wrote:

On 05/15/2014 09:25 PM, Ugo Bellavance wrote:
> Hi,
>
> I wrote my first puppet module and it is for lsyncd.  I know that there
> is already a module for that but I think it was written to manage lsyncd
> and csync2 syncs and I don't really understand everything it it so I
> can't really use it or learn from it.  I have been able to create a
> simple module, below is the code from init.pp (I know, I should
> modularize it so that init.pp calls ::install and ::config, but I'll do
> that once the module works as I want).  It is actually working, but only
> allows for one sync to be configured.  I'd like it to be able to
> configure more than one sync, but since all the configs for lsyncd is in
> one file, I wouldn't know how to do it unless I add some variables
> like $method, $source2, $dest2, $desthost2, $append2 and $method,
> $source3, $dest3, $desthost3, $append3, $method, $sourcex, $destx,
> $desthostx, $appendx. I think someone here would know how to do it.
>

I would suggest you to take a look at concat module.

I think I'll go that way as I don't know hiera much.

Thanks to you both for your advice, I'll try this soon.

Ugo 

[Ugo Bellavance]

On Thursday, May 15, 2014 5:45:57 PM UTC-4, Jakov Sosic wrote:

I worked on this tonight, but now I get an error when I try to apply it:

Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter dest at /etc/puppet/manifests/nodes/nodes.pp:29

If I comment out this variable, another one generate an Invalid parameter error...

I tried debugging it, but I think I need a little help.

Here is my init.pp:

class lsyncd {

  package { 'lsyncd':

        ensure => installed,

  }

  concat { 'lsyncd.conf':

    path    => '/etc/lsyncd.conf',

    owner   => root,

    group   => root,

    mode    => '0644',

  }

  ::concat::fragment { 'lsyncd.conf:settings':

    target  => 'lsyncd.conf',

    content => template('lsyncd/settings.conf.erb'),

    order   => '100',

  }

define lsyncd::sync (

  $method,

  $source,

  $dest,

  $desthost,)

  {

  include ::lsyncd

  ::concat::fragment { "lsyncd.conf:sync:${title}":

    target  => 'lsyncd.conf',

    content => template('lsyncd/sync.conf.erb'),

    order   => '200',

  }

    service { 'lsyncd':

        ensure      => running,

        enable      => true,

        hasrestart  => true,

        hasstatus   => true,

        require     => [Package['lsyncd'], File['lsyncd.conf']],

        subscribe   =>

            File[

                'lsyncd.conf'

            ],

    }

  }

}

My settings.conf.erb:

settings {

   logfile    = "/var/log/lsyncd.log",

   statusFile = "/tmp/lsyncd.status",

   inotifyMode = "CloseWrite or Modify",

   delay       = "1"

}

My sync.conf.erb:

sync {

    <%= @method %>,

    source     = "<%= source %>",

    host       = "<%= desthost %>",

    targetdir  = "<%= dest %>",

<% if @append -%>

    rsync      = {

       _extra = {"--append"}

    }

<% end -%>

}

Here is how I call it:

node "test.example.com" {

   class { "lsyncd":

       title    => ["src1"],

       method   => ["default.rsyncssh"],

       source   => ["/var/www/test/src"],

       dest     => ["/var/www/test/dst1"],

       desthost => ["www.example.com"],

       append   => ["true"]

   }

}

Thanks,

[José Luis Ledesma]

You have declarer lsyncd class without parameters

class lsyncd {

Regards,

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/c18e50c3-c978-48ba-9fb9-268d0652747d%40googlegroups.com.

[David Schmitt]

Hi,

On 16.05.2014 04:37, Ugo Bellavance wrote:
> I worked on this tonight, but now I get an error when I try to apply it:
>
> Could not retrieve catalog from remote server: Error 400 on SERVER:
> Invalid parameter dest at /etc/puppet/manifests/nodes/nodes.pp:29
>
> If I comment out this variable, another one generate an Invalid
> parameter error...
>
> I tried debugging it, but I think I need a little help.
>
> Here is my init.pp:
>
> class lsyncd {

[...]

> }
>
> define lsyncd::sync (
> $method,
> $source,
> $dest,
> $desthost,)
> {

[...]

> }
>
> Here is how I call it:
>
> node "test.example.com" {
> class { "lsyncd":
> title => ["src1"],
> method => ["default.rsyncssh"],
> source => ["/var/www/test/src"],
> dest => ["/var/www/test/dst1"],
> desthost => ["www.example.com"],
> append => ["true"]
> }
> }

You want to say it this way:

class { lsyncd: ; }
lsyncd::sync {
"src1":

method => "default.rsyncssh",
source => "/var/www/test/src",
dest => "/var/www/test/dst1",
desthost => "www.example.com",

append => "true";
}

For details see the language guide at

> http://docs.puppetlabs.com/puppet/latest/reference/lang_defined_types.html


Regards, David

[Poil]

Hi,

I was a little wrong yesterday, you can use hiera, but also you can use a hash in your nodes.pp


class lsync ($sync) {
}

$localsync = {
  instance1 => {
    'source' => 'toto',
    'host' => 'titi',
  },
  instance2 => {
    'source' => 'tutu',
    'host' => 'tata',
  },
}

class {'lsync':
  sync => $localsync

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/5375265A.6070702%40quake.fr.

[Ugo Bellavance]

Thanks,

Do I need the puppet concat module to do that?  I am on  puppet version 2.6 (using EPEL repo, "upgrading" to 2.7 soon.  I know it is not what this group recommends, but it's the best for me right now).

Error:

Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Resource failed with error ArgumentError: Invalid resource type concat at /etc/puppet/modules/lsyncd/manifests/init.pp:59 on node blabla

[Dirk Heinrichs]

Am 15.05.2014 21:25, schrieb Ugo Bellavance:

I wrote my first puppet module and it is for lsyncd.

Just looked into lsyncd and what it can achieve. While I like rsync and use it quite often, there's a better way to mirror directories to multiple destinations: BitTorrent Sync (btsync). btsync uses the BitTorrent algorithm to distribute the contents of a directory to an arbitrary number of other machines, where every other machine starts participating in the distribution process as soon as they have some chunks of the added file(s) available locally, thus producing less load on the originating machine.

There are some btsync modules available on Puppetforge, and I've also written one myself which provides a new resource type "btsync" which allows for adding/removing new sync folders to a running btsync instance, for example:

  btsync { 'test':
    secret   => hiera('btsync::test::secret'),    # The secret for this sync folder
    path     => 'path/to/test_folder',    # Where the sync folder should be stored on disc
    login    => hiera('btsync::login'),    # Username for btsync
    password => hiera('btsync::password'),    # Password
    ensure   => absent,    # Removes the sync folder from btsync AND on disc.
    require  => Exec['run_btsync'],   # Make sure btsync is installed and running.
  }

Bye...

    Dirk

--

Dirk Heinrichs, Senior Systems Engineer, Engineering Solutions
Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach
Tel: +49 2226 1596666 (Ansage) 1149
Email: d...@recommind.com
Skype: dirk.heinrichs.recommind
www.recommind.com

[Ugo Bellavance]

On Friday, May 16, 2014 8:12:55 AM UTC-4, Dirk Heinrichs wrote:

Am 15.05.2014 21:25, schrieb Ugo Bellavance:

I wrote my first puppet module and it is for lsyncd.

Just looked into lsyncd and what it can achieve. While I like rsync and use it quite often, there's a better way to mirror directories to multiple destinations: BitTorrent Sync (btsync). btsync uses the BitTorrent algorithm to distribute the contents of a directory to an arbitrary number of other machines, where every other machine starts participating in the distribution process as soon as they have some chunks of the added file(s) available locally, thus producing less load on the originating machine.

Thanks a lot for your input, but I'm not looking for one-many relationships, just multiple one-one relationships, so btsync would not be that better in my case.

Ugo 

[David Schmitt]

On 2014-05-16 12:34, Ugo Bellavance wrote:
> Do I need the puppet concat module to do that? I am on puppet version
> 2.6 (using EPEL repo, "upgrading" to 2.7 soon. I know it is not what
> this group recommends, but it's the best for me right now).

The "concat" proposed by Jakov is a pattern that's available in multiple
implementations (native type, script based define) in multiple modules.
If you can't find one working for your system, it won't be a big effort
to write one yourself.


Regards, David

[Jakov Sosic]

On 05/16/2014 12:34 PM, Ugo Bellavance wrote:

> Do I need the puppet concat module to do that? I am on puppet version
> 2.6 (using EPEL repo, "upgrading" to 2.7 soon. I know it is not what
> this group recommends, but it's the best for me right now).

Yes, you do if you use the code I provided.

[Ugo Bellavance]

Thanks,

I have finally been able to get what I wanted.  I open a issue (https://github.com/axkibe/lsyncd/issues/275) on the github project website and the main developper of lsyncd gave me a few lines of code to put in my main config file to have any file in a given directory to be imported in the main config file.  Then I created a module, based on a simple apache vhost module that was given as a sample in the "Pro Puppet" book.  It works for now, I have to test it more and implement it.  If someone would like to have the module I can provide the code, but it may not be good-looking or fully compliant with the highest standards...

Thanks for all your help.

[Ugo Bellavance]

I have finally been able to get what I wanted.  I open a issue (https://github.com/axkibe/lsyncd/issues/275) on the github project website and the main developper of lsyncd gave me a few lines of code to put in my main config file to have any file in a given directory to be imported in the main config file.  Then I created a module, based on a simple apache vhost module that was given as a sample in the "Pro Puppet" book.  It works for now, I have to test it more and implement it.  If someone would like to have the module I can provide the code, but it may not be good-looking or fully compliant with the highest standards...

Thanks for all your help.

By the way, I'd like to say that this community is fantastic.  The support has been provided by many different persons around the clock, no matter the version I use or my knowledge level about puppet.  I don't think that there is even one of our vendors (that we pay $) that matches that.  Even though I did help a lot in the past, I don't have much time right now, but I'm glad to see that this kind of support exists in the open-source software.