Re: [Puppet Users] Question about conditional exec notifications

[Henrik Lindberg]

Did you try using an update resource, and a build resource, where the
second depends on the first?

i.e. something like

exec { 'repository_update':
command => 'git pull',
}
exec { 'build':
command => 'make',
require => Exec['repository_update']
}

or, if you prefer:

exec { 'repository_update':
command => 'git pull',
}
exec { 'build':
command => 'make',
}

Exec['build'] -> Exec['repository_update']

- henrik

On 2012-09-10 13:12, Adrian Webb wrote:
> Hello,
>
> I've been trying to implement a puppet definition that uses vcsrepo and
> notifies the caller through an update_notify parameter when the
> repository contents change on disk. I have however found this very hard
> to accomplish.
>
> Basically I am trying to pull down a git repo and when the head of the
> repo changes on the disk of the agent, run a build command (such as
> make) into a release directory. So the system automatically tracks a
> branch or responds to changes in tags and the system automatically
> builds off of that information. In order to allow for multiple types of
> build processes I am trying to notify other resources through the exec
> notify property.
>
> My problem is this:
>
> The only way to trigger the notification in a conditional manner that I
> have found is to use the onlyif or unless properties on the exec
> resource. Notifications still happen on the exec command even if the
> command fails so I can not use an inline command to notify or not (that
> I know of). But the onlyif property which I have been trying seems to
> execute before any of the actual commands so it always runs before the
> vcsrepo update, which voids the intended purpose. I can not use
> functions with a conditional because they are only run on puppet master
> and also suffer from the same issue where they run before any commands,
> and I can not use a fact because I need a directory parameter and it is
> also runs before any of the commands.
>
> So I need a way to conditionally trigger a notify during the course of a
> execution run right after the vcsrepo gets done pulling down any
> updates. Does anyone know how I might accomplish this task through
> puppet? So far nothing I have tried works. I would think that this
> goal would not be that isolated (building a repository after and only
> after repository updates). In my case I normally build into release
> directories so absolutely can not afford to rebuild on every puppet
> execution which runs every 5 minutes.
>
> Thanks for any help you can provide!
>
> Adrian
>
> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/NNQsDH1nox8J.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.

[Adrian Webb]

Unfortunately what I need is a little more abstract because I have rolled my definition into a module that is used by other modules.  Below is my definition (that does not work as intended) right now

define git::repo (

  $repo_name            = $name,
  $user                 = $git::params::user,
  $group                = $git::params::group,
  $home                 = $git::params::home,
  $source               = $git::params::source,
  $revision             = $git::params::revision,
  $base                 = $git::params::base,
  $post_update_commands = $git::params::post_update_commands,
  $post_update_template = $git::params::post_update_template,
  $update_notify        = undef,

) {

  $repo_dir             = $home ? {
    ''                   => $repo_name,
    default              => "${home}/${repo_name}",
  }

  $repo_git_dir         = $base ? {
    'true'               => $repo_dir,
    default              => "${repo_dir}/.git"
  }

  $test_diff_cmd        = "diff ${repo_git_dir}/_NEW_REVISION ${repo_git_dir}/_LAST_REVISION"

  include git

  #--

  Exec {
    path => [ '/bin', '/usr/bin', '/usr/local/bin' ],
    cwd  => $repo_dir,
    user => $user,
  }

  #-----------------------------------------------------------------------------

  if $source and $revision {
    $revision_real = $revision
  }
  else {
    $revision_real = undef
  }

  vcsrepo { $repo_dir:
    ensure   => $base ? {
      'true'  => 'base',
      default => $source ? {
        ''      => 'present',
        default => 'latest',
      },
    },
    provider => 'git',
    owner    => $user,
    group    => $group,
    force    => true,
    source   => $source ? {
      ''      => undef,
      default => $source,
    },
    revision => $revision_real,
    require  => Class['git'],
  }

  #---

  exec { "${repo_dir}-new-revision":
    command     => "git rev-parse HEAD > ${repo_git_dir}/_NEW_REVISION",
    returns     => [ 0, 128 ],
    require     => Class['git'],
    subscribe   => Vcsrepo[$repo_dir],
  }

  exec { "${repo_dir}-update-notify":
    command => 'test true',
    onlyif  => $test_diff_cmd,
    notify  => $update_notify,
    require => Exec["${repo_dir}-new-revision"],
  }

  exec { "${repo_dir}-last-revision":
    command   => "git rev-parse HEAD > ${repo_git_dir}/_LAST_REVISION",
    returns   => [ 0, 128 ],
    subscribe => Exec["${repo_dir}-update-notify"],
  }

  #-----------------------------------------------------------------------------

  if $home and $base == 'false' {
    exec { "${repo_dir}-receive-deny-current-branch":
      command     => "git config receive.denyCurrentBranch 'ignore'",
      refreshonly => true,
      subscribe   => Vcsrepo[$repo_dir],
    }
  }

  file { "${repo_dir}-post-update":
    path      => "${repo_git_dir}/hooks/post-update",
    owner     => $user,
    group     => $group,
    mode      => '0755',
    content   => template($post_update_template),
    subscribe => Vcsrepo[$repo_dir],
  }
}

As you can see I use the vcsrepo module to update my repo and then desire to send a notify up to the caller through the parameters.  So my real problem is how to trigger the conditional exec notify after the vcsrepo resource pulls in updates.  It is very important that this stay abstract because this is used in multiple modules and in each triggers different build operations.

Thanks,
Adrian

[Adrian Webb]

Sorry, my last post got cut off:

So my real problem is how to trigger the conditional exec notify after the vcsrepo resource 
pulls in updates.  It is very important that this stay abstract because this is used in multiple 
modules and in each triggers different build operations.

[Ramin K]

The vcsrepo module should know when it updates code like a normal
provider. Something like this should work for you.

vcsrepo { '/home/someuser/mycode':
ensure => $mycode::data::ensure,
revision => $mycode::data::version,
source => 'g...@github.com:mycode/mycode.git',
notify => Exec['api bundle install'],
}

exec { 'api bundle install':
command => 'cd ~/mycode ; bundle install\'',
logoutput => on_failure,
notify => Class['apache::service'],
refreshonly => true,
}


Might get ugly with chains of refreshonly Exec statements.

Ramin

[Adrian Webb]

Are you sure this notify is only triggered when there is an actual change in the repo (working copy has changed) or is it when the vcsrepo resource is executed like most notify's?

[Ramin K]

On 10/9/2012 7:25 AM, Adrian Webb wrote:
> Are you sure this notify is only triggered when there is an actual
> change in the repo (working copy has changed) or is it when the vcsrepo
> resource is executed like most notify's?

I don't believe your logic on Puppet notifies is correct. Assuming the
provider is correctly written a notify should only be generated on a
change like in the case of File, Service, Package, etc.

In any case, test it first in a few scenarios and make sure. FWIW the
vcsrepo::git provider is low to middling quality and you're likely to
find cases that don't work as expected regardless.

Ramin

[Adrian Webb]

Will do.  Thanks!

Do you by any chance know how notifications are generated by providers?  In essence, what tells Puppet that there needs to be a notify?  I was looking through the vcsrepo git provider code but did not see anything that looked like it would tell but I am newer to the low level details of providers.  If you know where I would start I would be happy to troubleshoot issues with the vcsrepo git provider as they come up.

Thanks,
Adrian

[jcbollinger]

On Tuesday, October 9, 2012 6:12:30 AM UTC-5, Adrian Webb wrote:

Hello,

I've been trying to implement a puppet definition that uses vcsrepo and notifies the caller through an update_notify parameter when the repository contents change on disk.  I have however found this very hard to accomplish.

Basically I am trying to pull down a git repo and when the head of the repo changes on the disk of the agent, run a build command (such as make) into a release directory.  So the system automatically tracks a branch or responds to changes in tags and the system automatically builds off of that information.  In order to allow for multiple types of build processes I am trying to notify other resources through the exec notify property.

My problem is this:

The only way to trigger the notification in a conditional manner that I have found is to use the onlyif or unless properties on the exec resource.  Notifications still happen on the exec command even if the command fails so I can not use an inline command to notify or not (that I know of).  But the onlyif property which I have been trying seems to execute before any of the actual commands so it always runs before the vcsrepo update, which voids the intended purpose.  I can not use functions with a conditional because they are only run on puppet master and also suffer from the same issue where they run before any commands, and I can not use a fact because I need a directory parameter and it is also runs before any of the commands.

So I need a way to conditionally trigger a notify during the course of a execution run right after the vcsrepo gets done pulling down any updates.  Does anyone know how I might accomplish this task through puppet?  So far nothing I have tried works.  I would think that this goal would not be that isolated (building a repository after and only after repository updates).  In my case I normally build into release directories so absolutely can not afford to rebuild on every puppet execution which runs every 5 minutes.

In fact, your goal is not particularly well aligned with Puppet.   That doesn't mean you can't accomplish it via Puppet, but it's not surprising that you're having trouble getting there.  Puppet is focused on managing state, and the actual tasks performed for that purpose are intentionally de-emphasized, but your objective is characterized principally by performing specific tasks, as opposed to by the state achieved via doing so.

With that said, finding a suitable way forward probably depends on understanding the model.  The relevant parts work like this:

• A resource employing 'notify' sends an event to the designated other resources each time it is non-trivially synchronized, meaning that Puppet changes a managed property from out of sync to in sync
• Exec resources have no persistent state to examine.  Whether they are initially considered in sync is judged by their 'onlyif', 'unless', and 'creates' properties.  Execs for which none of those is set are considered out of sync on every Puppet run.
• How out of sync Exec resources are synchronized depends on the values of their 'refreshonly' parameters.  If an Exec's 'refreshonly' is false (the default) then it is synchronized by running its 'command'; otherwise, it is synchronized by doing nothing, but refreshed (if it receives an event) by running the 'command'.
  
• Resources that are initially in sync automatically succeed.  Resources that need to be synchronized succeed if and only if they can are successfully synchronized.  For an Exec to succeed, that means the 'command' exits with the status specified by the 'status' property, or with status 0 if an explicit status is not specified.

You have at least two relevant resources in your system: one managing the local copy of your VCS repo, and one managing the build status of the code therein.  To achieve your goal, the former must send an event to the latter if and only if it updates anything in the local copy, and the latter must do nothing unless it receives an event.

Much depends, therefore, on the specific resource types you are using, especially the one for the local copy of your repo.  You must use ones that support the needed signaling behavior.  You may be able to create one in Exec form if you don't have a suitable one available otherwise.  (Puppet does not have such a resource type built in, but there are add-in modules that might provide types that do what you want.)


John

[Adrian Webb]

That is definitely helpful information.  Thanks!

So about the repo notifications:

Could I assume that the true or false state of the def below in the vcsrepo.rb resource type controls whether the notification is fired or not for the related providers?  Forgive me I'm rather new to the lower level internals of resources and providers.

def insync?(is)

      @should ||= []

      case should

        when :present

          return true unless [:absent, :purged, :held].include?(is)

        when :latest

          if is == :latest

            return true

          else

            return false

          end

when :bare

return is == :bare

      end

    end

About the other resource type:

I can definitely see what your saying but it seems like if the notifications on the repo update worked then all the other build state resources should just work if notified provided they manage their state effectively (which so far they do)?  Admittedly, I have been using execs for things like my make, make install, etc...  It just seems like the repo notification is the major obstacle even with a custom resource to manage the state of the build.

Adrian

[Ramin K]

On 10/9/2012 7:43 AM, Adrian Webb wrote:
> Will do. Thanks!
>
> Do you by any chance know how notifications are generated by providers?
> In essence, what tells Puppet that there needs to be a notify? I was
> looking through the vcsrepo git provider code but did not see anything
> that looked like it would tell but I am newer to the low level details
> of providers. If you know where I would start I would be happy to
> troubleshoot issues with the vcsrepo git provider as they come up.
>
> Thanks,
> Adrian

I don't know much about building providers though I've hacked on the
vcsrepo a bit. My basic understanding is changes would be a create
property and Puppet would generate a notify from it.

http://docs.puppetlabs.com/guides/custom_types.html#properties

Most of my issues have centered around the git provider not pulling from
source before trying to apply tags, branches, or revisions, but that
looks like it was fixed three weeks ago.

Ramin