Pushing git repo to Windows agent

[Nick Miller]

Hi,

I'm just starting out with puppet, and I have a basic handle on it.  What I've been tasked to do is to be able to push files from a git repo to Windows servers.  The servers themselves will have no internet access, so installing cygwin & git on the agents is not an option.  What I'd like to do is create a manifest that clones the repo locally, then copies that directory to the Windows client.  I have vscrepo working, I just need to know how to pull the repo, save the files to the puppet master server, then copy that directory to the agent when it connects.

Thanks,

Nick

[David Schmitt]

Depending on your requirements to fidelity and coding skill, the tow
options coming two my mind are cronjob on the master or puppet function.

The cronjob on the master (or an equivalent vcsrepo resource in its
configuration) would take care that the repo is (relatively) up-to-date.
The agent would only recursively copy the result down to the nodes.
Upside: very easy to do. Downside: git repo might not be up-to-date,
race conditions while updating.

The puppet function could do the cloning in a temporary directory and
put all required files as finished resources into the catalog (ala
create_resource). That will bloat the catalog mightily but the clone
will be fresh and consistent.


Regards, David

[jcbollinger]

On Friday, March 7, 2014 2:13:47 PM UTC-6, Nick Miller wrote:

Hi,

I'm just starting out with puppet, and I have a basic handle on it.  What I've been tasked to do is to be able to push files from a git repo to Windows servers.  The servers themselves will have no internet access, so installing cygwin & git on the agents is not an option.

How does the latter follow from the former?

 

 What I'd like to do is create a manifest that clones the repo locally, then copies that directory to the Windows client.  I have vscrepo working, I just need to know how to pull the repo, save the files to the puppet master server, then copy that directory to the agent when it connects.

No, that's not how Puppet works.  Leaving aside for the moment the questions of initially cloning the repo to the master and of keeping it fresh (or not), you have two high-level alternatives:

1. The master puts appropriate resources in the target nodes' catalogs to direct them to fetch the needed files from it, or
   
2. The file transfer from master to agent is accomplished by a mechanism outside Puppet.

This being a Puppet forum, I'll limit my comments to option (1), which nevertheless affords many possibilities.  Among those are

• Use an Exec to have the agents clone the repo from the master's copy (no Internet access required for this).
  
• Declare a recursive File resource referring to the root of the cloned git repo, either
  
• using a puppet:// URL (no extra setup required, but likely to perform poorly), or
• referring to a network share exported by the master (requires the master to export a share, manage permissions, etc).
  
• Pack up the repo into a ZIP file; use a File resource to make the target nodes have a copy of the ZIP, and an Exec to have them unpack it when needed (well performing, relatively easy to recognize changes, but requires that the target nodes retain their copier of the ZIP).
  
• Pack up the repo into a Windows installer, and manage it on the target nodes as a Package resource.

No doubt there are other alternatives.  Personally I'd go with git-cloning from the master.


John

[Nick Miller]

Thank you for the replies.  What I've done is setup the agent on the puppet master to pull the latest version of the repo, then the agents recursively pull the directory.  There are a couple of problems I'm running into.  

The first is that line breaks seem to be stripped from any text files the nodes pull down.  On the master the file looks correct, but then on the node all of the line breaks are gone.

I occasionally run into the following error when pulling the latest version of the repo:

Error: /Stage[main]/Gitrepo/Vcsrepo[/tmp/git/puppet-test]: Could not evaluate: Execution of '/bin/git rev-parse origin/(detached from 3c40894)' returned 128: fatal: ambiguous argument 'origin/(detached from 3c40894)': unknown revision or path not in the working tree.

Use '--' to separate paths from revisions, like this:

'git <command> [<revision>...] -- [<file>...]'

origin/(detached from 3c40894)

It seems to happen when I add a file to the repo.  In the manifest I have the following:

class gitrepo {

         vcsrepo { '/tmp/git/puppet-test':

                ensure          => latest,

                provider        => git,

                source          => "g...@bitbucket.org:user/puppet-test.git",

                revision        => 'master',

        }

}

Thanks,

Nick

[jcbollinger]

On Monday, March 10, 2014 9:46:21 AM UTC-5, Nick Miller wrote:

Thank you for the replies.  What I've done is setup the agent on the puppet master to pull the latest version of the repo, then the agents recursively pull the directory.  There are a couple of problems I'm running into.  

The first is that line breaks seem to be stripped from any text files the nodes pull down.  On the master the file looks correct, but then on the node all of the line breaks are gone.

Most likely the files have UNIX line endings (LF instead of CRLF).  If you turn on the git option "core.autocrlf" for the repo on your clients, then git should convert convert line termination of text files automatically.

 

I occasionally run into the following error when pulling the latest version of the repo:

Error: /Stage[main]/Gitrepo/Vcsrepo[/tmp/git/puppet-test]: Could not evaluate: Execution of '/bin/git rev-parse origin/(detached from 3c40894)' returned 128: fatal: ambiguous argument 'origin/(detached from 3c40894)': unknown revision or path not in the working tree.

Use '--' to separate paths from revisions, like this:

'git <command> [<revision>...] -- [<file>...]'

origin/(detached from 3c40894)

I am not enough of a git illuminatus to immediately know what that means.  I find the appearance of the literal string '(detached from 3c40894)' suspicious, however.  My guess is that you are somehow creating a situation that vcsrepo does not handle appropriately.

 

It seems to happen when I add a file to the repo.  In the manifest I have the following:

class gitrepo {

         vcsrepo { '/tmp/git/puppet-test':

                ensure          => latest,

                provider        => git,

                source          => "g...@bitbucket.org:user/puppet-test.git",

                revision        => 'master',

        }

}

Supposing that none of the git experts around here jumps in with special insight on your problem, it would be helpful to identify exactly what steps are required to elicit the error.


John

[Nick Miller]

Steps to reproduce:

1) Create a new git repository (I'm using Bitbucket)

2) Create a file called 'test1.txt', and push it to the repo.

3) Create a class to pull the repo.  I'm using ssh cert authentication, but I'm not sure that matters

class gitrepo {

         vcsrepo { '/tmp/git/puppet-test':

                ensure          => latest,

                provider        => git,

                source          => "g...@bitbucket.org:user/puppet-test.git",

                revision        => 'master',

        }

}

4) Run 'puppet agent --test' to pull the repo down

5) Add another file to the repo, say 'test2.txt' and push the changes to git

6) Run 'puppet agent --test' again to pull the latest changes.

Any changes to existing files don't seem to cause the problem, only new files.

[jcbollinger]

On Tuesday, March 11, 2014 3:10:25 PM UTC-5, Nick Miller wrote:

Steps to reproduce:

1) Create a new git repository (I'm using Bitbucket)

2) Create a file called 'test1.txt', and push it to the repo.

3) Create a class to pull the repo.  I'm using ssh cert authentication, but I'm not sure that matters

class gitrepo {

         vcsrepo { '/tmp/git/puppet-test':

                ensure          => latest,

                provider        => git,

                source          => "g...@bitbucket.org:user/puppet-test.git",

                revision        => 'master',

        }

}

4) Run 'puppet agent --test' to pull the repo down

5) Add another file to the repo, say 'test2.txt' and push the changes to git

6) Run 'puppet agent --test' again to pull the latest changes.

Any changes to existing files don't seem to cause the problem, only new files.

Inasmuch as I am supposing that the problem is related to specifics of how you are manipulating the repo, I was looking for more detail at step (5), and maybe also at step (2) and even step (1).  Before that, however, have you tried running the agent with debug output turned on?  That should yield a record of all the external commands that are being run.  In particular, it may yield a clue as to where the "origin/(detached from 3c40894)" is coming from.


John