Gitfs backend with local git repo issue

[Jason Denning]

Hi All,
I am trying to use the Gitfs backend to serve state files as described here: http://docs.saltstack.com/topics/tutorials/gitfs.html using a local git repo as the 'remote'.  However, it seems that salt is not finding my top.sls file.  The log is showing "Loaded gitfs as virtual git" , so the backend is loading correctly, but when I try a salt 'test-host' state.highstate, no state is applied:

        changes:
            ----------
        comment:
            No Top file or external nodes data matches found
        name:
            No States
        result:
            False


The relevant part of my /etc/master file is:

fileserver_backend:
  - git

git_remotes:
  - file:///repos/base1.git

The repo was created (on ubuntu 12.04) like so:
mkdir -p /repos/base1.git
cd /repos/saltbase.git
git --bare init

I am able to clone the repo on the system, as expected with:
git clone file:///repos/saltbase.git

If I use a clone of the git repo as the "file_roots" directory, then the state gets applied properly.
i.e. /etc/salt/master:

file_roots:
  base:
    - /srv/saltbase (where this directory is a clone of the above git repo)

fileserver_backend:
  - roots



Again, everything works properly if I just clone the repo to a directory and use that directory as the "file_roots: base" parameter, so I really don't think there is a problem with my top.sls file but it's contents are:

base:
  'test-host':
    - test

and test.sls (in the same directory):

tmux:
  pkg:
    - installed


Everything I can find about the gitfs backend seems to be regarding using a truly remote git repo, but the config file indicates that using the file:/// URI format should work.  Does anyone have this working?  Any help would be greatly appreciated.

Thanks,
Jason

[Thomas S Hatch]

There is a bug here that usually triggers itself when using file:// with gitfs. Try stopping the master, clearing the gitfs cache (/var/cache/salt/master/gitfs/) and restart the master

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jason Denning]

Hi Thomas,
Sorry for the slow reply - I tried flushing the cache as you suggested, but to no avail.  I had to get things working quickly, and didn't have a chance to try again with the git backend until today. 

I setup a couple of VMs just for testing Salt (Ubuntu 12.04 running Salt 0.15.1), so this is a fresh attempt. 

I am now getting the following error when I try to start salt-master:

2013-05-16 10:50:48,743 [salt.loader      ][DEBUG   ] loading fileserver in ['/var/cache/salt/master/extmods/fileserver', '/usr/lib/pymodules/python2.7/salt/fileserver']
2013-05-16 10:50:48,744 [salt.loader      ][DEBUG   ] Skipping /var/cache/salt/master/extmods/fileserver, it is not a directory
2013-05-16 10:50:48,745 [salt.loader      ][ERROR   ] Failed to read the virtual function for fileserver: gitfs
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/salt/loader.py", line 686, in gen_functions
    virtual = mod.__virtual__()
  File "/usr/lib/pymodules/python2.7/salt/fileserver/gitfs.py", line 41, in __virtual__
    if not git.__version__ > '0.3.0':
NameError: global name 'git' is not defined
2013-05-16 10:50:48,747 [salt.master      ][ERROR   ] Failed to load fileserver backends, the configured backends are:
git
2013-05-16 10:50:48,747 [salt.master      ][ERROR   ] Master failed pre flight checks, exiting



I have repos hosted at /srv/repos/saltstates.git and /srv/repos/saltpillar.git .  My master config file is:

fileserver_backend:
  - git

gitfs_remotes:
  - file:///srv/repos/saltstates.git


  Am I missing something obvious, or should I file a bug report for this?

Thanks,
Jason

[Colton Myers]

Well, since you successfully loaded it earlier I'm going to assume that you have gitpython installed.

One thing people sometimes run into is that the base environment is remapped to the master branch on a repo.  So a master branch must exist -- when you clone the repo manually, does it default to a master branch, or something else?  If there is no remote ref called "master" then it won't bind it to your base environment.

Neither of these may be your problem, but since they are common, I wanted to make sure you checked them.

--

Colton Myers

Thanks,
Jason

[Jason Denning]

This is actually a fresh install, so missing gitpython actually did bite me.  (Details to follow) Yes, when I clone the repo manually it defaults to the master branch.  I just created a bare repo with a top.sls file and a test.sls that just installs a package to test things.  I have tested pushing and pulling to/from 'origin master' on this repo, and everything seems to work as expected (but I'm not a git guru)


My current state:
I installed the python-git package (Ubuntu), and that improved the situation but gave me this error message:

2013-05-16 11:25:24,719 [salt.loader      ][DEBUG   ] loading fileserver in ['/var/cache/salt/master/ex
tmods/fileserver', '/usr/lib/pymodules/python2.7/salt/fileserver']
2013-05-16 11:25:24,719 [salt.loader      ][DEBUG   ] Skipping /var/cache/salt/master/extmods/fileserve

r, it is not a directory

2013-05-16 11:25:24,725 [salt.master      ][ERROR   ] Failed to load fileserver backends, the configure
d backends are:
git
2013-05-16 11:25:24,725 [salt.master      ][ERROR   ] Master failed pre flight checks, exiting

I'm not sure why the /var/cache/salt/master/extmods directory wasn't created (maybe an issue with the ubuntu package?) as it existed in my previous installation.  I created the directory /var/cache/salt/master/extmods/fileserver, and now the error is simplified further, but still present:

2013-05-16 11:29:15,545 [salt.loader      ][DEBUG   ] loading fileserver in ['/var/cache/salt/master/ex
tmods/fileserver', '/usr/lib/pymodules/python2.7/salt/fileserver']
2013-05-16 11:29:15,550 [salt.master      ][ERROR   ] Failed to load fileserver backends, the configure
d backends are:
git
2013-05-16 11:29:15,551 [salt.master      ][ERROR   ] Master failed pre flight checks, exiting

Thanks for your help!

Jason

[Colton Myers]

So it's weird that it's looking for fileserver backends in the cache dir.  Can you please shut down your salt master and minion on that box, and then delete the /var/cache/salt directory?  That is generated on startup if it doesn't already exist.

--

Colton Myers

[Jason Denning]

Ok, I uninstalled the ubuntu package python-git and used pip to install gitpython, and that seems to have done the trick - so there must be some issue w/ the version of gitpython that is packaged with ubuntu 12.04.

Hopefully this will save someone else some time :)

Thanks again for all your help!

--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/YbJ-Gde8Q9A/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.

[Colton Myers]

Woot!  Glad it started working.  =)

--

Colton Myers

[Ștefan Rusu]

Hi,

Another thing to notice is the fact that the local git repo needs to be bare. Couldn't find any mention in the docs about this aspect. GitFS works with an usual repo, having a working tree attached, but the salt-master daemon must be restarted after every commit in order to reflect the changes.

Hope it helps others who evaluate the GitFS under a testing environment like I do. Doesn't hold much value under a production setup, therefore I don't insist about it.

Regards,

Ștefan

[Jean-Michel Smith]

It would be reasoanble to be able to

1. trigger a git refresh via a salt command of some kind
2. configure a time interval in which git refreshes occur

I've opened issue https://github.com/saltstack/salt/issues/8175 with this suggestion.

Actually, it would be nice if git commits could trigger a refresh, rather than having the master poll, but that's probably a bit more work.

regards,

Jean

[Tim O'Guin]

On Wed, Oct 30, 2013 at 8:49 AM, Jean-Michel Smith <jean.mi...@gmail.com> wrote:

It would be reasoanble to be able to

1. trigger a git refresh via a salt command of some kind
2. configure a time interval in which git refreshes occur

I've opened issue https://github.com/saltstack/salt/issues/8175 with this suggestion.

Actually, it would be nice if git commits could trigger a refresh, rather than having the master poll, but that's probably a bit more work.

Yea, there's no way for the salt master to know if a remote git repo has new commits without polling unless there is a commit hook that actually informs it. I imagine it'd be fairly easy to send a notification to trigger a pull on the salt master, perhaps via the salt-api (or any other API service that can accept JSON posts and then trigger an update on the master).