GitFS backend with environments - serving from master branch only

[monosij...@gmail.com]

In trying to learn how to use environments along with a GitFS backend I am running into some problems.

My environments work fine from  using file_roots, however when I use a GitFS backend it is not working.

...

So first I set up the master as:

file_roots:

  base:

    - /srv/salt

  webserver:

    - /srv/web

and it works fine. I deploy with Vagrant and map into /srv on the master the 2 directories salt and web.

...

Then I add on the GitFS as follows:

fileserver_backend:

  - git

gitfs_remotes:

  - https://github.com/mduttaroy-devops/Sandbox-SaltStack-Roots-01.git

gitfs_mountpoint: salt://

...

You can see the master file here: 

https://github.com/mduttaroy-devops/VagrantSandbox-Ansible-SaltStack/blob/master/provisioner.SaltStack/config.Master/master

My confusion is as below:

I use the gitfs_mountpoint to salt:// as I understand this will mount the entire repo under /srv?

So then I will have /srv/salt and /srv/web - from my repo which has the salt and web directories. 

I did read the documentation where each environment should be a different branch.

But I understand I can override that to serve everything from the master branch if I use the gitfs_mountpoint?

...

My repo with the salt and web directories here:

https://github.com/mduttaroy-devops/Sandbox-SaltStack-Roots-01

And topfile: https://github.com/mduttaroy-devops/Sandbox-SaltStack-Roots-01/blob/master/salt/top.sls

...

I keep getting the error:

          ID: states

    Function: no.None

      Result: False

     Comment: No Top file or external nodes data matches found

     Started: 

    Duration: 

     Changes:   

...

Salt Versions;

           Salt: 2014.7.1

         Python: 2.7.8 (default, Oct 20 2014, 15:05:19)

         Jinja2: 2.7.3

       M2Crypto: 0.21.1

 msgpack-python: 0.4.2

   msgpack-pure: Not Installed

       pycrypto: 2.6.1

        libnacl: Not Installed

         PyYAML: 3.11

          ioflo: Not Installed

          PyZMQ: 14.3.1

           RAET: Not Installed

            ZMQ: 4.0.4

           Mako: 1.0.0

...

Btw hope having a README file and LICENSE file in the repo does not throw off Salt in any way?

Thank you much for your help.

Mono

[monosij...@gmail.com]

An update (I am on Ubuntu 14.10):

pygit2 is not installing in my salt-master vm for some reason.

libgit2 installs fine.

When I try to install pygit2 I get the following error:

  Running setup.py (path:/tmp/pip_build_root/pygit2/setup.py) egg_info for package pygit2

    c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory

     #include <Python.h>

                        ^

    compilation terminated.

    Traceback (most recent call last):

      File "<string>", line 17, in <module>

      File "/tmp/pip_build_root/pygit2/setup.py", line 197, in <module>

        cmdclass=cmdclass)

      File "/usr/lib/python2.7/distutils/core.py", line 111, in setup

        _setup_distribution = dist = klass(attrs)

      File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 262, in __init__

        self.fetch_build_eggs(attrs['setup_requires'])

      File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 287, in fetch_build_eggs

        replace_conflicting=True,

      File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 631, in resolve

        dist = best[req.key] = env.best_match(req, ws, installer)

      File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 874, in best_match

        return self.obtain(req, installer)

      File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 886, in obtain

        return installer(requirement)

      File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 338, in fetch_build_egg

        return cmd.easy_install(req)

      File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 636, in easy_install

        return self.install_item(spec, dist.location, tmpdir, deps)

      File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 666, in install_item

        dists = self.install_eggs(spec, download, tmpdir)

      File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 856, in install_eggs

        return self.build_and_install(setup_script, setup_base)

      File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1078, in build_and_install

        self.run_setup(setup_script, setup_base, args)

      File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1066, in run_setup

        raise DistutilsError("Setup script exited with %s" % (v.args[0],))

    distutils.errors.DistutilsError: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    Complete output from command python setup.py egg_info:

    c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory

 #include <Python.h>

Thanks again.

Mono

------------------------------

[tim.h...@gmail.com]

  Running setup.py (path:/tmp/pip_build_root/pygit2/setup.py) egg_info for package pygit2

    c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory

     #include <Python.h>

 

Looks like you need to install the python-dev package on Ubuntu.

cheers,
Tim

[Monosij Dutta-Roy]

Hi Tim -

Thanks for your suggestion. However, installing the python-dev package did not help either.

I am now trying to incorporate a git pull of the develop branch during setup of Salt (from Vagrant) to see if that will install all the libraries.

So, to get GitFS to work properly, I will need: pygit2 and libgit2 - is that correct? Should the install of salt-master not install those automatically?

Also, if you get a chance please let me know if you think my original question on the GitFS setup is correct, so possibly setting up Salt with these libraries now should make it work?

Thanks again for all your help.

Mono

--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/D8XGHQITVjg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Colton Myers]

Hey Mono,

Gitfs currently supports GitPython as well. If you use libgit2, you need libgit2 and pygit2 (pygit2 is just the python bindings for libgit2). The major versions of each need to match, and you need 0.20.3 or newer.

But GitPython should work just fine as well.

To answer your original question, I think your configuration looks right. I will note that you cannot serve two environments out of a single branch with gitfs, even with gitfs_mountpoint.

That said, there's really not need to. Environments are completely optional, you can just put web stuff in web/ and core stuff in salt/ and just target the correct states to the correct servers using the topfile.

I should also note that with gitfs, it will not actually clone the repo into /srv/. It just *acts* as if the files are there, but it actually just cashes the repo in salt's cache directory.

Hope that helps!

--
Colton Myers

Platform Engineer, SaltStack

@basepi on Github/Twitter/IRC

SaltConf 2015 is Mar 3-5! http://saltconf.com

You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.

[RabidCicada]

Just to chime in a bit.  If you want per repo mount points and credentials etc (basically if you want per repo much-of-anything) then you will need libgit2 and pygit2.  Just make sure then when you compile libgit2 that you ALREADY have libssh2-dev installed or it will simply silently not pick up ssh support:).

~Kyle

--

"Ambush!?....I just ambushed you with a cup of coffee"
-CIA agent (RONIN)

[monosij...@gmail.com]

Hi Colton and Kyle -

Thanks for your replies. However still cannot get GitFS to work.

Generating several error messages in logs below. To confirm - I am running off the develop branch, being setup from Vagrant.

I installed GitPython and libnacl using pip (had to install pip).

Could not install pyghmi using pip. Installed python-dev from apt which satisfied the req.

Then on restart and trying a highstate, got the traceback part of the error log.

I am getting errors when I run salt versions report as well.

------------------------------------------------------------------

Hi Kyle -

I could not install libgit2 and pygit2. Pip cannot find libgit2, pygit2 generates errors.

libbssh2-dev is not available in pip either or as apt-get install in Ubuntu 14.04/10. Where do I get these?

This is after I have installed python-dev. I have tried installing on Ubuntu 14.10 and 14.04.

So to confirm - you re saying I will need libgit2 and pygit2 and installing GitPython alone will not work?

------------------------------------------------------------------

Hi Colton -

I will move back to stable to try out these aspects again, however to get to Colton's reply above:

Environments are completely optional, you can just put web stuff in web/ and core stuff in salt/ and just target the correct states to the correct servers using the topfile.

My question is - as per the example in your book, in getting to work the environments: base and webserver, but with only 1 git repo with master branch, what would be the best way to do it?

I am just trying to get your example to work from GitFS, but with a single Git repo with only master branch.

Will that be possible? Would that be defining the gitfs_mountpoint or some other directive?

I will then try it with two branches with a single repo and then perhaps 2 different repos 1 for base (master) and 1 for webserver.

But right now, I would like to get it working from the master branch off the repo I defined, w/o changing the config file for master in the base and webserver environment settings.

Also Colton you mention:

note that with gitfs, it will not actually clone the repo into /srv/. It just *acts* as if the files are there, but it actually just cashes the repo in salt's cache directory

Ok so then: /srv/salt and /srv/web are used as settings themselves(?), when GitFS is being used since no physical structure is being defined?.

Perhaps a physical structure is being defined within the cache directory, but really it is used a setting for the environment settings for base and webserver?

I am trying to get to an understanding based on logical vs physical modeling as used in DBMS terminology - I think it is appropriate here? Also, that is my background.

ie. as soon as you use GitFS, all structures in the environments are essentially logical structures, that may be rendered as a physical structure in cache?

I am putting some effort into this as I plan to be working off of Git, I think it is an important topic as various ways to set up git is possible and also because there are so many different options for git in the salt master configurations.

Again I am in Ubuntu 14.10 working off of develop branch.

-------------------------------------------------------------

Output from /var/logs/salt/master (after installing GitPython, libnacl, python-dev, but not libgit2 and pygit2).

-------------------------------------------------------------

2015-02-28 02:45:30,462 [salt.client      ][DEBUG   ][6319] Checking whether jid 20150228024525442068 is still running

2015-02-28 02:45:35,486 [salt.utils.lazy  ][DEBUG   ][6319] LazyLoaded no_return.output

2015-02-28 16:37:12,230 [rest_tornado     ][ERROR   ][942] ImportError! No module named tornado

2015-02-28 16:37:20,255 [salt.loader      ][ERROR   ][1412] Error loading module.nacl: libnacl import error, perhaps missing python libnacl package

2015-02-28 16:37:20,256 [salt.loader      ][ERROR   ][1409] Error loading module.nacl: libnacl import error, perhaps missing python libnacl package

2015-02-28 16:37:20,691 [salt.loader      ][ERROR   ][1409] Error loading module.ipmi: No module named pyghmi.ipmi

2015-02-28 16:37:20,697 [salt.loader      ][ERROR   ][1412] Error loading module.ipmi: No module named pyghmi.ipmi

2015-02-28 17:36:26,958 [salt.loader      ][ERROR   ][4009] Error loading module.nacl: libnacl import error, perhaps missing python libnacl package

2015-02-28 17:36:27,079 [salt.loader      ][ERROR   ][4018] Error loading module.nacl: libnacl import error, perhaps missing python libnacl package

2015-02-28 17:36:27,179 [salt.loader      ][ERROR   ][4009] Error loading module.ipmi: No module named pyghmi.ipmi

2015-02-28 17:36:27,275 [salt.loader      ][ERROR   ][4018] Error loading module.ipmi: No module named pyghmi.ipmi

2015-02-28 17:45:36,743 [salt.loaded.int.fileserver.gitfs][ERROR   ][5135] No suitable version of pygit2/libgit2, GitPython, or Dulwich is installed.

2015-02-28 17:45:38,164 [salt.loaded.int.fileserver.gitfs][ERROR   ][5143] No suitable version of pygit2/libgit2, GitPython, or Dulwich is installed.

2015-02-28 17:45:45,271 [salt.loaded.int.fileserver.gitfs][ERROR   ][5151] No suitable version of pygit2/libgit2, GitPython, or Dulwich is installed.

2015-02-28 17:45:45,280 [salt.loaded.int.fileserver.gitfs][ERROR   ][5154] No suitable version of pygit2/libgit2, GitPython, or Dulwich is installed.

2015-02-28 17:45:45,523 [salt.loader      ][ERROR   ][5156] Error loading module.nacl: libnacl import error, perhaps missing python libnacl package

2015-02-28 17:45:45,632 [salt.loader      ][ERROR   ][5156] Error loading module.ipmi: No module named pyghmi.ipmi

2015-02-28 17:45:46,009 [salt.loaded.int.fileserver.gitfs][ERROR   ][5155] No suitable version of pygit2/libgit2, GitPython, or Dulwich is installed.

2015-02-28 17:45:46,015 [salt.loaded.int.fileserver.gitfs][ERROR   ][5157] No suitable version of pygit2/libgit2, GitPython, or Dulwich is installed.

2015-02-28 17:50:00,057 [salt.loader      ][ERROR   ][8691] Error loading module.nacl: libnacl import error, perhaps missing python libnacl package

2015-02-28 17:50:00,170 [salt.loader      ][ERROR   ][8691] Error loading module.ipmi: No module named pyghmi.ipmi

2015-02-28 17:57:09,582 [salt.loader      ][ERROR   ][8692] Failed to import module nacl, this is due most likely to a syntax error:

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 925, in _load_module

    ), fn_, fpath, desc)

  File "/usr/lib/python2.7/dist-packages/salt/modules/nacl.py", line 71, in <module>

    import libnacl.secret

  File "/usr/local/lib/python2.7/dist-packages/libnacl/__init__.py", line 75, in <module>

    nacl = _get_nacl()

  File "/usr/local/lib/python2.7/dist-packages/libnacl/__init__.py", line 73, in _get_nacl

    raise OSError(msg)

OSError: Could not locate nacl lib, searched for libsodium.so, libsodium.so.13, libsodium.so.10, libsodium.so.5, libsodium.so.4,  and tweetnacl.so

2015-02-28 17:57:13,341 [salt.daemons.masterapi][ERROR   ][8673] Exception [Errno 2] No such file or directory: '/var/cache/salt/master/gitfs/af0caa4c199ef95d71ceb726f0842f17/update.lk' occurred in file server update

2015-02-28 17:58:07,628 [salt.loader      ][ERROR   ][10144] Failed to import module nacl, this is due most likely to a syntax error:

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 925, in _load_module

    ), fn_, fpath, desc)

  File "/usr/lib/python2.7/dist-packages/salt/modules/nacl.py", line 71, in <module>

    import libnacl.secret

  File "/usr/local/lib/python2.7/dist-packages/libnacl/__init__.py", line 75, in <module>

    nacl = _get_nacl()

  File "/usr/local/lib/python2.7/dist-packages/libnacl/__init__.py", line 73, in _get_nacl

    raise OSError(msg)

OSError: Could not locate nacl lib, searched for libsodium.so, libsodium.so.13, libsodium.so.10, libsodium.so.5, libsodium.so.4,  and tweetnacl.so

-------------------------------------------------------------

Error when trying to install pygit2 (after installing GitPython, libnacl, python-dev, but not libgit2).

-------------------------------------------------------------

Downloading/unpacking pygit2

  Downloading pygit2-0.22.0.tar.gz (423kB): 423kB downloaded

  Running setup.py (path:/tmp/pip_build_root/pygit2/setup.py) egg_info for package pygit2

    c/_cffi_backend.c:13:17: fatal error: ffi.h: No such file or directory

     #include <ffi.h>

    c/_cffi_backend.c:13:17: fatal error: ffi.h: No such file or directory

 #include <ffi.h>

-------------------------------------------------------------

Cannot find packages from Ubuntu or pip when trying to install libgit2, pygit2, libssh2-dev.

-------------------------------------------------------------

vagrant@salt:~$ sudo pip install libgit2

Downloading/unpacking libgit2

  Could not find any downloads that satisfy the requirement libgit2

Cleaning up...

No distributions at all found for libgit2

Storing debug log for failure in /home/vagrant/.pip/pip.log

vagrant@salt:~$ apt.install libgit2 pygit2

Reading package lists... Done

Building dependency tree       

Reading state information... Done

E: Unable to locate package libgit2

E: Unable to locate package pygit2

vagrant@salt:~$ apt.install python-libgit2 python-pygit2

Reading package lists... Done

Building dependency tree       

Reading state information... Done

E: Unable to locate package python-libgit2

E: Unable to locate package python-pygit2

vagrant@salt:~$ sudo pip install libssh2-dev

Downloading/unpacking libssh2-dev

  Could not find any downloads that satisfy the requirement libssh2-dev

Cleaning up...

No distributions at all found for libssh2-dev

Storing debug log for failure in /home/vagrant/.pip/pip.log

vagrant@salt:~$ apt.install libssh2-dev

Reading package lists... Done

Building dependency tree       

Reading state information... Done

E: Unable to locate package libssh2-dev

==================================================================================

[RabidCicada]

You do NOT need both GitPython and the libgit solution.  EITHER one will do.  If you want per-repo mountpoints and other goodies....you need libgit2 instead of GitPython.

In installed libgit2 from source to get the latest version and match pygit2 in pip's repo.

You can follow the instructions here to do that: http://www.pygit2.org/install.html

They include instructions on how to compile and provide the download link to libgit2.  It is important that the versions of libgit2 and pygit2 are compatible

The salt instructions (http://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html) warn to have libssh2 dev libraries installed before compiling libgit2 or it will silently not pick up ssh capability.

To use 0.22.0 (the latest) of pygit and libgit2, and assuming you are using python 2.7.x on Ubuntu 14.04":

• Have a compiler installed
  
• apt-get install python-pip
  
• apt-get install python-cffi
  
• apt-get install libssh2--1-dev
  
• wget https://github.com/libgit2/libgit2/archive/v0.22.0.tar.gz
  
• tar xzf v0.22.0.tar.gz
  
• cd libgit2-0.22.0/
  
• cmake .
  
• make
  
• sudo make install
  
• sudo ldconfig
  
• pip install pygit2
  

That should get you close enough you can figure out the rest.  

Your cffi error was because you don't have the prereq installed (python-cffi)

I think when I investigated the libgit2 package from apt was not a good version for pygit2 0.22.0.  Hence why the compile from source.

~Kyle

On Sat, Feb 28, 2015 at 2:10 PM, <monosij...@gmail.com> wrote:

------------------------------------------------------------------

Hi Kyle -

I could not install libgit2 and pygit2. Pip cannot find libgit2, pygit2 generates errors.

libbssh2-dev is not available in pip either or as apt-get install in Ubuntu 14.04/10. Where do I get these?

This is after I have installed python-dev. I have tried installing on Ubuntu 14.10 and 14.04.

So to confirm - you re saying I will need libgit2 and pygit2 and installing GitPython alone will not work?

[monosij...@gmail.com]

Thank you Kyle. I will try it out. But as you mention libgit2 would only be required if I wanted to use per-repo mountpoints.

I am just trying to get the example from Colton's book working for now.

So I should be able to get by with GitPython only. But the salt-server is still generating the errors after I install GitPython stating libgit2 and pygit2 are not installed.

Perhaps what I am doing as per Colton's book is about per repo mountpoints since it is about 2 different environments?

This is the master I am trying to work with - which has 2 environments mapped to base and webserver, which I am trying to serve from my repo, but just from master branch:

https://github.com/mduttaroy-devops/VagrantSandbox-Ansible-SaltStack/blob/master/provisioner.SaltStack/config.Master/master

Thank you again.

Mono