Authentication error when using gitfs

[Anna Berruezo]

I'm trying to setup gitfs to connect salt with my states in a salt repository. I'm following the guide but I'm having some problems.

Changes made to /etc/salt/master:

fileserver_backend:
  - roots
  - git

gitfs_provider: pygit2

gitfs_remotes:
  - git@somerepository:name/salt-states.git:
    - pubkey: /root/.ssh/id_rsa.pub
    - privkey: /root/.ssh/id_rsa

I installed libgit2 (with libssh2) and pygit2.
But then, it's impossible to connect the minions with the master.
With the master running, I start the minion and it just says: "Attempt to authenticate with the salt master failed"

If the master has already accepted the minion's keys and I try to run a state, I get the following error.

In the master:

master# [ERROR   ] Salt request timed out. If this error persists, worker_threads may need to be increased.
Failed to authenticate!  This is most likely because this user is not permitted to execute commands, but there is a small possibility that a disk error occurred (check disk/inode usage).

In the minion:

minion# [ERROR   ] Unhandled exception running mine.update
Traceback (most recent call last):
  File "/home/soft/python-2.7.8/lib/python2.7/site-packages/salt/utils/schedule.py", line 437, in handle_func
    ret['return'] = self.functions[func]()
  File "/home/soft/python-2.7.8/lib/python2.7/site-packages/salt/modules/mine.py", line 99, in update
    load['tok'] = _auth().gen_token('salt')
  File "/home/soft/python-2.7.8/lib/python2.7/site-packages/salt/modules/mine.py", line 25, in _auth
    __context__['auth'] = salt.crypt.SAuth(__opts__)
  File "/home/soft/python-2.7.8/lib/python2.7/site-packages/salt/crypt.py", line 810, in __init__
    self.crypticle = self.__authenticate()
  File "/home/soft/python-2.7.8/lib/python2.7/site-packages/salt/crypt.py", line 831, in __authenticate
    self.opts.get('auth_tries', 1)
  File "/home/soft/python-2.7.8/lib/python2.7/site-packages/salt/crypt.py", line 649, in sign_in
    raise SaltClientError('Attempt to authenticate with the salt master failed')
SaltClientError: Attempt to authenticate with the salt master failed

Anyone with a similar problem?

[monosij...@gmail.com]

I am also having issues with GitFS to work, in both stable and dev.

I installed GitPython but that did not help either.

Meanwhile this is what I got from Kyle:

...

You do NOT need both GitPython and the libgit solution.  EITHER one will do.

If you want per-repo mountpoints and other goodies....you need libgit2 instead of GitPython.

In installed libgit2 from source to get the latest version and match pygit2 in pip's repo.

You can follow the instructions here to do that: http://www.pygit2.org/install.html

They include instructions on how to compile and provide the download link to libgit2.

It is important that the versions of libgit2 and pygit2 are compatible

The salt instructions (http://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html) warn to have libssh2 dev libraries installed before compiling libgit2 or it will silently not pick up ssh capability.

To use 0.22.0 (the latest) of pygit and libgit2, and assuming you are using python 2.7.x on Ubuntu 14.04":

Have a compiler installed

apt-get install python-pip

apt-get install python-cffi

apt-get install libssh2--1-dev

wget https://github.com/libgit2/libgit2/archive/v0.22.0.tar.gz

tar xzf v0.22.0.tar.gz

cd libgit2-0.22.0/

cmake .

make

sudo make install

sudo ldconfig

pip install pygit2

That should get you close enough you can figure out the rest.  

Your cffi error was because you don't have the prereq installed (python-cffi)

I think when I investigated the libgit2 package from apt was not a good version for pygit2 0.22.0.  Hence why the compile from source.

...

I am continuing to try it.

WIll let you know how it works.

Thanks.

mono

[Colton Myers]

Hey Anna,

What version of libgit2 did you install? Additionally, are there errors in the master log? Debug logs might also give more info.

I will note that there were some regressions in gitfs in 2014.7.2, which you may be running into. They will be fixed in the upcoming 2014.7.3 (the fixes are already in the 2014.7 branch).

--
Colton Myers

Platform Engineer, SaltStack

@basepi on Github/Twitter/IRC

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Monosij Dutta-Roy]

Btw the basic git pulls do work fine with GitPython.

I am trying to now mount the repo using GitFS and not having much luck yet.

If you still need libgit2 et.al. you can link to this script from a vagrant box to be run when you setup salt-master:

https://github.com/mduttaroy-devops/configs-machines-testing/blob/master/environment.scripts/setup.gitFS.sh

Or you can use it directly.

Mono

--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/bvkmYb_2aNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.

[Tehmasp Chaudhri]

Colton,

What regressions? Would love to know cause we heavily use GitFS and I was anticipating the .2 release to upgrade to but don't want to if GitFS might be broken.

Thanks very much.

Tehmasp

[Tehmasp Chaudhri]

Getting GitFS to work properly was the most difficult part of my initial Salt installation.

We use pygit for our setup. Please see this GitHub issue for very useful information in getting pygit working properly. This is how I got GitFS working and I am still using this process.

This is from my salt master .sls file:

////////////////////////////////////////////////////////////////////////////////////////////////

# Install extra system/development pkgs per our SaltStack setup.

sysdev-pkgs:

  pkg.installed:

    - names:

      - make

      - cmake

      - build-essential

      - libffi-dev

      - libssh2-1-dev

      - python-dev

      - python-pip

# Install extra Python packages per our SaltStack setup.

pip-pkgs:

  pip.installed:

    - names:

      - pymongo

      - boto

    - require:

      - pkg: sysdev-pkgs

# We use GitFS w/ SSH URLs and need pygit2 support for this.

# See the compatibility version matrix here: http://www.pygit2.org/install.html

#

# Manually compile/install the following pkgs:

# libgit2 , cffi, pygit2

#

# See: https://github.com/saltstack/salt/issues/15424

#   wget https://github.com/libgit2/libgit2/archive/v0.21.2.tar.gz && tar zxpvf v0.21.2.tar.gz

#   cd libgit2-0.21.2

#   cmake . && make && make install

#   ldconfig

#   pip install cffi && pip install pygit2==0.21.4

////////////////////////////////////////////////////////////////////////////////////////////////

You can see from the commented out section that I am currently doing that manually during a new salt master installation. And the versions are very important between libgit2 and pygit2.

The following is how my gitfs_remotes.conf configuration is setup and the setup is important from what I discovered - namely the username:

% cat files/master.d/gitfs_remotes.conf

gitfs_provider: pygit2

gitfs_user: git

gitfs_pubkey: /root/.ssh/id_rsa.pub

gitfs_privkey: /root/.ssh/id_rsa

gitfs_remotes:

  - ssh://g...@server.com/repo.git

  ...

  ...

Hope that helps!!!

Tehmasp Chaudhri

[Tehmasp Chaudhri]

This is the issue to see - forgot to paste it at the top of my reply - https://github.com/saltstack/salt/issues/15424

[monosij...@gmail.com]

I am new to Salt and cannot GitFS to work. I will post separately on what I am trying to see if I can get some advice.

Thanks.

Mono

[Anna Berruezo]

Hi, sorry for answering so late.

At the time I was using whatever versions I got installed with pip. But after learning it's important I installed libgit2-0.21.2 and pygit2-0.21.4.

#salt --versions
           Salt: 2014.7.2
         Python: 2.7.8 (default, Mar  5 2015, 14:47:47)
         Jinja2: 2.7.3
       M2Crypto: 0.22
 msgpack-python: 0.4.5
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.11
          ioflo: Not Installed
          PyZMQ: 14.5.0
           RAET: Not Installed
            ZMQ: 4.0.5
           Mako: Not Installed

I also, I changed some parameters in the minion config file:

random_reauth_delay: 270
recon_default: 1000
recon_max: 30000
recon_randomize: True
auth_timeout: 60

But it didn't work either.

I tried to get debug logs and I got the following: (I just tried to ping.test one of the minions)

2015-03-23 16:08:09,535 [salt.loaded.int.fileserver.gitfs][DEBUG] Gitfs received 0 objects for remote git@repo-address/salt-states.git
2015-03-23 16:08:09,572 [salt.config][DEBUG] Reading configuration from /etc/salt/master
2015-03-23 16:08:09,619 [salt.config][DEBUG] Reading configuration from /etc/salt/master
2015-03-23 16:08:25,648 [salt.config][DEBUG] Reading configuration from /etc/salt/master
2015-03-23 16:08:25,736 [salt.config][DEBUG] Guessing ID. The id can be explicitly in set /etc/salt/minion
2015-03-23 16:08:25,759 [salt.config][INFO] Found minion id from generate_minion_id(): master.address.org
2015-03-23 16:08:25,760 [salt.config][DEBUG] Missing configuration file: ~/.saltrc
2015-03-23 16:08:25,761 [salt.utils.event][DEBUG] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
2015-03-23 16:08:25,762 [salt.utils.event][DEBUG] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
2015-03-23 16:08:30,770 [salt.payload][INFO] SaltReqTimeoutError: after 5 seconds. (Try 1 of 3)
2015-03-23 16:08:35,776 [salt.payload][INFO] SaltReqTimeoutError: after 5 seconds. (Try 2 of 3)
2015-03-23 16:08:40,781 [salt.payload][INFO] SaltReqTimeoutError: after 5 seconds. (Try 3 of 3)
2015-03-23 16:08:40,782 [salt.client][ERROR] Salt request timed out. If this error persists, worker_threads may need to be increased.

Thanks!

[Colton Myers]

> What regressions? Would love to know cause we heavily use GitFS and I was anticipating the .2 release to upgrade to but don't want to if GitFS might be broken.

I don't remember the details offhand. The problems were due to some file locking that a community member put in. Erik (terminalmage) fixed it fairly promptly but not before 2014.7.2 went out. 2014.7.4 has been tagged and is currently being packaged, so I recommend upgrading to 2014.7.4 instead.

[Anna Berruezo]

Hi, I just updated to 2014.7.4 and it doesn't work yet.

This is what I get when I try to ping a minion:

Error on the master terminal:

master # salt tomillo test.ping

[ERROR   ] Salt request timed out. If this error persists, worker_threads may need to be increased.
Failed to authenticate!  This is most likely because this user is not permitted to execute commands, but there is a small possibility that a disk error occurred (check disk/inode usage).

Master logs. I guess the error is "Missing configuration file: ~/.saltrc".

2015-04-09 11:08:50,133 [salt.config      ][DEBUG   ] Reading configuration from /etc/salt/master
2015-04-09 11:08:50,206 [salt.config      ][DEBUG   ] Guessing ID. The id can be explicitly in set /etc/salt/minion
2015-04-09 11:09:37,181 [salt.config      ][DEBUG   ] Reading configuration from /etc/salt/master
2015-04-09 11:09:37,269 [salt.config      ][DEBUG   ] Guessing ID. The id can be explicitly in set /etc/salt/minion
2015-04-09 11:09:37,292 [salt.config      ][INFO    ] Found minion id from generate_minion_id(): master.fib.upc.es
2015-04-09 11:09:37,294 [salt.config      ][DEBUG   ] Missing configuration file: ~/.saltrc
2015-04-09 11:09:37,295 [salt.utils.event ][DEBUG   ] MasterEvent PUB socket URI: ipc:///var/run/salt/master/master_event_pub.ipc
2015-04-09 11:09:37,295 [salt.utils.event ][DEBUG   ] MasterEvent PULL socket URI: ipc:///var/run/salt/master/master_event_pull.ipc
2015-04-09 11:09:42,303 [salt.payload     ][INFO    ] SaltReqTimeoutError: after 5 seconds. (Try 1 of 3)
2015-04-09 11:09:47,308 [salt.payload     ][INFO    ] SaltReqTimeoutError: after 5 seconds. (Try 2 of 3)
2015-04-09 11:09:52,314 [salt.payload     ][INFO    ] SaltReqTimeoutError: after 5 seconds. (Try 3 of 3)
2015-04-09 11:09:52,315 [salt.client      ][ERROR   ] Salt request timed out. If this error persists, worker_threads may need to be increased.

I restarted both the master and the minion. I've also cleared the cache (/var/cache/salt) on both of them.

master # salt --versions
           Salt: 2014.7.4

         Python: 2.7.8 (default, Mar  5 2015, 14:47:47)
         Jinja2: 2.7.3
       M2Crypto: 0.22

 msgpack-python: 0.4.6

   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.11
          ioflo: Not Installed
          PyZMQ: 14.5.0
           RAET: Not Installed
            ZMQ: 4.0.5

           Mako: 1.0.1

If I change back to the configuration (/etc/salt/master) I had before trying GitFS, it works just fine.
Changes I made to /etc/salt/master:

fileserver_backend:
  - roots
  - git

gitfs_provider: pygit2

gitfs_user: git
gitfs_pubkey: /root/.ssh/id_rsa.pub
gitfs_privkey: /root/.ssh/id_rsa
gitfs_remotes:

  - git@myrepo:group/salt-states.git

Also, am I supposed to have a special structure on the git repository?
Right now I just have a simple structure just to try it until it works properly.
Something like:

-salt-states
----hello
------init.sls

The state hello simply creates a directory.

Tell me if you need any more log reports.

Thanks!

[Monosij Dutta-Roy]

Hi Anna -

I was wondering if you got GitFS to work?

I have been having some issues with GitFS as well: https://mail.google.com/mail/u/0/#inbox/14c116afad008182

I hope it is ok to link it.

I will look for any updates and will post separately again in a day or so.

Thank you.

Mono

--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/bvkmYb_2aNU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.

[monosij...@gmail.com]

Sorry its here: https://groups.google.com/forum/#!topic/salt-users/bvkmYb_2aNU

Mono

To unsubscribe from this group and all its topics, send an email to salt-users+unsubscribe@googlegroups.com.

[Anna Berruezo]

I haven't managed to make it work yet. I have the same error from my previous message.

[Colton Myers]

Hey Anna,

A couple of things to try:

1. Ensure that you can `import pygit2` on a python shell on the master

2. Can you try this alternate format for your gitfs remote?

gitfs_remotes:
  - ssh://git@myrepo/group/salt-states.git

It would also be useful if you verified that you can clone the repo manually using git and the specified remote address.

The git repository should be structured exactly as you would structure your /srv/salt directory. The `master` branch will automatically resolve to the `base` environment.

--

Colton Myers

Platform Engineer, SaltStack

@basepi on Twitter/Github/IRC

[Anna Berruezo]

1. I can import pygit2 on a python shell on the master! (just tried it)

2. I get the following error when I use this format:
[ERROR   ] Exception 'Malformed URL 'ssh://g...@repo.address.edu:group/salt-states.git'' caught while fetching gitfs remote ssh://g...@repo.address.edu:group/salt-states.git

Also, I can clone the repo manually. My git structure is something like: /salt-states/base/*folders-states*
In the salt master configuration file I have these lines:
file_roots:
  base:
    - /srv/salt/base

[Colton Myers]

What version of pygit2 and libgit2 do you have installed?

>>> import pygit2

>>> pygit2.__version__

'0.22.0'

>>> pygit2.LIBGIT2_VERSION

'0.22.0'

--

Colton Myers

Platform Engineer, SaltStack

@basepi on Twitter/Github/IRC

[Anna Berruezo]

I get the following:

>>> import pygit2
>>> pygit2.__version__
'0.21.4'
>>> pygit2.LIBGIT2_VERSION
'0.21.2'

[Colton Myers]

Those versions should be fine....I'm not sure what's going on here. Mind opening an issue on Github so we don't lose track of this? We can get terminalmage (Erik Johnson) involved, maybe he'll have more ideas of how to debug this.

--

Colton Myers

Platform Engineer, SaltStack

@basepi on Twitter/Github/IRC

[Anna Berruezo]

Ok! I'll do that.
But first I'll try to test with 2015.5, in case it has been fixed somehow.

Thanks!

[Anna Berruezo]

I don't know how or why, but it works now.
Maybe it's the new version. Awesome!

On Tuesday, May 12, 2015 at 6:44:57 PM UTC+2, basepi wrote:

[Colton Myers]

Awesome! Glad it's working! =)

--

Colton Myers

Platform Engineer, SaltStack

@basepi on Twitter/Github/IRC