gitfs https using pygit2

[Ross Black]

Hi,

I have been trying to setup gitfs to retrieve via https, and hoped that someone can provide some pointers to help me fix it...
The setup fails with 'Unexpected HTTP status code: 401'

I have checked that the user/password is correct, and I can clone the git repo manually from the command line with that login.

Any help would be appreciated.

Thanks,
Ross


I am using Ubuntu 14.04.3 LTS (trusty), running SaltStack 2015.8.0
pip reports the following versions of the relevant packages:
  gitdb (0.6.4)
  pygit2 (0.23.2)
  cffi (1.2.1)

I installed pygit2 and dependencies the the following sequence:

sudo apt-get install libssh2-1-dev libssl-dev python-dev libffi-dev cmake pkg-config libcurl4-openssl-dev libhttp-parser-dev python-pip

wget https://github.com/libgit2/libgit2/archive/v0.23.3.tar.gz
tar -xvf v0.23.3.tar.gz
cd libgit2-0.23.3/
cmake .
make
sudo make install

sudo pip install --upgrade pip
sudo pip install cffi
sudo pip install pygit2
sudo ldconfig
python -c 'import pygit2'

in my master config:

fileserver_backend:
  - roots
  - git
gitfs_provider: pygit2
gitfs_remotes:
  - https://a.com/test-salt-depot:
    - user: bob
    - password: bobpass

The full error in the master log is:

2015-10-13 05:30:47,950 [salt.utils.gitfs ][ERROR ][28058] Exception 'Unexpected HTTP status code: 401' caught while fetching gitfs remote 'https://a.com/test-salt-depot'
Traceback (most recent call last):
 File "/usr/lib/python2.7/dist-packages/salt/utils/gitfs.py", line 1702, in fetch_remotes
 if repo.fetch():
 File "/usr/lib/python2.7/dist-packages/salt/utils/gitfs.py", line 882, in fetch
 fetch_results = origin.fetch()
 File "/usr/local/lib/python2.7/dist-packages/pygit2/remote.py", line 378, in fetch
 check_error(err)
 File "/usr/local/lib/python2.7/dist-packages/pygit2/errors.py", line 64, in check_error
 raise GitError(message)
GitError: Unexpected HTTP status code: 401
2015-10-13 05:30:47,952 [salt.utils.gitfs ][DEBUG ][28058] Removed lock for https://a.com/test-salt-depot

[Erik Johnson]

http://tools.ietf.org/html/rfc7235#section-3.1

Double check your auth info, you may have an incorrect password. If the password is all numbers, then it would probably have been loaded by PyYAML as an integer (unless we're forcing it to be a string, I'd have to check the code), and if so then it would need to be quoted to be loaded as a string.

[Ross Black]

Hi,

I have double-checked that my user/password is correct.  I can get that exact user/password to successfully clone using command line git.
After digging through the salt code I have reduced it to a simple python program that I think implements the same steps as salt.

import logging
import pygit2

cachedir = '/home/vagrant/tmp-git/cache'
url = 'https://gerrit.zz.com/test-salt-depot'

print("Init repository at {0}".format(cachedir))

repo = pygit2.init_repository(cachedir)

repo.create_remote('origin', url)
repo.config.set_multivar(
    'remote.origin.fetch',
    'FOO',
    '+refs/tags/*:refs/tags/*'
)
repo.config.set_multivar(
    'http.sslVerify',
    '',
    'False'
)

origin = repo.remotes[0]
origin.credentials = pygit2.UserPass('b...@gerrit.zz.com', 'aaa')
print("Fetching repository at {0}".format(cachedir))
fetch_results = origin.fetch()

This fails in the same way:

Init repository at /home/vagrant/tmp-git/cache
Fetching repository at /home/vagrant/tmp-git/cache

Traceback (most recent call last):

  File "/vagrant/stuff.py", line 26, in <module>

    fetch_results = origin.fetch()
  File "/usr/local/lib/python2.7/dist-packages/pygit2/remote.py", line 378, in fetch
    check_error(err)
  File "/usr/local/lib/python2.7/dist-packages/pygit2/errors.py", line 64, in check_error
    raise GitError(message)

_pygit2.GitError: Unexpected HTTP status code: 401

If I change into the temp directory after the failure, and then fetch via the command line (supplying user/password copied directly from the python code) it gets the repo OK.

$ cd /home/vagrant/tmp-git/cache
~/tmp-git/cache$ git fetch
Username for ...
Password for ...
... Receiving objects: 100% (556/556), 152.34 KiB | 0 bytes/s, done.
 * [new branch]      master     -> origin/master

I upgraded to using Python 2.7.10, but to no effect.
I am now completely stuck (my current guess is some interaction with how gerrit is configured)
Any assistance, or suggestions as to how I might debug this issue would be appreciated (or pointers to where I could get help .. I could not find a forum for pygit2)

Thanks,
Ross

[Ross Black]

Traceback (most recent call last):

  File "/vagrant/stuff.py", line 26, in <module>

    fetch_results = origin.fetch()
  File "/usr/local/lib/python2.7/dist-packages/pygit2/remote.py", line 378, in fetch
    check_error(err)
  File "/usr/local/lib/python2.7/dist-packages/pygit2/errors.py", line 64, in check_error
    raise GitError(message)

_pygit2.GitError: Unexpected HTTP status code: 401

If I change into the temp directory after the failure, and then fetch via the command line (supplying user/password copied directly from the python code) it gets the repo OK.

$ cd /home/vagrant/tmp-git/cache
~/tmp-git/cache$ git fetch
Username for ...
Password for ...
... Receiving objects: 100% (556/556), 152.34 KiB | 0 bytes/s, done.
 * [new branch]      master     -> origin/master

I upgraded to using Python 2.7.10, but to no effect.
I am now completely stuck (my current guess is some interaction with how gerrit is configured)
Any assistance, or suggestions as to how I might debug this issue would be appreciated (or pointers to where I could get help .. I could not find a forum for pygit2)

Thanks,
Ross


On Saturday, 17 October 2015 11:27:07 UTC+11, Erik Johnson wrote:

[michae...@gmail.com]

I am having this same issue, were you able to figure out a solution for this?

[michae...@gmail.com]

I figured this out, in case anyone else comes across this same problem. 

The cause of this issue lies in the libgit2 library and a fix can be found here:

https://github.com/libgit2/libgit2/issues/3338

My git server GitLab Omnibus running with the built-in nginx had a response auth header of "Www-Authenticate" instead of the "WWW-Authenticate" that the libgit2 code was looking for. 

The merge in the above link has a very simple fix to make the header check case-insensive.

My config which is now working

SaltStack 2015.8.1

pygit2 0.22.1

libgit2 0.22.3

I had tried using the latest version of libgit2 and pygit2, but there is currently an issue that it will not work with the current version of SaltStack (2015.8.1)

[Ross Black]

Thanks for the solution.

I ended up switching to use git + ssh, which also required me to downgrade to pygit 0.22.1 + libgit 0.22.3 due to the version incompatibility with 2015.8.0
(https://github.com/saltstack/salt/issues/28072#issue-112062519)

Ross