Storing pillar data in GIT

[Dave Page]

Hi

Can anyone give me a little guidance on using git as an ext_pillar? I
want to move my entire pillar into git, and have created an
appropriate repo containing top.sls and the rest of the files in the
same structure as I had on the filesystem. I've added

ext_pillar:
- git: master file:///srv/salt-pillar.git"

in the master config file and changed

pillar_roots:
base:
- /srv/salt-pillar

to

pillar_roots:
base:
- /

But it doesn't seem to find the pillar. I'm a little confused about
what I need in pillar_roots in this setup, which may be the source of
my problem. I was similarly confused when I moved my state files into
git, and in the end had success using the following (hence my similar
edit to pillar_roots).

file_roots:
base:
- /

fileserver_backend:
- git

gitfs_remotes:
- file:///srv/salt-states.git

Appreciate any hints. Thanks.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

[Dave Page]

Can anyone offer any suggestions on this?

Thanks.

[Colton Myers]

I haven't used the new git external pillar.  I will say, however, that you don't need to define file_roots for gitfs.  Just the fileserver_backend and gifs_remotes declarations are needed.

It's possible that removing pillar_roots will solve your problem, but I'm not sure.

--

Colton Myers

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Thomas Jackson]

Yea, the git pillar doesn't need the directory set as the pillar base. All you need in your config is:

ext_pillar:

  - git: master git://gitserver/git-pillar.git

Docs: http://docs.saltstack.com/ref/pillar/all/salt.pillar.git_pillar.html?highlight=ext_pillar%20git

If that doesn't work for you let me know-- i wrote it ;)

Just keep in mind it is (currently) a single branch/environment unlike the other gitfs_remotes. I'm working on that, but its a bit of a project to modularize the pillar filesystem walking.

[Andres Douglas]

Hi Thomas, I'm trying to use ext_pillar with git and running into the following problem:

My /etc/salt/master has the following

For the salt roots

fileserver_backend:

  - git

gitfs_remotes:

  - git+ssh://g...@github.com:bakodo/salt-django-example.git

# according to http://docs.saltstack.com/topics/tutorials/gitfs.html#gitfs-remotes-over-ssh

What is throwing me off is that the way to configure the pillar is inconsistent with the above, and in my mind, the documentation is incorrect. If I were to follow the docs I'd end up with something along the lines of:

ext_pillar:

  - git: master git://github.com/bakodo/salt-pillar.git

I clear everything in /var/cache/salt/master, re-start the master, and run salt '*' state.highstate and get the following

[DEBUG   ] Loaded git_pillar as virtual git

[ERROR   ] Failed to load ext_pillar git: 'git fetch' returned exit status 128: fatal: remote error: 

  Repository not found.

Traceback (most recent call last):

  File "/usr/lib/pymodules/python2.7/salt/pillar/__init__.py", line 411, in ext_pillar

    ext = self.ext_pillars[key](self.opts['id'], pillar, val)

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 140, in ext_pillar

    if not update(branch, repo_location):

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 96, in update

    repo = init(branch, repo_location)

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 85, in init

    repo.git.fetch()

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 231, in <lambda>

    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 488, in _call_process

    return self.execute(make_call(), **_kwargs)

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 381, in execute

    raise GitCommandError(command, status, stderr_value)

GitCommandError: 'git fetch' returned exit status 128: fatal: remote error: 

  Repository not found.

[INFO    ] AES payload received with command _serve_file

[INFO    ] AES payload received with command _ext_nodes

[INFO    ] AES payload received with command _return

[INFO    ] Got return from tote.bakodo.jp for job 20131018100544563588

... wait a few seconds...

[WARNING ] GitPython exception caught while fetching: len([]) != len(['ssh: Could not resolve hostname github.com:bakodo: Name or service not known', ''])

There's something interesting here, because at first it can't seem to find the repo specified under ext_pillar - which makes sense, since that's the wrong schema for a github repo to begin with. But a few seconds later, it seems to be having trouble with the other schema, which I can only assume is coming from the gitfs remotes.

I was wondering if it had to do with the private key not being under id_rsa, so I moved it there, and same result.

On the salt client's end, I get the following error:

   State: - no

    Name:      states

    Function:  None

        Result:    False

        Comment:   No Top file or external nodes data matches found

        Changes:   

If I try changing the repo to read something close to the URL github gives you for ssh access to the repo:

ext_pillar:

  - git: master git://github.com:bakodo/salt-pillar.git

I still get the intermitent warnings that were happening earlier, which I think is safe to assume are coming from gitfs 

[WARNING ] GitPython exception caught while fetching: len([]) != len(['ssh: Could not resolve hostname github.com:bakodo: Name or service not known', ''])

If I now go ahead and do a highstate, I still get 

----------

    State: - no

    Name:      states

    Function:  None

        Result:    False

        Comment:   No Top file or external nodes data matches found

        Changes:   

 on the salt client

and the salt-master log now shows:

[DEBUG   ] loading pillar in ['/var/cache/salt/master/extmods/pillar', '/usr/lib/pymodules/python2.7/salt/pillar']

[DEBUG   ] Skipping /var/cache/salt/master/extmods/pillar, it is not a directory

[DEBUG   ] Loaded git_pillar as virtual git

[ERROR   ] Failed to load ext_pillar git: 'git fetch' returned exit status 128: fatal: unable to connect to github.com:

github.com: Servname not supported for ai_socktype

Traceback (most recent call last):

  File "/usr/lib/pymodules/python2.7/salt/pillar/__init__.py", line 411, in ext_pillar

    ext = self.ext_pillars[key](self.opts['id'], pillar, val)

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 140, in ext_pillar

    if not update(branch, repo_location):

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 96, in update

    repo = init(branch, repo_location)

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 85, in init

    repo.git.fetch()

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 231, in <lambda>

    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 488, in _call_process

    return self.execute(make_call(), **_kwargs)

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 381, in execute

    raise GitCommandError(command, status, stderr_value)

GitCommandError: 'git fetch' returned exit status 128: fatal: unable to connect to github.com:

github.com: Servname not supported for ai_socktype

If I add 

ext_pillar:

  - git: master git://g...@github.com:bakodo/salt-pillar.git

I get 

DEBUG   ] loading pillar in ['/var/cache/salt/master/extmods/pillar', '/usr/lib/pymodules/python2.7/salt/pillar']

[DEBUG   ] Skipping /var/cache/salt/master/extmods/pillar, it is not a directory

[DEBUG   ] Loaded git_pillar as virtual git

[ERROR   ] Failed to load ext_pillar git: 'git fetch' returned exit status 128: fatal: unable to connect to g...@github.com:

g...@github.com: Servname not supported for ai_socktype

Traceback (most recent call last):

  File "/usr/lib/pymodules/python2.7/salt/pillar/__init__.py", line 411, in ext_pillar

    ext = self.ext_pillars[key](self.opts['id'], pillar, val)

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 140, in ext_pillar

    if not update(branch, repo_location):

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 96, in update

    repo = init(branch, repo_location)

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 85, in init

    repo.git.fetch()

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 231, in <lambda>

    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 488, in _call_process

    return self.execute(make_call(), **_kwargs)

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 381, in execute

    raise GitCommandError(command, status, stderr_value)

GitCommandError: 'git fetch' returned exit status 128: fatal: unable to connect to g...@github.com:

g...@github.com: Servname not supported for ai_socktype

[INFO    ] AES payload received with command _serve_file

[INFO    ] AES payload received with command _ext_nodes

[INFO    ] AES payload received with command _return

[INFO    ] Got return from tote.bakodo.jp for job 20131018102133219402

Finally if I set it to what I think should be the winning formula:

ext_pillar:

  - git: master git+ssh://g...@github.com:bakodo/salt-pillar.git

I get the dissapointing:

[DEBUG   ] Skipping /var/cache/salt/master/extmods/pillar, it is not a directory

[DEBUG   ] Loaded git_pillar as virtual git

[ERROR   ] Failed to load ext_pillar git: 'git fetch' returned exit status 128: ssh: Could not resolve hostname github.com:bakodo: Name or service not known

fatal: The remote end hung up unexpectedly

Traceback (most recent call last):

  File "/usr/lib/pymodules/python2.7/salt/pillar/__init__.py", line 411, in ext_pillar

    ext = self.ext_pillars[key](self.opts['id'], pillar, val)

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 140, in ext_pillar

    if not update(branch, repo_location):

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 96, in update

    repo = init(branch, repo_location)

  File "/usr/lib/pymodules/python2.7/salt/pillar/git_pillar.py", line 85, in init

    repo.git.fetch()

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 231, in <lambda>

    return lambda *args, **kwargs: self._call_process(name, *args, **kwargs)

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 488, in _call_process

    return self.execute(make_call(), **_kwargs)

  File "/home/ubuntu/src/gitpython/git/cmd.py", line 381, in execute

    raise GitCommandError(command, status, stderr_value)

GitCommandError: 'git fetch' returned exit status 128: ssh: Could not resolve hostname github.com:bakodo: Name or service not known

fatal: The remote end hung up unexpectedly

[INFO    ] AES payload received with command _serve_file

[INFO    ] AES payload received with command _ext_nodes

[INFO    ] AES payload received with command _return

[INFO    ] Got return from tote.bakodo.jp for job 20131018102341198618

...

[WARNING ] GitPython exception caught while fetching: len([]) != len(['ssh: Could not resolve hostname github.com:bakodo: Name or service not known', ''])

I've ran out of ideas at this point. The only thing I can think of is something wrong with GitPython. I installed GitPython 0.3.1beta2 with pip, but I did look in my site-packages and found the 0.1.7 version. I renamed it and when I load it in python it is not the 0.3.1beta2. Not sure if there's a better way to check if salt is using the correct version (btw, salt v 0.17.0)

Any pointer would be much appreciated, as I am in the middle of trying to deploy a number of sites using salt, implementing this - https://groups.google.com/forum/#!topic/salt-users/bh83KWlw1nc

Thanks a lot,

Andy

[Andres Douglas]

Clarification:

on. I renamed it and when I load it in python it is not the 0.3.1beta2. 

should be:

on. I renamed it and when I load it in python it is NOW the 0.3.1beta2. 

So in python the correct version of GitPython does load up.

[Alexey Ivanov]

TLDR; Try:
- git: master g...@github.com:bakodo/salt-pillar.git

Long version:
Problem is that you are using private repo and it is only accessible via ssh protocol, but you trying to use git://.
When you try something like git://github.com/bakodo/salt-pillar.git github can't authorize you, so it throws "repository not found" error to protect repository's privacy. When you use git://github.com:bakodo/salt-pillar.git git client assumes that host name is "github.com:bakodo" which throws error inside libc when it's unable to resolve such a weird hostname. When you use git://g...@github.com:bakodo/salt-pillar.git it also throws error inside libc (probably in getaddrinfo or getservbyname) saying that service name is invalid.

Also user under which actual git command is run should have an access to unencrypted ssh private key either directly(via default location or custom specified in .ssh/config) or via ssh-agent.

Probably git-pillar should use modules/git.py instead of gitpython, otherwise some work must be done to support user-provided identities, like: https://github.com/saltstack/salt/commit/ba907ad8837cae58a886320316dc723960009bef

[Andres Douglas]

Alexey, thanks for the pointer. I think that worked....

Now it seems like GitFS is choking on the github URL schema. 

The docs seems to want something along the lines of 

git+ssh://g...@github.com/andresdouglas/salt-django-example.git

which is the incorrect schema for github repo access over ssh. That should be

git+ssh://g...@github.com:andresdouglas/salt-django-example.git

but when given that, it isn't able to resolve github.com:andresdouglas as the host. 

For public repos an immediate workaround is using the https schema
https://github.com/andresdouglas/salt-django-example.git

This seems like a bug. Should I file it in the github?

[Alexey Ivanov]

Strange, judging by code (it uses same GitPython module) both of those should work:
* g...@github.com:$USER/$REPO.git
* git+ssh://g...@github.com/$USER/$REPO.git

But I'm not really a salt developer and don't have any kind of salt testing environment set up so my testing is based purely on IPython =)

PS. Yet again for git-over-ssh to work for git_remotes now salt-MASTER should be run under user that has access to unencrypted private key either directly under location specified in ssh_config(5) or via ssh-agent(1).

--
Sincerely,
Alexey Ivanov

[Colton Myers]

Andres, did you actually try both URL schemas?  Wanted to make sure you didn't just assume the docs were wrong and not even try the version with a slash instead of a colon.

--

Colton Myers

[Andres Douglas]

Let's see, just did a bunch of testing, wiping out the cache.

Does not work: - git+ssh://g...@github.com:bakodo/salt-django-example.git

Does work: - git+ssh://g...@github.com/bakodo/salt-django-example.git

Also works: - g...@github.com:bakodo/salt-django-example.git

So yes, I was mixing up the two styles thinking they would match. Maybe a warning in the docs for muppets like me would make sense...

[Andres Douglas]

I have advanced to the following issue level! W00t.

As I mentioned before I'm trying to store the pillar data in a separate git repo. 

It's in a private github repo, and the structure is 

github.com/bakodo/salt-pillar.git/

    pillar/

      top.sls

      apps/

         app1/

            settings.sls

            deploy_keys/

                id_rsa

                id_rsa.pub

            django/

                settings.sls

            [... more subdirectories] 

The top.sls in that repo looks like:

base:
    'id:app1.bakodo.jp':
        - match: grains
        - apps.app1.django.settings
        - apps.app1.settings

dev:
    'id:app1.bakodo.jp':
        - match: grains 
        - apps.app1.django.settings
        - apps.app1.settings
        - apps.app1.deploy_keys.id_rsa
        - apps.app1.deploy_keys.id_rsa.pub # is this going to be interpreted as a file extension and not as a subdirectory

is there a way to specify the gitfs_root?

Is the format for the top.sls correct?

I'm getting the following when running state.highstate, which makes me believe that the pillar data isn't being picked up or made available/accessible for the salt states 

app1.bakodo.jp:

    Data failed to compile:

----------

    Rendering SLS webserver failed, render error: Traceback (most recent call last):

  File "/usr/lib/pymodules/python2.7/salt/utils/templates.py", line 63, in render_tmpl

    output = render_str(tmplstr, context, tmplpath)

  File "/usr/lib/pymodules/python2.7/salt/utils/templates.py", line 116, in render_jinja_tmpl

    output = jinja_env.from_string(tmplstr).render(**context)

  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 894, in render

    return self.environment.handle_exception(exc_info, True)

  File "<template>", line 16, in top-level template code

UndefinedError: 'dict object' has no attribute 'git_repo'

Traceback (most recent call last):

  File "/usr/lib/pymodules/python2.7/salt/state.py", line 1904, in render_state

    rendered_sls=mods

  File "/usr/lib/pymodules/python2.7/salt/template.py", line 68, in compile_template

    ret = render(input_data, env, sls, **render_kwargs)

  File "/usr/lib/pymodules/python2.7/salt/renderers/jinja.py", line 41, in render

    tmp_data.get('data', 'Unknown render error in jinja renderer')

SaltRenderError: Traceback (most recent call last):

  File "/usr/lib/pymodules/python2.7/salt/utils/templates.py", line 63, in render_tmpl

    output = render_str(tmplstr, context, tmplpath)

  File "/usr/lib/pymodules/python2.7/salt/utils/templates.py", line 116, in render_jinja_tmpl

    output = jinja_env.from_string(tmplstr).render(**context)

  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 894, in render

    return self.environment.handle_exception(exc_info, True)

  File "<template>", line 16, in top-level template code

UndefinedError: 'dict object' has no attribute 'git_repo'

[Robert White]

Hi,
I wasn't sure if I should ask here or create a new entry.

I'm having a similar issue. I am however using a local install of gitorious so not sure if that is making any different?

My /etc/salt/master ext_pillar looks like

pillar_roots:

  base:
    - /srv/pillar

ext_pillar:
#  - git: master git+ssh://g...@gitorious.internal.co.uk/styx/interfaces.git
  - git: master g...@gitorious.internal.co.uk:styx/interfaces.git

I've tried both of the methods above but nether worked. The error is the same on both when running salt-master -l debug:
GitCommandError: 'git fetch' returned exit status 128: fatal: 'g...@gitorious.internalt.co.uk/styx/interfaces.git' does not appear to be a git repository

fatal: The remote end hung up unexpectedly

On both the : "co.uk:sty" changes "to co.uk/sty"

I've created a user on the repository and set up an unpassworded key.

Using the following code I've managed to get a pull from git on the same server with the same user
import git
import os
import shutil

DIR_NAME = "interface"
REMOTE_URL = "g...@gitorious.internal.co.uk:styx/interfaces.git"
if os.path.isdir(DIR_NAME):
    shutil.rmtree(DIR_NAME)

os.mkdir(DIR_NAME)
repo = git.Repo.init(DIR_NAME)

origin = repo.create_remote('origin', REMOTE_URL)
origin.fetch()
origin.pull(origin.refs[0].remote_head)

I'm fairly new to salt starting this week. Any help would be greatly appreciated.

[Robert White]

Hi,
Nevermind. I've used g...@gitorious.internal.co.uk:styx/interfaces.git and rm -r /var/salt/master/pillar_gitfs which means it is now working.

I'm still not sure what to put into the pillar file my top.sls
base:
  'server*':
    - interface

gives me
    _errors:
        - Specified SLS 'interface' in environment 'base' is not available on the salt master

but it is in /var/salt/master/pillar_gitfs/0/
I think its not the right environment...
I'll play around to see if I can work it out but if anyone has a quick pointer it would be greatly appreciated.

[Andres Douglas]

Robert, sorry not answering your question here.

To answer my own question, the root of the pillar has to be at the root of the directory.

Also: note to self - anything you include in the top.sls has to be a .sls file

You can't reference pillar variables from inside jinja templates that act as source files, it seems. Am I wrong?

It's unclear to me when/how gitfs or pillar data stored in an external git repo gets pulled/refreshed. It's only after wiping out the cache and restarting the master or minion that changes in the salt states or pillar data seem to take effect. This is pretty lame.

[Colton Myers]

Does it refresh when you explicitly call `salt ‘*’ saltutils.refresh_pillar`?

-- 
Colton Myers

[Andres Douglas]

How can I tell? I assume the pillar data and gitfs salt states are stored in the cache, in a git repo, but I'm not sure how to view the files in it. Any suggestions?

[Colton Myers]

I haven’t looked at the git pillar enough to know exactly where it’s caching, but you can use `salt ‘*’ pillar.items` to view the pillar and see if the new values are in there.

-- 
Colton Myers

[Andres Douglas]

Yup, cool. How about salt states stored in gitfs. Any idea about that?

[Colton Myers]

`cp.list_states` might be what you're looking for.  =)

--

Colton Myers

[Andres Douglas]

Sorry, I was explicit there. I was looking for a way to get the states stored in gitfs in the master to get refreshed - like a force update of state, ensuring the master pulls the latest from github, for example. Is that doable?

[Colton Myers]

`fileserver.update` should do the trick.

--

Colton Myers