Some basic configs with gitfs

500 views
Skip to first unread message

monosij...@gmail.com

unread,
Mar 13, 2015, 12:36:02 AM3/13/15
to salt-...@googlegroups.com

In working off some examples of Learning SaltStack:

Trying to get GitFS to work by converting from a file_roots backend to a gitfs-remotes backend.

...
The file_roots was defined as below in master and worked great:

file_roots:
  base:
    - /srv/salt
  webserver:
    - /srv/web

Just trying to understand how to get and if the following set up in just moving the above to a git repo will work?

...
and put the roots in a directory in the repo called: roots.set.A/{pillar,salt,web} - same structure as previously served from /srv/{pillar,salt,web}

I extended the above master config as:

file_roots:
  base:
    - /salt
  webserver:
    - /web

fileserver_backend:
  - git

gitfs_remotes:

gitfs_mountpoint: salt://roots.set.A

...
Does not work.

Tried quite a few things but no luck.

I am trying to see if this set up will work to tell gitfs that while gitfs_roots is configured to the git repo, ...
I want to mount roots.set.A as the point from serving the state tree? ie this directory:

So later I can change to a different path - such as roots.set.B.

...
Or is this totally wrong? And I would have to go with different branches or even different repos if I want to try different configurations of state trees?

I am also trying with raw content as: https://raw.githubusercontent.com/mduttaroy-devops - no luck there either.

I understand I can combine state trees from different repos, but I am not there yet.

...
Your help in trying to see if I can get this to work will be much appreciated.

Thank you.

Mono

--------------------------- ERRORS ----------------------------
provisionDEBI:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or external nodes data matches found
     Started: 
    Duration: 
     Changes:   

Summary
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
provisionCENT:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or external nodes data matches found
     Started: 
    Duration: 
     Changes:   

Summary
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
provisionUBUN:
----------
          ID: states
    Function: no.None
      Result: False
     Comment: No Top file or external nodes data matches found
     Started: 
    Duration: 
     Changes:   

Summary
------------
Succeeded: 0

Colton Myers

unread,
Mar 23, 2015, 5:55:02 PM3/23/15
to salt-...@googlegroups.com
> file_roots:
> base:
> - /srv/salt
> webserver:
> - /srv/web
>
> Just trying to understand how to get and if the following set up in just moving the above to a git repo will work?
>
> ...
> I moved it into a git repo: https://github.com/mduttaroy-devops/configs-provisioner-saltstack
> and put the roots in a directory in the repo called: roots.set.A/{pillar,salt,web} - same structure as previously served from /srv/{pillar,salt,web}

I think this is where you're running into trouble. I inspected that repo and it looks like you're trying to put multiple environments plus your pillar data into a single branch in GitFS, but this isn't how GitFS works.

GitFS assumes that each branch is its own environment, and that the `master` branch is the `base` environment. So to set it up the same way you had it with `file_roots`, you should have a `master` branch with just the contents of your `base` environment, and a `webserver` branch with just the contents of your `webserver` environment.

Pillar must be managed completely separately, in a different repo. Look at the documentation for external pillars and `git_pillar` for information there.

>
> I extended the above master config as:
>
> file_roots:
> base:
> - /salt
> webserver:
> - /web

When you're using `git` as your only fileserver_backend, the `file_roots` configuration no longer matters, `file_roots` is only used when you're using the `roots` backend.

>
> fileserver_backend:
> - git
>
> gitfs_remotes:
> - https://github.com/mduttaroy-devops/configs-provisioner-saltstack.git
>
> gitfs_mountpoint: salt://roots.set.A

To show an example from above, if you wanted to get your `base` environment working with your current config, just change gitfs_mountpoint like this:

gitfs_mountpoint: salt://roots.set.A/salt

However, now your webserver stuff won't work at all. That needs to go in a separate `webserver` branch.

Hopefully this is helpful!

--
Colton Myers
Platform Engineer, SaltStack
@basepi on Github/Twitter/IRC

signature.asc

monosij...@gmail.com

unread,
Mar 23, 2015, 10:07:15 PM3/23/15
to salt-...@googlegroups.com
Hi Colton -

I understand regards the branches but this config with gitfs_roots should work right?

gitfs_roots should run the tree from the directory in the repo w/o needing branch is that not so?

Then I should not need to use gitfs_mountpoint - which I understand needs the different branches?

Can i put the pillar data under gitfs_roots as well? Are there some examples on how to run your setup from the book using gitfs_roots and gitfs_mountpoint including pillar?

If you would like I can work on it and write it up fro you to be included with the book.

---------------------------------------------------------------
fileserver_backend:
  - git

gitfs_remotes:

gitfs_roots: roots.set.A

---------------------------------------------------------------------------------------------
Btw nothing seems to work for me right now including file_roots. I am generating ssh keys using ssh-keygen and the master is complaining regarding the keys! Strange.


2015-03-23 12:54:52,256 [rest_tornado     ][ERROR   ][1048] ImportError! No module named tornado
2015-03-23 13:01:08,883 [salt.master      ][ERROR   ][1258] Corrupt public key "/etc/salt/pki/master/minions/provisionUBUN": no start line

What are these errors I am getting on the git/develop branch? Ubuntu 14.10. Are these errors I need to log? The stable does not even work for me. Nothing gets returned as I mentioned I was trying dual master.


Your help would be highly appreciated in sorting through this.

...
Thank you.

Mono

monosij...@gmail.com

unread,
Mar 23, 2015, 10:13:44 PM3/23/15
to salt-...@googlegroups.com
And Colton - Much appreciated for taking a look through my repo.

Hope my questions are not too moot in the various things you all are working through.

I hope to be able to contribute to Salt sometime.

Thanks again.

Mono

Colton Myers

unread,
Apr 2, 2015, 1:42:41 PM4/2/15
to salt-...@googlegroups.com
I understand regards the branches but this config with gitfs_roots should work right?

gitfs_roots should run the tree from the directory in the repo w/o needing branch is that not so?

Then I should not need to use gitfs_mountpoint - which I understand needs the different branches?

First, it's gitfs_root, not gitfs_roots. That's probably your primary problem. 

gitfs_root and gitfs_mountpoint are quite different. The first cuts off the beginning of paths inside of the repo. Thus if you want to serve files out of a subdirectory of the repo, but have them appear as if they're in the root of the fileserver (salt://) you use that option.

On the other hand, gitfs_mountpoint makes files within a repo appear to the minion as if they're in a subdirectory, even if they're not. So a file in the root of your repo will have the mountpoint path prepended when the minion sees it. 

So yes, in the config below you are using gitfs_root correctly (once you remove the trailing s)
 

Can i put the pillar data under gitfs_roots as well? Are there some examples on how to run your setup from the book using gitfs_roots and gitfs_mountpoint including pillar?

I don't recommend putting your pillar data into the same repo, because it's generally more sensitive data, and if you misconfigure your gitfs_root you could expose sensitive data to minions which might not be allowed to see that data. I recommend using a secondary git repo exclusively for pillar data, with the git_pillar external pillar.
 

If you would like I can work on it and write it up fro you to be included with the book.

---------------------------------------------------------------
fileserver_backend:
  - git

gitfs_remotes:

gitfs_roots: roots.set.A

---------------------------------------------------------------------------------------------
Btw nothing seems to work for me right now including file_roots. I am generating ssh keys using ssh-keygen and the master is complaining regarding the keys! Strange.


2015-03-23 12:54:52,256 [rest_tornado     ][ERROR   ][1048] ImportError! No module named tornado
2015-03-23 13:01:08,883 [salt.master      ][ERROR   ][1258] Corrupt public key "/etc/salt/pki/master/minions/provisionUBUN": no start line


This sounds like perhaps you changed the key when you transferred it to your master. Or maybe the permissions on the key are wrong? Though I feel like that would cause a more relevant error...
 
What are these errors I am getting on the git/develop branch? Ubuntu 14.10. Are these errors I need to log? The stable does not even work for me. Nothing gets returned as I mentioned I was trying dual master.

Anyway, I hope that helps you get closer. Sorry for my delayed reply.

--
Colton Myers

Monosij Dutta-Roy

unread,
Apr 6, 2015, 2:39:22 PM4/6/15
to salt-...@googlegroups.com
Hi Colton -

Thank you for your response. No problem with the delay.

I will post separately that - it is great to have the latest RC, everything with multi-master from local filesystem works great! Thanks for all the great work and help to all..

...
Regards this gitfs_remote - still having issues even after I changed as per:

fileserver_backend:
  - git
gitfs_remotes:
gitfs_root: config.roots/roots.set.A

The directory on git: config.roots/roots.set.A has the salt and web directories as per the file_roots setup.

i am trying to still incorporate from your book to moving the roots to be served from git and based on documentation here:
https://salt.readthedocs.org/en/v0.17.0/topics/tutorials/gitfs.html#serving-from-a-subdirectory

I would assume that to serve as per your file_roots setup:
file_roots:
  base:
    - /salt
  webserver:
    - /web


I assume I would need something like:

gitfs_root:
  base:
    - config.roots/roots.set.A/salt
  webserver:
    - config.roots/roots.set.A/web

Thus essentially I should have same setup as mentioned in file_roots so as to properly indicate which is base and then which is webserver. Is there a good way to do this?

Or would the setup be different? Am I thinking about this all wrong?

Also, then it should be gitfs_roots with an 's' and not gitfs_root? Not that I am complaining about the singular right now, but you all use plural from pillar as well.

...
While this repo and subdirs is at this GitHub location:

And then I would want to serve the pillars from this subdir in the git repo:
configs.pillars/pillars.set.A

Would this be possible to set up in master under pillar_roots, I have this currently:

pillar_roots:
  base: 
    - /srv/saltstack.pillars.d/pillar

-------------------------------------------------------------------------------
Finally (not related to above discussion) ...

I have the same setup at this other git location, with only the structure in your book that I hope I can use gitfs_mountpoint from, i.e. if I cannot use gitfs_mountpoint from the first repo given the many subdirs:

-------------------------------------------------------------------------------
And now ... here is the error I am getting on salt master logs.
I had noticed on your error reports from 2013 regards https - which is what I have serving out of. But it seems all those were fixed.

2015-04-06 17:00:14,764 [salt.loaded.int.fileserver.gitfs][ERROR   ][15906] Exception 'Unsupported URL protocol' caught while fetching gitfs remote https://github.com/mduttaroy-devops/configs-provisioner-saltstack.git
2015-04-06 17:01:15,155 [salt.loaded.int.fileserver.gitfs][ERROR   ][15906] Exception 'Unsupported URL protocol' caught while fetching gitfs remote https://github.com/mduttaroy-devops/configs-provisioner-saltstack.git

...
And here is the output from /var/cache/salt/master:

vagrant@saltstackMaster02:~$ sudo cat /var/cache/salt/master/gitfs/
40474ff176d44629e8592c09b7b63d26/ envs.p                            remote_map.txt                    
vagrant@saltstackMaster02:~$ sudo cat /var/cache/salt/master/gitfs/remote_map.txt 
# gitfs_remote map as of 06 Apr 2015 16:44:06.784412
vagrant@saltstackMaster02:~$ sudo cat /var/cache/salt/master/gitfs/envs.p 
vagrant@saltstackMaster02:~$ 

Thank you again for all your help.

Mono



--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/382sUN6o_OM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Colton Myers

unread,
Apr 20, 2015, 1:58:24 PM4/20/15
to salt-...@googlegroups.com
So I think you're misunderstanding how gitfs works with environments. You don't have to define each environment by hand, it uses the git branches. So if you have a master branch, that branch will be used as the `base` environment, and if you have a webserver branch, that branch will be used as the `webserver` environment. You can't serve two different environments out of the same branch using gitfs.


--
Colton Myers
Platform Engineer, SaltStack
@basepi on Twitter/Github/IRC

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages