How long should initialization for state.highstate take?

[Kevin Dodge]

I just want to verify how long initialization should take when doing a state.highstate?

On my minion I run "salt-call state.highstate" and it takes 2 minutes before my state.highstate starts running.  Is this normal?

This is the output during those two minutes.

salt-call state.highstate

[INFO    ] Loaded configuration file: /etc/salt/minion

[INFO    ] Loading fresh modules for state activity

[INFO    ] Fetching file 'salt://top.sls'

[INFO    ] Fetching file 'salt://top.sls'

[INFO    ] Creating module dir '/var/cache/salt/extmods/modules'

[INFO    ] Syncing modules for environment 'base'

[INFO    ] Loading cache from salt://_modules, for base)

[INFO    ] Caching directory '_modules' for environment 'base'

[INFO    ] Creating module dir '/var/cache/salt/extmods/states'

[INFO    ] Syncing states for environment 'base'

[INFO    ] Loading cache from salt://_states, for base)

[INFO    ] Caching directory '_states' for environment 'base'

[INFO    ] Creating module dir '/var/cache/salt/extmods/grains'

[INFO    ] Syncing grains for environment 'base'

[INFO    ] Loading cache from salt://_grains, for base)

[INFO    ] Caching directory '_grains' for environment 'base'

[INFO    ] Creating module dir '/var/cache/salt/extmods/renderers'

[INFO    ] Syncing renderers for environment 'base'

[INFO    ] Loading cache from salt://_renderers, for base)

[INFO    ] Caching directory '_renderers' for environment 'base'

[INFO    ] Creating module dir '/var/cache/salt/extmods/returners'

[INFO    ] Syncing returners for environment 'base'

[INFO    ] Loading cache from salt://_returners, for base)

[INFO    ] Caching directory '_returners' for environment 'base'

[INFO    ] Loading fresh modules for state activity

[Thomas S Hatch]

No, it should take less than a second in almost all cases, do you have any custom modules or grains?

[Kevin Dodge]

I do not use grains except to grab the hostname.   As for custom modules, I assume that isn't talking about writing my own sls files right?   If not, then no, I have not written my own modules.

Running "salt 'server.*' test.ping"  is almost instant.

Running "salt 'server.*' state.show_highstate" takes about 22 seconds which returns 45 states.

I do have a bunch of jinja logic in my sls files and my pillar files.  

I do have other programs running on both the master and the minion and recognize if they are causing load that this would affect my time outcome, but right now there is almost no extra load.

Any ideas on why it would be going so slow or ideas on how to figure out why it is slow?

[Thomas S Hatch]

You might be calling out to modules in your sls files and the calls are taking a while

[Kevin Dodge]

I found out my problem.   In my sls setup I had this structure setup

build/files/build-repo

build-old/files/build-repo

The build-repo directory was a symbolic link that points to where all my successful builds are kept.  Under this directory there was around 100000 files/dirs (of which I am only directly using 10 in salt).   So I found that when I did a state.show_highstate it would take around 22 seconds.   Removing 1 of the symbolic links dropped it down to 12 s and dropping the second symbolic link reduced the time to 2 s.   

In the end I changed the symbolic links to just point to the 10 neccessary files and this made my state.show_highstate run VERY fast (2 seconds).

[Thomas S Hatch]

This says that some of the query functions could be optimized, I will keep this in mind

--
 
 

[Ryan Hewitt]

Can someone explain how I can fix this on my machine?  I recently decided to choose Salt Stack over my existing puppet config thanks to the guys at Scale, but it's so incredibly slow right now that I can't even use it on my dev cluster.

I'm using VM with ubuntu 12.04.

Also unrelated, but I had to set DEBIAN_FRONTEND to noninteractive to get aptitude working correctly.

[Thomas S Hatch]

Ryan,

What is in your file_roots? Are you trying to deploy an extremely large group of files?

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Ryan Hewitt]

I am, I suppose - I have about 15,236 files in a single folder that I need to be present on the system.  Do you recommend doing a once only decompression, or pulling it conditionally instead of placing it in file_root?

[Thomas S Hatch]

Yes, at least until we get some better optimization in there to handle very large file deployments like that

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

[David Ward]

This is interesting.

A highstate in my environment takes 3:30 minutes.
I thought this was normal.

This:
salt '*a-typical-host*' state.show_highstate|egrep '^    [a-z\/-]*:' | wc -l

Returns 73

Very minimal jinja logics. I think about 3-5 points where there is a {% if 'something' in grains[id] %} type logic.

The two directories this host would get its sls's and files from is 425k totalling 93 files and folders.

Is there a way to workout where the slow point is? Eyeball a salt-call state.highstate on the client? Not very scientific but all I can think of.

[Thomas S Hatch]

This is very strange, highstate should take less than a second to compile. what is the output from:

tree | wc -l

in the root of the file_roots?

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

[Kevin Dodge]

The way I debugged my issue was to remove items in the highstate trying to isolate the problem.   I eventually found that after removing everything from the highstate of the minion it was still taking a large amount of time.   At that point ( I made sure I had a backup of my salt files in git) I started pruning my /srv/salt and /srv/pillar directory structure seeing if that would give me any performance gains.   Quickly I found that removing a symbolic link to a large repository of files gave me huge gains.

Hopefully that might help.

[David Ward]

root@ops-master-ap-southeast-2b:/srv/production-salt# find -type f ! -path '*.git/*' | wc -l

637

root@ops-master-ap-southeast-2b:/srv/production-salt# tree |wc -l

809

Didn't know about this tree app. Nice. Thanks for the find :)

Do I have too many files?

I have the install files of an LDAP server in here..

tree base/ldap/ | wc -l

191

Not too much. It is not being used as yet but a process to get it to the client will need to be sorted in the near future.

[David Ward]

PS I am running the file_roots on the master out of memory to avoid any disk IO.

ie a git clone in /dev/shm/salt

I did this about a month or more ago but didn't notice any sizable speed ups in running highstates.

On Thursday, 28 March 2013 05:03:52 UTC+11, Thomas Hatch wrote:

[Thomas S Hatch]

Yes, those numbers are not too high, the highstate should still be very fast, are you doing anything exotic in pillar?

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

[David Ward]

Oh ok. What would be a more ideal number of files and folders be?

I have 1 var setup in pillar when I was playing with it. Not using it.

[Thomas S Hatch]

it should handle well into the tens of thousands of files just fine, so I doubt this is it, and with nothing in pillar to process I am more stumped.

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

[David Ward]

Thomas,

I have on my plans to do some optimising in the near future as this project heads to "business as usual".

I'll start with cleaning up files etc and see how I go and will post back when I have.

Thanks Thomas.

[David Ward]

I have cleaned up some files.

Tree returns:

143 directories, 455 files

and find:

144 directories, 461 files

A find for all files returns 5533 due to the .git/ folder.

So anyway, I thought this was helping, but now I am getting 3mins+ for a highstate run.

I will do a salt-call highstate on a client and video it to show the timing of all that happens.

[David Ward]

I have a 2:46min video for you Thomas.

Looking at it, there a huge stops at these lines:

[INFO    ] Executing command 'aptitude search "?name(^.+$) ?virtual ?reverse-provides(?installed)"' in directory '/root'


It runs about 15+ of them.

Thanks.

[David Ward]

Hey Thomas. Any thoughts on this? Want to see my video of a salt highstate run with many stoppages?

[David Ward]

Hey Thomas

Attached is a graph of CPU usage of a minion. highstate is being run from the master every 20mins, (on the hour, :20 and :40)
This graph correlates to these salt highstate runs.

I just thought this info might help...

[Thomas S Hatch]

Sorry, I have really missed this thread, it looks like the aptitude call is downright poison! We need to look into getting it out of there!

Thomas S. Hatch  |  Founder, CTO

5272 South College Drive, Suite 301 | Murray, UT 84123

tha...@saltstack.com | www.saltstack.com

[Corey Quinn]

Aptitude is heavy, particularly for checking if a package is present.  However, it's probably worth retaining for package installation, based on its superior dependency resolution behavior.

-- Corey

[David Ward]

So we are using aptitude purely because of virtual packages?

I noticed 

aptitude search "?name(^.+$) ?virtual ?reverse-provides(?installed)"

Is being called multiple times. Being so heavy, why would we not call it once, dump output to a temp file and work with that? Only if salt installs or removes a package during that run would you update that file.

Would mean possibly 1 call to that command.

Alternatively there is:

dctrl-tools

http://manpages.ubuntu.com/manpages/gutsy/man1/grep-dctrl.1.html

eg:

grep-available -F Provides -s Package mail-transport-agent

http://mummila.net/nuudelisoppa/2012/08/23/list-dpkg-packages-that-provide-a-virtual-package/

A look at the performance difference.....

\time -p aptitude search "?name(^.+$) ?virtual ?reverse-provides(?installed)"|grep mail-transport-agent

v   mail-transport-agent            -                                           

real 2.28

user 2.18

sys 0.08

david@david: ~# 

\time  -p grep-available -F Provides -s Package mail-transport-agent

real 0.06

user 0.06

sys 0.00

That is 38 times faster!

Or a virtual package I don't have:

\time  -p grep-available -F Provides -s Package zcav

Command exited with non-zero status 1

real 0.06

user 0.06

sys 0.00

You can work on the exit status from the command. Looking at the man page, there is lots of ways to search and manipulate the output.

Something like this will give you a full list of virtual packages on the system.

grep-available -F Provides -s Provides -e ".*" | cut -d' ' -f2-|sed 's/, /\n/g' | sort -u

[Oliver Kelly]

Hi All,

For anybody eagerly following this issue, it looks as if some progress has been made via https://github.com/saltstack/salt/pull/4625

Personally, It would be great to see this fix released soon, as I've been planning to roll out Salt to a fleet of Ubuntu 12.04 servers but the performance impact of aptitude trying to check 15-20 packages on each highstate call causes 2-3 mins of 100% CPU.

Cheers,

Oliver

[David Ward]

You're right Oliver.

Discussion on this issue continued on this revived old Issue on github

https://github.com/saltstack/salt/issues/2034

I am in the same boat as you Oliver and I am also very keen to get this change into production.