Prepare for meeting tonight: take a look at your Search Heads' disk space...

[Gregg Woodcock]

It has become obvious that Search Heads, at least in some configuration of apps, require careful monitoring of disk usage.
I have found recently that our main general Search Head has something that is eating up all available disk space and has made it unusable for ad-hoc searches, which is a BIG PROBLEM!  I do not understand what is causing it but I do have an open ticket with support and will share once I do.  In the mean time, I suggest you all log in and take a look 2 places:

/opt/splunk/etc/users/
/opt/splunk/var/lib/

In the former, if you are not doing AD, you will surely find that users you have deleted long ago still have directories here and some of them are probably quite large.  I have reported this bug but this was many GB of wasted space for me.

The latter is more disturbing because it is constantly growing and does not appear to ever recede.  There are Splunk index DBs there (*.tsidx, *.db, etc) and I don't see why that should be.  I assume that one of the apps we installed did this (right now my leading suspect is SoS, but I am not done checking).  Run this command and you may be unpleasantly surprised:

cd ${SPLUNK_HOME}/var/lib/; du -sh ./* 2> /dev/null | sort -nr

[Gregg Woodcock]

As a followup, apparently, Splunk keeps the directories around BY DESIGN, just in case there is any cross-referencing of Knowledge Objects that the users owns referenced elsewhere so it is UP TO US to watch the bloat of this space.  Here is what Splunk has to say on the issue:
http://docs.splunk.com/Documentation/Splunk/6.0.1/Security/BestpracticeforremovinganLDAPuser