Real disk usage

[Robert Soeting]

Who can tell me how I can get the "real" disk usage of a directory?

Commands like du take forever on 60TB and I need to know the disk usage with out protection.

Like what you can see in insightIQ but than commandline.

[Peter Serocka]

Quotas do it for us.
Can be queried in real-time inclusive/exclusive *protection*
("overhead"),
after inital scan has finished.
Scan is usually must faster than du as it runs in parallel.
Choice to include/exclude *snaphots* must be decided when defining
a quota domain, though.

[Jerry Uanino]

You can use du --apparent-size in linux.  This will, however, still take longer than using quotas.

Add accounting only dirs and remove them as necessary.

isi quota create --dir --path=/whatever --accounting (I believe from the top of my head).

isi quota list --dir --path=/whatever or 

isi quota list --dir --recurse-path=/toplevel/ if you have a bunch of nested quotas.

[Chris Pepper]

Use "du -A" in OneFS (must be first arg). On Linux, use "du --apparent-size". It is pretty fast if you have SSDs. If you need immediate answers, creat quotas in advance (requires license). But quotas have compatibility bugs with 'mv', SyncIQ, and TreeDelete jobs.

Chris

[Robert Soeting]

I guess quota's would be the best solution "IF" we had the license. We don't yet.

Du -Ash will provide the information needed, but takes for ever (even with SSD as meta).

Running it with time as I type this message to get an idea about really how long.

[Robert Soeting]

55TB 

du -Ash *  389.32s user 22328.24s system 48% cpu 13:04:47.88 total

13 hours.

[Robert Soeting]

I have been playing around with the FSA output.

Does anyone have experience with that?

sqlite3 -column -header /ifs/.ifsvar/modules/fsa/pub/latest/results.db 'select * from disk_usage where lin is 4305454022 and parent is 4304930131'

lin         parent      name        dir_cnt     file_cnt    ads_cnt     other_cnt   log_size_sum    phys_size_sum   log_size_sum_overflow

----------  ----------  ----------  ----------  ----------  ----------  ----------  --------------  --------------  ---------------------

4305454022  4304930131  eneco       8688736     89714494    3833098     0           61944742556892  91137921008128  0

This should allow me to read directory disk usage. 

Somewhere this should say 55TB.

[Peter Serocka]

On Sun 25 Nov '12 md, at 18:13 st, Robert Soeting wrote:

> I have been playing around with the FSA output.
> Does anyone have experience with that?

Not actually, but I like the idea...
Are you really running the expensive InsightIQ license
(i.e. is the FSA results.db up-to-date??), but no SmartQuotas?
Interesting combination.

Here is another idea for the command line:

If you suspect you have a ***HUGE*** number of t i n y
files collected in a comparatively *small* number
of directories which slow down the du,
you can identify those directories pretty *quickly*
with "find" -- you just need to prevent find from
descending into those directories.

find "toplevel-dir" -type d -size +100k -prune

The "size" of a directory gives only a vague hint
about the number of files in it, but it should do.
The key twist is "-prune", which suppresses descending
into found directories.

You then might "deal" with those directories separately
somehow before running du - any specific knowledge about their
purpose or the data workflow might help.
(Worked very well for us when "tuning" for NDMP backups:
TreeDelete became our new friend, along with
"mv" and "tar"/"zip" of course.)

[Peter Serocka]

On Sun 25 Nov '12 md, at 18:03 st, Robert Soeting wrote:

> 55TB
> du -Ash * 389.32s user 22328.24s system 48% cpu 13:04:47.88 total
> 13 hours.

Here is some wierd result: du -Ash seems to be unnecessary slow:

isilon-2# time find . | wc -l
103206
find . 0.16s user 1.94s system 96% cpu 2.186 total


isilon-2# time ls -R . | wc -l
119147
ls -z -R . 0.29s user 1.93s system 95% cpu 2.331 total


isilon-2# time ls -Rli . | wc -l
127119
ls -z -Rli . 1.44s user 5.12s system 82% cpu 7.942 total
wc -l 0.09s user 0.00s system 1% cpu 7.941 total



isilon-2# time du -sh .
27G .
du -sh . 0.11s user 2.03s system 95% cpu 2.250 total


isilon-2# time du -Ash .
16G .
du -Ash . 0.35s user 13.12s system 68% cpu 19.617 total

(On a lowly loaded X200 with meta on SSD,
but all runs actually repeated and therefore from cache!)

So we observe:
find
ls -R
du -sh
are pretty quick.

ls -Rli
somewhat slower, reads more metadata than the others -> explained.

du -Ash
drastically slower than du -sh (how come?)
and still significantly slower than ls -Rli,
which basically does provide all relevant data
(apparent file sizes) which du -Ash would need.

WTH...?





find . 0.17s user 1.95s system 95% cpu 2.219 tota

[Jason Davis]

If you are just looking for general disk usage then you can query SNMP and get that stuff. I use SNMP with Zenoss and I get the utilization and graph it.

[Robert Soeting]

Can you check what the number is for "actual disk usage"

[Jason Davis]

Yup.

The OIDs you are looking for are:

Used  .1.3.6.1.4.1.12124.1.3.2.0

Available  .1.3.6.1.4.1.12124.1.3.3.0

# snmpwalk -v2c -c public isilonnode1.local .1.3.6.1.4.1.12124.1.3.2.0

SNMPv2-SMI::enterprises.12124.1.3.2.0 = Counter64: 10314260029440

# snmpwalk -v2c -c public isilonnode1.local .1.3.6.1.4.1.12124.1.3.3.0

SNMPv2-SMI::enterprises.12124.1.3.3.0 = Counter64: 291972478722048

So output is in bytes, so I am using 9.38076 TB of storage. So this is actual disk usage with overhead. Same applies with Available space.

[Jerry Uanino]

but is that for the whole cluster, or can you target directories?  I assume you can't target directories without have quotas on them, right?

[Jason Davis]

Correct, this is global. If you are looking for specific directories then you will be using du or pony up for a InsightIQ license. 

I'd imagine that you could extend SNMP so you could query specific directories via SNMP gets but this sounds like a lot of customization.

[Peter Serocka]

SNMP will give you the quotas (as licensed and configured)

without further customization, just through the shipped ISILON-MIB.txt.

Peter Serocka

CAS-MPG Partner Institute for Computational Biology (PICB)

Shanghai Institutes for Biological Sciences (SIBS)

Chinese Academy of Sciences (CAS)

320 Yue Yang Rd, Shanghai 200031, China

pser...@picb.ac.cn

[Peter Serocka]

            

I found an outdated InsightIQ results.db on our cluster

and querying the disk_usage table goes at 

nearly 100,000 records(dir-entries)/second.

Scanning the live /ifs with du, the rate is

a few 1000 dir-entries/second...

It can be assumed that InsightIQ (FSAnalyse job) creates

the results.db much faster than du does its scans, 

probably due to smarter file system access and 

cluster-parallel execution.

In total, it might be in fact worthwile to consider crafting

some nice db queries...

[Blake Golliher]

If the query's respond back fast enough, they might be hooked up to a
snmp extension pretty easily. That'd be quite handy! What else is in
that DB, like performance stuff. Nice.

[Peter Serocka]

On 2012 Nov 27. md, at 11:35 st, Blake Golliher wrote:

> If the query's respond back fast enough, they might be hooked up to a
> snmp extension pretty easily. That'd be quite handy! What else is in
> that DB, like performance stuff. Nice.
>

;-)

isilon-3# sqlite3 /ifs/.ifsvar/modules/fsa/pub/latest/results.db .tables
disk_usage stats_disk_pool_ctime_path_ext
list_top_n_dirs_by_atime stats_disk_pool_ctime_phys_size
list_top_n_dirs_by_btime stats_disk_pool_log_size_path_ext
list_top_n_dirs_by_ctime stats_disk_pool_path_ext_phys_size
list_top_n_files_by_atime stats_path_squash_atime_attribute
list_top_n_files_by_btime stats_path_squash_atime_log_size
list_top_n_files_by_ctime stats_path_squash_atime_path_ext
list_top_n_files_by_log_size stats_path_squash_atime_phys_size
list_top_n_files_by_phys_size stats_path_squash_attribute_ctime
stats_all stats_path_squash_attribute_log_size
stats_disk_pool_atime_attribute stats_path_squash_attribute_path_ext
stats_disk_pool_atime_log_size stats_path_squash_attribute_phys_size
stats_disk_pool_atime_path_ext stats_path_squash_ctime_log_size
stats_disk_pool_atime_phys_size stats_path_squash_ctime_path_ext
stats_disk_pool_attribute_ctime stats_path_squash_ctime_phys_size
stats_disk_pool_attribute_log_size stats_path_squash_log_size_path_ext
stats_disk_pool_attribute_path_ext stats_path_squash_path_ext_phys_size
stats_disk_pool_attribute_phys_size system_properties
stats_disk_pool_ctime_log_size system_squash_map

You could also check the full ".schema"...

[Erik Weiman]

Just keep in mind that you will need to have the InsightIQ module licensed in order for the FSA job to be allowed to run on the cluster.

If this job cannot run, the db you are looking at does not get updated.

[Peter Serocka]

Sure. I am aware that our result.db came from an InsightIQ

evaluation license when we started our cluster. 

Decided then to go with SmartQuotas and free "isi statistics ...".

[Jerry Uanino]

What would be nice is if the insightIQ tools had CLI tools.  This would abstract us more advanced admins away from future insightIQ changes. Tapping into the data from the fsanalyze job is an awesome idea.  We love insightIQ here, but it would be really cool to be able to push the data into our other monitoring tools as well.  Neat stuff guys!

I smell an "isi insightiq --query" request. ;-)

[Robert Soeting]

I get the point that if you want a nice graphic tool, you need to pay for it.

But do you need a license to run the FSA job?

[Brendan Scherer]

As far as OneFS is concerned, you do. Just tested on my virtual 6.5.4.12 and a few others. I can't think of a legitimate way of getting around it without licensing InsightIQ.

[Peter Serocka]

Waht

On 2012 Nov 29. md, at 17:56 st, Brendan Scherer wrote:

> As far as OneFS is concerned, you do. Just tested on my virtual 6.5.4.12 and a few others. I can't think of a legitimate way of getting around it without licensing InsightIQ.

From the 7.0 release notes:
Platform API — The Isilon OneFS Platform API provides a RESTful programmatic interface that enables you to automate cluster management, configuration, and monitoring. The OneFS Platform API integrates with the OneFS role-based authentication functionality for increased security and fine-grained control. The earlier version of the OneFS Platform API, which required a license, is no longer supported. The new OneFS Platform API does not require a license.


Nice step into the right direction - no license required any more for the API.

Why not integrating FSA with the API, and just licensing the GUI on top of it...

(The API gives access to quotas for example - if quotas have a valid license...)

> --

[Yoga Damal]

Pete, is there a way to look in to the data present the tables listed by command "sqlite3 /ifs/.ifsvar/modules/fsa/pub/latest/results.db .tables"

[Damal]

The values from log_size_sum  and  phys_size_sum, is that number of blocks ?

[Peter Serocka]

Hi Damal,

that's Bytes! And Robert's example also answers your other 

question, how to read the data... ;-)

Cheers

-- Peter

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Damal]

I ran the command and i am looking to find out space occupied by specific directory and tried it on ifs, but the values shown here are not correct. Any suggestions please ?

Isilon-1# sqlite3 -column -header /ifs/.ifsvar/modules/fsa/pub/latest/results.db 'select * from disk_usage' | grep ifs

4294967297  2           .ifsvar     1168        2563        0           1           18485886120     74152168960     0

4295032832  4294967297  ifs         1           2           0           0           115             91136           0

4295884800  4295294976  ifs         1           0           0           0           0               4096            0

4305287935  4305287932  gifs        1           23          0           0           18655           635392          0

4305339559  4306613762  gifs        1           24          0           0           19491           661504          0

4306111910  4306111902  ifservices  1           5           0           0           35183           181760          0

4306401359  4306401358  calgifs     1           0           0           0           0               2048            0

[Peter Serocka]

Going to re-engineer the details of InsightIQ? ;-)

If it helps, you can get the full path for
a LIN (first column) with something like

isi get -L $(printf %x 4295032832)



— P.


On Mon 14 Apr '14 md, at 22:16 st, Damal <d.y...@gmail.com> wrote:
>
> 4295032832

[Damal]

Thanks Pete

[Brian Clark]

I am a little late to the game, but how do I get the LIN from the full path?

I can run isi get -Dd /ifs/some/path | grep LIN

However it returns with something that looks like 1:289e:5ce0 

Is that number that you are referencing the LIN or something else?

thanks,

Brian

[Brian Clark]

Nevermind... I feel like an idiot.

I just needed to convert from Hex to decimal.

(1:289e:5ce0 = 1289e5ce0)

Decimal = 4976434400

Good stuff guys. 

thanks,

Brian