Monitoring pool usage

[Jean-Baptiste Denis]

Hi everybody,

I feel quite stupid here because I can't find a way besides parsing the output
of "isi status -q -d" to monitor the usage of my different pools. I've looked
over the snmp MIB and the REST API without luck.

Am I missing something here ?

Jean-Baptiste

[Peter Serocka]

sysctl efs.bam.disk_pool_db

also needs some parsing, but at least it
reports blocks rather than rounded TBs,
so smaller changes can be detected earlier.

(and of course there are
isi statistics drive -nall --long --noconversion
and
isi statistics query -s node.ifs.bytes.used
but these require parsing AND computation...)

Disk pools are not "containers" like LUNs,
but rather sign posts that only direct the
placement of new data and restripes
at the time of writing. This is why
they can be changed on the fly without
affecting access to present data ;-)
But this also makes disk pools just kind of
"second class citizens" in computer science speak.
Which might explain why they don't appear on
their on rights in too many places (SNMP, API) yet(?).

Cheers

-- Peter

> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

Peter Serocka
CAS-MPG Partner Institute for Computational Biology (PICB)
Shanghai Institutes for Biological Sciences (SIBS)
Chinese Academy of Sciences (CAS)
320 Yue Yang Rd, Shanghai 200031, China
pser...@picb.ac.cn

[Jean-Baptiste Denis]

On 04/25/2014 08:15 AM, Peter Serocka wrote:
> sysctl efs.bam.disk_pool_db

OK. I've got some difficulties to understand what I'm seeing.

There is the notion of "PHYSICAL POOLS". When I look at the drives/drivemap
entrie of each section, it seems that PHYSICAL POOLS designate subset of drive
among all nodes of a given class (NL, X or S).

For example :

...
{


id=5

name=x400_96tb_1.6tb-ssd_24gb-ram:5
parent=x400_96tb_1.6tb-ssd_24gb-ram

drivemap={
2 => { 13, 14, 15, 32, 33 },
3 => { 13, 14, 15, 32, 33 },
4 => { 13, 14, 15, 32, 33 },
5 => { 13, 14, 15, 32, 33 },
20 => { 13, 14, 15, 32, 33 },
}
...

2, 3, 4, 5, 20 correspond to the device ID of my 5 X nodes. I don't see how to
extract the information I want. Maybe there was a misunderstanding here. By pool
I meant the set of X nodes, NL nodes or S nodes. From what I can't see, I could
reconstruct this information by parsing.

Maybe I should have used "Node pools" instead of "Disks pools", sorry about that.

The "POOL GROUPS" section could have been what I'm looking for, but there is no
usage report.

By the way, what kind of format is this ? It's not json. How do you parse it
efficiently ?

> Disk pools are not "containers" like LUNs,
> but rather sign posts that only direct the
> placement of new data and restripes
> at the time of writing. This is why
> they can be changed on the fly without
> affecting access to present data ;-)
> But this also makes disk pools just kind of
> "second class citizens" in computer science speak.
> Which might explain why they don't appear on
> their on rights in too many places (SNMP, API) yet(?).

OK, thank you for this heads up.

[Peter Serocka]

Seems there have been changes in 7.0...
In 6.5 there is one section per pool like below,
but no notion of "PHYSICAL POOLs":

{
id=4
...
default_policy=+2:1 w18
name=x200_performance
...
drives={ ... }
drivemap={
....
}

...
total_blocks=4063432878
free_blocks=2501439554
....
total_ssd_blocks=72242367
free_ssd_blocks=17625069
...
avail_blocks=2220613635
avail_ssd_blocks=17625069
...
}

pretty straightforward.

(I'll check it on the virtual 7.0/7.1 nodes later.)

parsing: home-made I'd suggest, perl or awk or the like.

-- P.

[Jean-Baptiste Denis]

On 04/25/2014 10:17 AM, Peter Serocka wrote:
> Seems there have been changes in 7.0...
> In 6.5 there is one section per pool like below,
> but no notion of "PHYSICAL POOLs":

> ...
> pretty straightforward.

Looks like, yes. But you still have the drive/drivemap entries. Are they
containing all entries for a node pool or just a subset. If this is a subset,
I understand you still have to add up total_blocks to have the usage of each
node pool.

> (I'll check it on the virtual 7.0/7.1 nodes later.)

Please find the sysctl output if this is of any help before firing up you VM
nodes. It's certain I can reconstruct the information I want, but I don't want
to build a specific parser for such a trivial information =)

EMC support was not able to give me a satisfying answer.

> parsing: home-made I'd suggest, perl or awk or the like.

Obviously, yes !

Thank you for your help/

[Jean-Baptiste Denis]

I was looking at the python code used for the webui and stumble upon interesting
things in :

/usr/local/www/webkit/webui/ChartSPPhysicalUsage.py

and

/usr/share/papi/scripts/1/diskpool/diskpools/diskpools_to_json.py

I'm not gonna copy and paste the content for obvious licensing reason, but the
output of the last python contains all that I need :

python /usr/share/papi/scripts/1/diskpool/diskpools/diskpools_to_json.py
[{"disk_usage": {"available": 37967087501312, "total": 483843316776960, "used":
439461871255552}, "entry_id": 1, "name": "x400_96tb_1.6tb-ssd_24gb-ram"},
{"disk_usage": {"available": 65445317869568, "total": 213982500618240, "used":
146413177634816}, "entry_id": 2, "name": "s200_22tb_48gb-ram"}, {"disk_usage":
{"available": 192818716934144, "total": 535421646766080, "used":
336192738328576}, "entry_id": 3, "name": "n400_108tb_12gb-ram"}]

Now, I'm not sure if I'm used to use this as end-user =)

[Peter Serocka]

> /usr/share/papi/scripts/1/diskpool/diskpools/diskpools_to_json.py

Nice…! so let’s hope it will show up in the
user-visible Platform API (PAPI).

That python script also exists in 7.1 (but not in 6.5).

The sysctl I mentioned for 6.5 is indeed
different in 7.0 and 7.1 (but consistent in these releases).

Cheers

— Peter

[Rob Peglar]

Folks,

The OneFS Platform API (PAPI) contains a nice section for 'storagepools'.  Remember you can always parse the tree by doing a

GET /platform/1/storagepool?describe?list?all

to see what is available.  In fact, if you execute this at the /platform level, you see the entire PAPI tree.

Storagepools are a handy way to look at consumption, in fact quite preferable to the internal-only diskpools.  Here's an example I executed from my Mac to my virtual Isilon (7.1) using curl.

rpeglar$ curl -k -u "root:<redacted>" -X "GET /platform/1/storagepool/storagepools" https://192.168.137.100:8080

{

"storagepools" : 

[

{

"id" : 2,

"lnns" : [ 1, 2, 3, 4 ],

"manual" : false,

"name" : "iq_vmware",

"protection_policy" : "+2:1",

"type" : "nodepool",

"usage" : 

{

"avail_bytes" : "317306822656",

"avail_ssd_bytes" : "0",

"balanced" : false,

"free_bytes" : "403288186880",

"free_ssd_bytes" : "0",

"total_bytes" : "407634837504",

"total_ssd_bytes" : "0",

"virtual_hot_spare_bytes" : "85981364224"

}

}

],

"total" : 1

}

This same cluster shows 295 GB free in the GUI display, which corresponds to the decimal base 10 'avail_bytes' count above.  The GUI display shows base 2 values; to get to the avail_bytes figure, it's 295*1024*1024*1024, truncated to the nearest base 2 GB.  If you do the math you get 316,753,838,080 instead of 317,306,822,656 but again it's a truncated figure.  So you actually have 295.515 GB base 2 available over the four LNNs which comprise this pool.

Hope that helps.  Please use the PAPI and comment...it's a very handy tool.

Cheers

Rob

------------------------------------------

[Peter Serocka]

Thanks Rob!

It seems storage pools are only in the 7.1 PAPI
(according to the PAPI docs).

Any idea when 7.1. might reach the “recommended” status?

Cheers

— Peter

[Jason Davis]

Ah very good... I'll get this integrated with my Dashing Dashboards here in the office :) We just happen to have a somewhat large OneFS 7.1 cluster that's been popular with Isilon Engineering as of late.

[Peter Serocka]

On Tue 29 Apr '14 md, at 00:02 st, Jason Davis <scr...@gmail.com> wrote:

> Ah very good... I'll get this integrated with my Dashing Dashboards here in the office :) We just happen to have a somewhat large OneFS 7.1 cluster that's been popular with Isilon Engineering as of late.

What kind of issues (other than SMB/AD and IB if I remember correctly)?

Thanks

— Peter

>

[Jason Davis]

IB but different from what is publicly known about the "Blackhole" bug that was addressed with the FW update on the Intel/Qlogic 12800 series IB switches. Issue likely not just specific to OneFS 7.x.

Beyond that, we've documented some issues with SmartConnect which have been around for awhile. Again, not just specific to OneFS 7.x.

Afraid I can't go into too much detail just yet.

-- Peter

[Peter Serocka]

Got it, thanks again.

— P.

[Rob Peglar]

Hi Pete,

Thanks - as I understand it, the 7.1 stream is nearing recommendation status.  That designation turns out to be all about field exposure, some folks call this "stick time".  Isilon Support makes the call.  Having said that, if you are on the 7.1 stream already, the 7.1.0.2 MR is what one should aim at.  That release (GA) happened recently, April 7th.  The original 7.1.0.0 released on October 30, 2013. So the 7.1 stream has about 6 months of stick time.

By far the best way to proceed is to work closely with your Isilon folks on the ground.  

Cheers

Rob

[Peter Serocka]

Sounds great, thanks Rob!

BTW, I was able to update (a copy of) my 7.0.2 virtual

nodes to 7.1.0.2 using the *regular* 7.1.0.2 

installation/update package.

Pretty remarkable that this works; and it might by useful

for testing updates in more complex (AD, ...) environment.

(No guarantee, but at least a sandbox test.)

Cheers

-- Peter