isi get -D - An explanation

[gtjones]

All

We have a large file that's being written to a seven node Isilon cluster. We expect the file to live across all the nodes but for some reason it only resides on two. Isilon said in training that files are broken into 128K stripes and the stripes distributed across all nodes. This file would certainly qualify for being distributed across more nodes than two.

My question is this, what do these fields mean especially the 1st line and device list below for data and meta? If you know why this might be landing on two nodes instead of seven, that would be great too.

Thanks,

Greg

Here's the isi get command for one such file.

default    6+1 concurrency on    UTF-8         A20073._2.fq      <3,11,24898042871808:512>, <4,0,64294125322240:512> ct: 1352147791 rt: 0

*************************************************

* IFS inode: [ 3,11,3039311872:512, 4,0,7848403968:512 ]

*************************************************

*

*  Inode Version:      4

*  Dir Version:        2

*  Inode Revision:     7699

*  Inode Mirror Count: 2

*  Recovered Flag:     0

*  Recovered Groups:   0

*  Link Count:         1

*  Size:               10941756381

*  Mode:               33188

*  Flags:              0xe0

*  File Data Blocks:   1335664

*  LIN:                1:c3cd:4b1c

*  Restripe Cursor:    -1

*  Abort Cursor:       -1

*  Last Access:        1352147791.545872103

*  Last Modified:      1352161382.125009642

*  Last Inode Change:  1352161382.309300035

*  Create Time:        1352147791

*  Rename Time:        0

*  Write Caching:      Enabled

*  Parent Lin          1:67fc:00bd

*  Parent Hash:        588546

*  Habanero Inode:     True

*  Snapshot IDs:       None

*  Last Paint ID:          9841

*  Domain IDs:         None

*  LIN needs repair:   False

*  Manually Manage:

*       Access         True

*       Protection     False

*  Protection Policy:  Diskpool default

*  Current Protection: 6+1

*  Future Protection:  0x

*  Disk pools:         policy ssd(4) -> target ssd(4)

*  SSD Strategy:       metadata

*  SSD Status:         complete

*  Data Width Device List:

*  (d: 12, b: 222624, i: no, drv(250): 00000ff6

*  (d:  4, b: 222608, i: no, drv(250): 00000ff6

*  (d: 10, b: 222608, i: no, drv(250): 00000ff6

*  (d:  3, b: 222608, i: no, drv(250): 00000ff6

*  (d:  2, b: 222624, i: no, drv(250): 00006bf4

*  (d:  5, b: 222608, i: no, drv(250): 00000ff6

*  (d: 11, b: 222608, i: no, drv(250): 00000ff6

*  Meta Width Device List:

*  (d:  5, b: 330, i: no, drv(250): 00000fff

*  (d: 12, b: 317, i: no, drv(250): 00000fff

*  (d:  3, b: 328, i: no, drv(250): 00000fff

*  (d: 10, b: 327, i: no, drv(250): 00001ff7

*  (d:  4, b: 308, i: no, drv(250): 00000fff

*  (d:  2, b: 342, i: no, drv(250): 00007bfc

*  (d: 11, b: 308, i: no, drv(250): 00000fff

*

*  File Data (16 bytes):

*    Metatree Depth: 3

*    2,12,11034361856:8192

*    3,10,948181499904:8192

*

*  Dynamic Attributes (226 bytes):

*

        ATTRIBUTE                OFFSET SIZE

        Quota Governance         0      42

        Layout info              42     9

        Last snapshot paint time 51     9

        Disk pool policy         60     9

        Disk pool target         69     5

        Logical Size             74     9

        v5 Meta wdl              83     66

        v5 Data wdl              149    66

        Access time              215    9

        Isilon flags             224    2

Live snapids: { 9843, 9845 }

[Curt Harpold]

Greg,

Looks to me like you're only listing the detailed metadata information. At 6+1 (7 nodes, N+1), there would be two metadata mirrors, and they reside on two nodes.

To list detailed information including the layout of file data, use "isi get -DD".

Cheers...

......curt

[Jeff Hughes]

Curt is right. The first line is only listing the mirrors of the
inode and not the data itself. For most metadata within OneFS,
mirroring is used with an equivalent resiliency to the data protection
scheme. In this case, a 6+1 FEC layout would have an equivalent
mirroring of 2x (both can survive one node/drive failure). Using the
'isi get -DD' command will print out a listing of all the protection
groups that make up the file data layout. There you would see the 6+1
layout groups that you're expecting.

-Jeff

[gtjones]

Curt and Jeff, 

Thanks for the clarification. Given what you've said about the isi get -D showing you information about the metadata, why does the metadata exist on HDD instead of the SSD that are in the node? The diskpool policies put all metadata on SSDs. The output seems to indicate that the metdata is on HDDs.

I did run the -DD option and do see the data groups, however I'm not sure how to interpret this. Due to file size, I can't paste the entire -DD output here, so I'll include a small snippet. I assume these numbers represent the node, block size, offset, etc. Can someone elaborate on what these mean?

PROTECTION GROUPS

        lbn 0: 6+1

                4,9,797053558784:8192#16

                10,6,791138009088:8192#16

                3,9,789871329280:8192#2

                3,9,789871624192:8192#14

                2,7,795761754112:8192#16

                5,7,787711934464:8192#16

                11,8,782850064384:8192#16

                12,5,791113777152:8192#16

        lbn 96: 6+1

                12,5,791114432512:8192#16

                10,6,791138140160:8192#16

                3,9,789872001024:8192#16

                2,7,795761885184:8192#16

                5,7,787712065536:8192#16

                11,8,782850195456:8192#16

                4,9,797053820928:8192#16

        lbn 192: 6+1

                12,5,791114825728:8192#16

                4,9,797059457024:8192#16

                3,9,789872132096:8192#16

                2,7,795762147328:8192#16

                5,7,788119707648:8192#16

                11,8,782850326528:8192#16

                10,6,791138271232:8192#16

[Jeff Hughes]

On Tue, Nov 6, 2012 at 11:52 AM, gtjones <gtjo...@gmail.com> wrote:
> Curt and Jeff,
>
> Thanks for the clarification. Given what you've said about the isi get -D
> showing you information about the metadata, why does the metadata exist on
> HDD instead of the SSD that are in the node? The diskpool policies put all
> metadata on SSDs. The output seems to indicate that the metdata is on HDDs.

There's a lot of factors here that determine where the metadata gets
put. You're probably best to work with support if you think something
is ending up not on SSD that you think should be based on the
configuration.

> I did run the -DD option and do see the data groups, however I'm not sure
> how to interpret this. Due to file size, I can't paste the entire -DD output
> here, so I'll include a small snippet. I assume these numbers represent the
> node, block size, offset, etc. Can someone elaborate on what these mean?

It is in the form of <node,drive,offset:blocksize#blockrun>

[J. Lasser]

On Tue, Nov 6, 2012 at 12:07 PM, Jeff Hughes <jef...@gmail.com> wrote:

On Tue, Nov 6, 2012 at 11:52 AM, gtjones <gtjo...@gmail.com> wrote:
> Curt and Jeff,
>
> Thanks for the clarification. Given what you've said about the isi get -D
> showing you information about the metadata, why does the metadata exist on
> HDD instead of the SSD that are in the node? The diskpool policies put all
> metadata on SSDs. The output seems to indicate that the metdata is on HDDs.

There's a lot of factors here that determine where the metadata gets
put.  You're probably best to work with support if you think something
is ending up not on SSD that you think should be based on the
configuration.

Note that with most SSD configurations (including GNA), only one copy of the metadata is written to the SSD.

--
Jon Lasser                     j...@lasser.org                      206-326-0614
. . . and the walls became the world all around . . .  (Maurice Sendak)