Smart Failing Nodes - How long to plan for?

[gtjones]

I support a 600TB Isilon cluster running 6.5.5.7 of OneFS. We have a three node disk pool with 72NL nodes. We will be adding nine X400 nodes and plan to smart fail the 72NL nodes and using them in a separate cluster. The 72NL nodes are full.

Does anybody know how long it might take to smart fail three full 72NL nodes? Keep in mind that the smallest number of nodes in a diskpool is three.

Thanks,

Greg

[Ethan Richard]

the X400s are what size disk? and are they all in the cluster before the smartfails will take place? 

-ethan

[Keith Nargi]

I would recommend getting a temporary smart pools license and set up disk pool of the x400s and then create a file pool policy to move all of your existing data over to your new x400s. Once the smatpools job finishes you should then smart fail out each node one at a time. The nodes should have no data on them but just for safe keeping you should do the smart fail.

If you do a smart fail on all of those nodes it's probably going to take you a long time since you really should only do one at a time to keep your protection in tact. Which means each node that you smart fail out will cause a restripe and you'll have to do it 3 times.

Keith

[Blake Golliher]

I do these smartpools migrations myself, works very well (but we
noticed if you have a LOT of smaller files the process takes longer,
then if you have fewer bigger files). And with nodes down to just a
few gig's smartfails take a lot less time to complete.

-Blake

[Greg Jones]

My cluster currently has three diskpools (10X nodes, 36NL nodes, and 72NL nodes). We'll be adding a fourth pool (X400 nodes) then removing all the 72NLs. The 72NL nodes contain 437m files with 377M less than 128K. The X400s will be added before we fail the 72NLs.

Greg

[Jerry Uanino]

On a 10 node cluster of 9T nodes, I ran about a week per node, but I ran extremely low priority and paused a few times during the day.

My users were extremely sensitive to performance impacts and wanted to not run at high priority and pause the smartfail on certain intervals where there was concern.

I think the fastest I could do was 12-24 hours a node at the time.

What we did was add 14 new nodes into the cluster when we did then let autobalance run.... this took quite a while because we didn't care how long it took (low priority).

That saved a lot of movement for smartfail because the large number of new nodes brought the total amount of data on each node much lower ne

At the time we also modified a few things in XML config files to prevent the new nodes from "bullying" the old nodes.  I'm not sure if this is still the recommended case, but the SEs usually have pretty good procedures around this.

[Jerry Uanino]

smart pools! nice. Didn't think of that.

[gtjones]

Responding to myself.

We removed three 72NL nodes from our existing 22 node cluster. Here's how we did it and how long it took.

1. Used Smartpools to move as much data off the nodes as possible (216 Hours)

2. Eeach node still had 16TB of data, we started to smartfail the 1st node (120 Hours)

3. Once the 1st node completed, we started the 2nd, then the 3rd (48 hours, 24 hours)

4. We had a drive fail during the smartpools job that took 18 hours to recover from.

The above numbers are not precise figures. All jobs ran as low priority. The entire time to execute all of these was roughly 420 hours.

[John Williams]

gt - Joining the game late here. I am investigating the pros and cons of doing exactly what you did. Yea its slow but Im happy and relieved to hear that it worked.
This thread is excellent.

Q1) Just out of curiosity, did you confirm where the 16 TB of data that remained on each node (at the start of step 1) was from? Was this simply data that was missed in the initial SmartPools configuration?
Q2) You mentiond that the 72NL nodes contain 437m files with 377M less than 128K. Do you have any information regarding the storage efficiency of those files? In the limited Isilon engagements that I have worked on, it was my understanding to steer Isilon clear of applications with a lot of sub-300KB files. My understanding is that files under 128K are 2X mirrored (in 8KB block units, not by the full 128KB stripe unit)... and also that a file of greater than 1 stripe unit (128KB) will actually pre-allocate enough blocks to fill a whole protection group. I can see evidence of that in the "isi get -DD" command when viewing the protection group details.
Q3) 420 hours? Any thoughts, now that you finished, on how you might have been able to speed that time up?

Thanks... J

[Greg Jones]

Q1) I'm don't know what the data was. We got it down far enough that it felt safe to fail the nodes. I'm not sure how you can tell which files are on a particular node without doing an isi get command across a whole bunch of files.

Q2)We do believe files less than 128K are mirrored.I haven't heard the < 300KB recommendation, but I can see what you're talking about. If a file is 130K in size it uses 256K of storage. That's nearly 50% wasted.

Q3)There's the obvious, don't have 100's of millions of files and make sure you have SSDs. 

Because the nodes were full and the pool consisted of 3 nodes (the minimum), I don't think there's a different way to do this. Had the nodes not been full, we probably would have done a smartfail without the smartpools job first.

[John Williams]

GT - Actually, a 130k file with 4+2/2 protection consumes 390KB.  With 16+4 protection, a 130k file consumes 650KB

The Isilon docs say that a file under 128KB is mirrored. But, more specifically, what really happens is that all of the parity protection group units (stripe units) of the first protection group need to be allocated with the first protection group unit of data, to attain the desired resiliency of the file's protection level.

Use the "du -h" command to verify...

The storage efficiency isnt as bad if the files are on average over 500 KB, and preferably 2 MB or more.