Smartpools file policies and tiering

[Jean-Baptiste Denis]

Hello everybody,

I've got a question about implementing smartpools policies to move file
around between different node pool.

I'm aware that policies design is fully coupled with your workflow and
data type, but I'd like some feedback of what you're doing on a day to
day basis regarding tiering within an Isilon cluster. I'm also certainly
not fully aware of all the power of smartpools, hence my questions may
sounds obvious. Please enlighten me ! =)

I've got a 1 PB cluster with a bunch of S, X and NL nodes. I've
statically identified some parts of the filesystem I want on specific
pool and wrote the adequate smartpools policies (path based) to achieve
that.

I'm now in a situation where my X nodes are almost full and my NL and S
nodes at about 40% of their capacity. I could identify more directories
to "manually" balance the data, but that's not satisfactory.

To build my policies, there are a bunch of different criteria I can use,
like create/access/modification/metadata change time.

I could express something like :

"move data that have been accessed less that 30 days to nodes S"

"move data that have not been accessed since 30 days AND less than 60
days to nodes X".

"move data that have not been accessed since 60 days to nodes NL"

What are your general feelings about those kinds of policies ?

What happens when a data pool is full and have spillover activated ? In
that case in which pool will the data go (it doesn't seem to be
configurable) ?

Is that a bad thing to have a full pool despite having spillover
activated ?

I'd like to be able to define data movement with the kind of threshold
to be able to express : "put hot data on the fastest pull until it reach
95% of utilization, then put them on the X pool, then on the NL". But it
doesn't seem to be possible. Maybe the Isilon technology is smart enough
to not have to want something like that ?

The " NEXT GENERATION STORAGE TIERING WITH EMC ISILON SMARTPOOLS"
whitepaper is quite interesting in this regard.

It shows data movement between pool based on time criteria. For example
: "Move files not accessed in 30 days move to disk pool 3". Do you
confirm that there is no implicit policy that will move my data again if
it is suddenly accessed 75 days after its last access : I've have to
explicitely express this within a policy (like I did in the three
previous policies I wrote) ?

Any advice ?

Jean-Baptiste

[Peter Serocka]

On Wed 5 Feb '14 md, at 17:50 st, Jean-Baptiste Denis <jbd...@pasteur.fr> wrote:

>
> It shows data movement between pool based on time criteria. For example
> : "Move files not accessed in 30 days move to disk pool 3". Do you
> confirm that there is no implicit policy that will move my data again if
> it is suddenly accessed 75 days after its last access : I've have to
> explicitely express this within a policy (like I did in the three
> previous policies I wrote) ?
>

Hello J-P:

There are no “implicit” policies (for regular stuff like /ifs/data),
but there exists always the final “default” policy which gets applied
when no other rules match. In your example, you will need to
set pool “3" as target pool for that default rule.
Otherwise, freshly accessed data will be moved to the
specified pool.

Obviously, this is more critical when using “access” time stamps
than for “modification” times, and least critical for “creation” times ;-)

Cheers

— Peter

[Chris Pepper]

Jean-Baptiste,

Time-based policies make a lot of sense but we don't use them. Instead we put everything into directories like /ifs/x, /ifs/x400, /ifs/108nl, and create subdirectories and shares under those per-pool top-level directories. The disadvantage is that we can run into problems when there is still plenty of space elsewhere, but the advantage is that performance is much simpler and more predictable. We also create quotas for file data and raw capacity (including snapshots & overhead) on these pool directories to monitor usage. Without these directories it gets very hard to see the user-level (no snapshots, no overhead) utilization of pools.

Part of the reason we do it that way is that Isilon still doesn't offer good visibility into where data is at the pool/tier level. If I have live data in /ifs/x and /ifs/nl, with SmartPool policies moving snapshots to /ifs/nl, there is no way to see how much snapshot data from either data set is in /ifs/nl or /ifs/x. After SmartPools runs (if nobody is deleting or modifying data), snapshot usage on /ifs/x will be 0 bytes, but as soon as someone deletes or modifies a files that will increase, and Isilon doesn't let us see how much -- aside from inspecting every single file with "isi get".

How much data has overflowed from where to where? No idea.

We eventually turned off the SmartPools policy that was sloshing snapshots onto NL storage because it was preventing any other jobs from running. It wasn't worthwhile for the (significant) capacity we were getting back.

Based on our experience I suggest you avoid frequent/large migration between pools. Certainly keep an eye on the SmartPools job with "isi job status" to see if it's blocking other jobs, and after changes watch "isi job history -j SmartPools" to see if it's taking a long time to complete.

I don't know where overflow goes, but I suspect it's the pool with the lowest utilization percentage. That's probably an unusual question to have -- I'm pretty sure the vast majority clusters have 1-2 pools rather than 3+.

Yes, it is bad to have full pools. I have been warned by several different reps about different utilization numbers in the range from 85% to 99%, and told not to worry about it by others. It is definite, though, that people with high utilization have had serious problems in the past. This is per-pool -- having extra space in a different pool doesn't really help with problems, aside from giving you a way to recover/overflow *after* you hit the problem.

At minimum you should be sure you have enough free space in each pool to lose 2 disks and recover data to remaining disks in that pool. Ideally you should have an entire node's worth of capacity free in each pool. That way if you lose an entire node (which does happen, for both hardware and software reasons) you can get back to a healthy state without installing a new node.

Don't trust Virtual Hot Spares -- in my experience they are never used, so to take advantage of them you just need to reduce your VHS number manually, which raises the total/free capacity numbers and lets OneFS slosh data around more efficiently.

Chris

[Saker Klippsten]

Hi

We currently use file path based policies. 

We have S200s and 400NL's, 

Our default Pool Policy is to Spill over from Performance to Nearline if Performance is full. This happens often since we run our cluster at 95%+ on a regular basis.. ( not advised ). When we clear up space. SmartPools policy will then move the data back to Performance on its next run..  We recently set a 5% VHD to give room for the filesystem to do it thing , where as before we did not set the VHD and we reached 100% all hell broke loose and no job policies ( smpartpools, snapshot delete , treedelete  ) would work.. to clear up space.. 

We do not use time based Policies because it does not suit our workflow as we need to guarantee a certain level of performance for our Artists for specific directories they are working out of.. but for other workflows where this is not critical,  it might... I like knowing for sure that X file is on X pool, at all times.  We also disable access time on cluster to give us more performance...

It gets a little tricky to figure out how much space is being used where. As there is still no easy way to know what is where. Its been at the top of my request since we first beta tested smartpools.

We want an "ls" command to show us what pool that folder is associated with when we do directory listings. We also want a du you command if we run on a folder to tell us how much and what pool, even just a % break out would be nice

.

We also want insightIQ to show the same info properly... when browsing the file tree etc.. 

Example

/ifs/prod/Film/xyz/ defaults to the S200 "Performance" Pool

/ifs/prod/Film/xyz/3D defaults to the S200 "Performance" Pool

/ifs/prod/Film/xyz/vault defaults to the "Nearline" Pool

/ifs/prod/Film/xyz/transfer defaults to the "Nearline" Pool

We then put Advisory Quotas on all these paths.

So we put one on 

/ifs/prod/Film/xyz/ this gives us the total for the entire XYZ project folder

We put child quotas on all the folders under it. We then have a little cron job script that runs every 5 min which outputs the isi quota list command spits that into python pickle file which we then have an interface that parses that file and displays the data visual using fusion charts 

We have "mix" which is the total of performance and nearline and then we have nearline and performance broken out..

This is how we know how much is living where.. 

We use the isi stat -q -d command to keep tabs on the total amount of storage space for the cluster

Here is a snapshot

Jean-Baptiste







--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jean-Baptiste Denis]

Hello everyone,

It seems that there is a lack of integrated tools for efficient/easy
data planning. But it's quite sure that having more than two pool type
will not help me here =)

I don't really know exactly what I'm going to do right now, but you gave
me some good leads.

Thank you very much for all this feedback, very appreciated.

Jean-Baptiste

[Saker Klippsten]

In 6.5 you did not have an option to specify where the data spills over to. In 7.0 you can select a specific pool that it can spill over to but you can only select a single pool for all pools to spillover to. I have it set to anywhere, so if any of my pools  fills up it will find a pool that has space and write to that. **It would be nice to have setting on a per pool basis so you can specify that some pools should spill, others should return a write error.   I will send that feature in..

Thanks

-s

On Wed, Feb 5, 2014 at 1:50 AM, Jean-Baptiste Denis <jbd...@pasteur.fr> wrote:

Jean-Baptiste

[Jean-Baptiste Denis]

I'm struggling with the general concept of tiering based on the last access
condition.

Basically, I would define a threshold for distinguish "hot" data from "cold"
data. Let's say data accessed less than 30 days ago are considered "hot", and
the rest are considered "cold". For performance reason, I want my hot data to be
on my faster and most expansive storage, and the cold data on my cheaper
one.

Of course, what defines "hot" data depends on your field or your workflow.
For example, on my 400 TB of "projects" data (used by hundreds of users), hot
data (defined before) represent 25% of the total storage capacity. I still don't
know the evolution pattern of this percentage, I'll need to wait a few month =)

Now some cold data become hot for whatever reason. Next time the tiering engine
fire up, this cold data will move to the faster disk array, potentially
involving huge amount of data transfer. But what if this data has just
been accessed again, just once. It would have been transfered for nothing.

Of course this is a simplistic point of view, and there is no silver bullet, but
it makes me think about the relevance of automatic tiering between hot and cold
data. The ability to be able to move data from slow to fast storage in advance
because I know that a specific project will need it is of course nice, but quite
cumbersome to deal with in a day to day basis.

Tell me if I'm wrong, but you cannot define smartpool policies on the CLI (at
least in 6.5.5.22) and you cannot run a specific policy also which add annoyance
to manually define smartpool policy for specific case.

From the feedback Saker and Chris gave (thank you again), I've got the feeling
that while time-based policies looks dynamic and cool, in practice it's not that
nice. The best compromise in terms of management, performance, number of bad
surprises and capacity planning seems to be defining static rules to place your
data.

Jean-Baptiste

[Chris Pepper]

Jean-Baptiste,

I agree (surprise!). I would like two things:

1) An option for ephemeral caching in a faster tier. Some data should *always* live on long-term (NL) storage, but making a copy on fast (X/SSD) storage would be cheaper if it didn't require recopy back to NL later.
2) An *augmentation* to atime to record what days a file was accessed. If it's been accessed 3 days this week or 10 days this month, it should probably be on fast storage. A single access day/time isn't really enough data to move a file to a faster tier -- SmartPools is just too expensive.

I suspect these capabilities are both far enough from the way OneFS works now that they will not happen, but together they would make tiering much more useful.

Chris

[Peter Serocka]

With a busy performance pool and a mostly idle archive pool,
it shouldn't hurt too much to access "certain portions" of old data
right from the archive pool:

The disk IOPS are there otherwise unused, and the CPUs
aren't loaded with NFS/SMB cycles. Global namespace acceleration
can help too, and can be applied selectively...

Therefore it's worthwhile checking wether tiering from
archive back to performance is really needed after all.

-- Peter

Peter Serocka
CAS-MPG Partner Institute for Computational Biology (PICB)
Shanghai Institutes for Biological Sciences (SIBS)
Chinese Academy of Sciences (CAS)
320 Yue Yang Rd, Shanghai 200031, China
pser...@picb.ac.cn

[Eugene Lipsky]

Hi Chris,

You mention that it's bad to have full pools and that you've received mixed messages from support but that people with high utilization had serious problems. Any chance you can elaborate on this? What kind of problems, suggestions for best way to deal with a situation where you are at a point of very high utilization?

We are currently at 99%+ on a 3-node s200 pool with plenty of space on a 3-node n400 pool and talking to support last night was initially told things like "get another s200 node" "move data between node pools" and "stop writing to the s200 pool"  Unfortunately it would take some time to procure an additional node, I asked how to move data between nodes (besides pool policies) and there was no answer. We can't stop writing to the cluster.

Just some additional info on how we got into this mess. Looks like our smart pool policies which are supposed to run nightly and move data older than x days from s200s to x400s haven't run since January. Kicking off a manual job yesterday got us 2% complete after 8+ hours with an ETA of 200+ hours and then support had us cancel that job after initially changing its priority from medium to high. Instead they suggested we run AutoBalance in high priority and let that complete. currently 13 hours in still on phase 2 of 5 with not eta. 

I'm not feeling a lot of confidence from support and am concerned of our worse case scenario is here if we run out of space completely.

[Peter Serocka]

Eugene:

just a quick tip and immediate means to prevent the worst case from happening:
make sure you have “spill over” enabled in Smartpools,
so the cluster will at least not reject the incoming data.

AutoBalance will not “balance between tiers”;
but is this what support told you?

— Peter

> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

[Jean-Baptiste Denis]

> We are currently at 99%+ on a 3-node s200 pool with plenty of space on a 3-node
> n400 pool and talking to support last night was initially told things like "get
> another s200 node" "move data between node pools" and "stop writing to the s200
> pool" Unfortunately it would take some time to procure an additional node, I
> asked how to move data between nodes (besides pool policies) and there was no
> answer. We can't stop writing to the cluster.

We were in the same case a few months ago, combined with an upgrade from 6 to 7.
That was crazy and the support was not helpful at all.

Could you activate the spillover to the n400 pool ? In that case, it would allow
the write to occur. and while not ideal, the production kept going.

Once you manage to configure the spillover, you have multiple possibilities :

- let the smartpool move data around. It can take a very long time (more than
one month in our case) because of a side effect of the full pool

- use isi set -R -g return --diskpool ...

- identify big old files, write a smartpool to match them and move them to the
n400 pool by using "isi smartpools apply --path" on each one. That what I did,
in parallel using xargs (achiving 120 MBytes/s instead of 30 MBytes/s)

You can have a look at my post "Isilon-Users OneFS 6.5 -> 7.0 upgrade : after
the battle" in which I explore those solutions. Some users also provide very
valuable feedback.

Good luck !

Jean-Baptiste

[Eugene Lipsky]

Yep we have spill over enabled and yes support told us to run autobalacnce and actually had me stop the smartpools job :(

[Peter Serocka]

I see. Perhaps check the support emails for the fine print which
may have an e-mail address of the junior supporter’s supervisor(!)
for feedback.

[Eugene Lipsky]

It's not in an email but it is in the case notes. I've just updated the case asking to be reassigned.

@Jean-Baptiste - thanks for your feedback, I'll look for your thread. I'd like to go back to smartpools policy as well just want to make sure I hear back from support first.

[Eugene Lipsky]

Just a quick update on my issue. Stopped the AutoBalance job and re-started the SmartPool job at priority 1. Took 3 days to complete and the S200 pool went from 99.6% full to a little over 50%. I've also had a very large snapshot to deal with, 30TB of data was deleted and that took a few days to clean up.

I did get to speak to a more senior support engineer on Friday and it looks like the reason our SmartPool jobs haven't run since January is due to all the snapshots that we have going constantly. Snapshot delete jobs have a higher priority over SmartPool jobs and so SmartPool jobs never get a chance to complete. It was recommended that we modify SmartPool jobs from priority 3 to priority 2. I've done this and a new job kicked off after that long 3 day old job completed. This job at priority 2 has now been running for 24 hours and is 40% complete. I'm not really sure why it would take so long for the new job to complete as in theory all/most of the data that was on the S200 tier has moved to the NL tier and each subsequent job going forward should really be moving files that haven't been accessed in 30 days. 

[Andrew Stack]

Just out of curiosity...do you have what is commonly referred to as 'overlapping snapshots'?  In other words one path (i.g. /ifs/really_important_stuff) with two snapshot policies. The reason I ask is that this is considered a bad practice within Isilon OneFS.  It can produce long snapdelete efforts.

Regards,

Andrew Stack

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Andrew Stack

Sr. Storage Administrator

Genentech

Cell - 650.867.5524

[Eugene Lipsky]

I do have this. I have hourly snapshots that we keep for 2 weeks, then daily that we keep for 4 weeks and weekly that we keep for 8 weeks. The reason I didn't want to just keep hourly snapshots for 8 weeks is because of the amount of snapshots that would generate.

[Andrew Stack]

Yeah, I learned the hard way...this is in brief a problem for OneFs and frowned upon.  I encourage you to speak to Isilon about this but if you want the cliff notes:  One path - One policy like below.

In your instance you have 3 overlapping policies.  What this means is that snapdelete must compare the hourly, daily and weekly looking for blocks to reclaim.  At a minimum you have increased your job duration by 3x.  If you have long SnapDelete periods or your disk pools fill to near capacity you impact the cluster performance negatively while it tries to reconcile reclaimed blocks. 

Best to use a single policy as much as possible and work with your customer SLA's to reduce overall retention or to better classify your SLA by path as exhibited above.  I hope this helps.

Regards,

Andrew Stack

[Alistair Stewart]

You can schedule SnapshotDelete to run only at certain times of day in order to let other jobs run uninterrupted for longer. It doesn't only run to delete user snapshots: sometimes when you delete a large file, OneFS will move the file into a snapshot to speed up the delete operation and SnapshotDelete will be run in the background to clear up afterwards. This is why you might see SnapshotDelete run even when you don't have a SnapshotIQ license.

Al...

--
Regards

Al…

[Eugene Lipsky]

Hmm okay that kind of sucks. We don't have specific SLAs in place so it should be relatively easy to change our policies but it helps tremendously being able to go to more point in time intervals as well as going back as far as possible without having to resort to backup restores so it made sense to create hourly, daily, weekly policies. In fact we were doing snapshots every 15 minutes earlier in the year but moved away from those.

So what about snapshots that are taken for SyncIQ replication? Netbackup jobs? They would also be overlapping. 

[Andrew Stack]

I found this difficult to stomach at first as well, especially if you admin Netapp.  But if you think about it a bit there are boundaries (volumes/aggr) within Netapp.  Isilon does not limit you in this sense but rather in others like Snapshots.  

SyncIQ basically only uses two snapshots for comparison and is separate from your normal snapshots (see below).  

There really is no impact as the compare is just between these two snaps.

SIQ-694053663a3be9a40b37c8bfd5a57733-latest	2014-09-10 19:00:05 PDT	2015-09-10 19:00:05 PDT	/ifs/p01	2.88 GB	0%	0%
SIQ-694053663a3be9a40b37c8bfd5a57733-new	2014-09-13 20:59:03 PDT	2015-09-13 20:59:03 PDT	/ifs/p01	1.09 TB	0%	0%

  

As far as backups or other 3rd party options that generate snaps...that's a different discussion altogether but the same principles apply...don't overlap.

Regards,

Andrew

[Peter Serocka]

On 2014 Sep 16. md, at 23:33 st, Eugene Lipsky wrote:

> . I'm not really sure why it would take so long for the new job to complete as in theory all/most of the data that was on the S200 tier has moved to the NL tier and each subsequent job going forward should really be moving files that haven't been accessed in 30 days.

Once finished, you can compare the SmartPools reports
for past and recent jobs, obtained via

isi job hist -v -j JOBNUM

Maybe it's "just" a huge number of (small) files
that got moved around. Would also explain the long
SnapshotDelete runs -- it's all about the metadata...

-- Peter

[Peter Serocka]

On 2014 Sep 17. md, at 04:45 st, Andrew Stack wrote:

I found this difficult to stomach at first as well, especially if you admin Netapp.  But if you think about it a bit there are boundaries (volumes/aggr) within Netapp.  Isilon does not limit you in this sense but rather in others like Snapshots.  

SyncIQ basically only uses two snapshots for comparison and is separate from your normal snapshots (see below).  

I don't think they are technically seperate...

these just happen to be small and adjacent in order,

so deleting them is in fact not a big deal, just as you found.

There really is no impact as the compare is just between these two snaps.

SIQ-694053663a3be9a40b37c8bfd5a57733-latest	2014-09-10 19:00:05 PDT	2015-09-10 19:00:05 PDT	/ifs/p01	2.88 GB	0%	0%
SIQ-694053663a3be9a40b37c8bfd5a57733-new	2014-09-13 20:59:03 PDT	2015-09-13 20:59:03 PDT	/ifs/p01	1.09 TB	0%	0%

  

As far as backups or other 3rd party options that generate snaps...that's a different discussion altogether but the same principles apply...don't overlap.

Recent OneFS docs explain that the overhead for deleting

overlapping snaps is up to about two-fold of the regular time.

That should help in making an informed decision

on overlapping.

Cleary, two-fold overhead *may* break the camel's back, but not

necessarily so if deletions take only a few hours per day.

(for example over here: wether it's 1-2 hours or 2-4 hours 

per night, doesn't matter). Deleting hourly snapshots not

right at expiry time, but together in a single batch at night 

keeps overhead somewhat lower than 2x, as far as I can observe.

[Eugene Lipsky]

Hey Peter,

I think you are talking about https://www.emc.com/collateral/TechnicalDocument/docu50220.pdf (starting on page 174) was reading it last night and it looks like it may be okay to have overlapping policies after all. Though for the time being we've decided to to consolidate ours for now.

Regarding the isi job hist -v -j JOBNUM command. Is there an updated command in 7.1? hist doesn't seem to be valid.

[Peter Serocka]

Eugene

Yep -- that doc is more elobarated on the examples than I remembered ;-) 

Right now I don't have a SmartPools license for 7.1,

but the SmartPools statistics should show up with either

isi job events list --verbose --job-id <JOBNUM>     (the new "history" syntax)

or

isi job reports list --job-id <JOBNUM>           (job reports as new concept in 7.1)

How the SmartPools statistics look like in ** 7.0 **:

09/11 01:36:16  SmartPools[26307]          Phase 1: end lin policy update 

        FILEPOLICY JOB REPORT

        Elapsed time:        51951 seconds

        {'default' :

          {'Policy Number': -2,

          'Files matched': {'head':2150, 'snapshot': 0},

          'Directories matched': {'head':15, 'snapshot': 689},

          'ADS containers matched': {'head':0, 'snapshot': 0},

          'ADS streams matched': {'head':0, 'snapshot': 0},

          'Access changes skipped': 0,

          'Protection changes skipped': 0,

          'File creation templates matched': 15,

          },

        'system':

          {'Policy Number': -1,

          'Files matched': {'head':80581, 'snapshot': 0},

          'Directories matched': {'head':53698, 'snapshot': 0},

          'ADS containers matched': {'head':0, 'snapshot': 0},

          'ADS streams matched': {'head':0, 'snapshot': 0},

          'Access changes skipped': 2,

          'Protection changes skipped': 0,

          'File creation templates matched': 53698,

          },

        'x200 metadata-accel':

          {'Policy Number': 0,

          'Files matched': {'head':25563018, 'snapshot': 149971},

          'Directories matched': {'head':2687615, 'snapshot': 35433},

          'ADS containers matched': {'head':0, 'snapshot': 0},

          'ADS streams matched': {'head':0, 'snapshot': 0},

          'Access changes skipped': 0,

          'Protection changes skipped': 0,

          'File creation templates matched': 2687615,

          },

          ... etc ...

      }

Cheers

-- Peter

[Eugene Lipsky]

Thanks Peter. Looks like the command in 7.1 is:

isi job reports view --id=JOBNUMBER

You can also view it via web interface.

Also it looks like our smartpool jobs finally caught up and we've managed to go down from 99.6% to 68% utilization on the S200 pool based on files older than 30 days now all residing on the NL pool.