Implementing SmartPool Tiering - Is there any good reporting?

42 views
Skip to first unread message

Kenneth Van Kley

unread,
Mar 5, 2024, 2:11:42 PMMar 5
to Isilon Technical User Group
We're about to implement a mix of F600 and A300 nodes with Smartpools directing files that have access times older than "x" years to the A300's.

Is there any good reporting that will show file counts per pool?  How do I know in which pool a file is located?

Luc Simard

unread,
Mar 5, 2024, 2:44:10 PMMar 5
to isilon-u...@googlegroups.com
Have you tried InsightIQ.

You can also prioritize your storage decisions and data movements with DiskoverData


Cheers.

/ls

Luc Simard
Google Personal Email:
- simard.j...@gmail.com
Mobile gVox: 415-501-0438
Washington State, USA




On Mar 5, 2024, at 11:11, Kenneth Van Kley <kjv...@gmail.com> wrote:

We're about to implement a mix of F600 and A300 nodes with Smartpools directing files that have access times older than "x" years to the A300's.

Is there any good reporting that will show file counts per pool?  How do I know in which pool a file is located?

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isilon-user-group/11df966e-b1a3-4c3c-a378-184db99d41ecn%40googlegroups.com.

Kenneth Van Kley

unread,
Mar 6, 2024, 12:38:55 PMMar 6
to Isilon Technical User Group
InsightIQ is ok, but it relies on the file report which only runs weekly (since it takes over 3 days to run).

Luc Simard

unread,
Mar 6, 2024, 2:29:46 PMMar 6
to isilon-u...@googlegroups.com
The newer version have been addressed such that the deltas for runtime are smaller, internally we had it running in test lab 1000+ times per day without issues, Alas you need OneFS 8.2+, which is not longer compatible with Gen3/Gen4 gear.

DiskoverData’s tool can be accessed as community edition, or with Trial, Trial to Buy (PoC). It’s platform agnostic and can scan all your platform including OneFS , it needs a couple of VM instances to scan and it can move data , either manually or with policies.


You will need at least 3 VM CentOS nodes (works with some other distros too), works best with Elastic Search 7.14 (hard requirements) , Python 3.11. See recommendations, ES demands proper spec’ing or RAM allocation per usage.



That is likely your best plan if you don’t want to do that by hand. Alternatively, I would poll your internal customers and find out what can be done”off-lined” off Isilon and live onto newer or better positioned for access for your data to alleviate the load on the system and free up some space.

Sorry to say you are only putting a bandaid on the larger issue. The gear is EoL, you could look into H700/H7000 and aggregate multiple workflows, or if the data is colder, then A300/A3000. They also carry their pros/cons, up to you to determine the best fit for your org.






HPE just announced newer solutions with a breath of new features, something to consider also perhaps. There are a number of solutions.




Cheers.

/ls

Luc Simard
Google Personal Email:
- simard.j...@gmail.com
Mobile gVox: 415-501-0438
Washington State, USA



Saker Klippsten

unread,
Mar 6, 2024, 2:34:23 PMMar 6
to isilon-u...@googlegroups.com
is DataIQ Dead? Have not checked in on that in a bit.
-S


To view this discussion on the web visit https://groups.google.com/d/msgid/isilon-user-group/D76BAFB9-B00A-4A96-A524-160C5D64BFDA%40gmail.com.
dellemc-powerscale-f900-00015lf.psd

Luc Simard

unread,
Mar 6, 2024, 2:52:11 PMMar 6
to isilon-u...@googlegroups.com
No more dev time for a while now (~2y), for effect DiskoverData replaces it functionally speaking, can be purchased directly from them. DiskoverData was created by them same great team as the source project that lead to DataIQ.

Lots of plugin and features.



Cheers.

/ls

Luc Simard
Google Personal Email:
- simard.j...@gmail.com
Mobile gVox: 415-501-0438
Washington State, USA



On Mar 6, 2024, at 11:34, Saker Klippsten <sak...@gmail.com> wrote:

is DataIQ Dead? Have not checked in on that in a bit.
-S


On Wed, Mar 6, 2024 at 11:29 AM Luc Simard <simard.j...@gmail.com> wrote:
The newer version have been addressed such that the deltas for runtime are smaller, internally we had it running in test lab 1000+ times per day without issues, Alas you need OneFS 8.2+, which is not longer compatible with Gen3/Gen4 gear.

DiskoverData’s tool can be accessed as community edition, or with Trial, Trial to Buy (PoC). It’s platform agnostic and can scan all your platform including OneFS , it needs a couple of VM instances to scan and it can move data , either manually or with policies.


You will need at least 3 VM CentOS nodes (works with some other distros too), works best with Elastic Search 7.14 (hard requirements) , Python 3.11. See recommendations, ES demands proper spec’ing or RAM allocation per usage.



That is likely your best plan if you don’t want to do that by hand. Alternatively, I would poll your internal customers and find out what can be done”off-lined” off Isilon and live onto newer or better positioned for access for your data to alleviate the load on the system and free up some space.

Sorry to say you are only putting a bandaid on the larger issue. The gear is EoL, you could look into H700/H7000 and aggregate multiple workflows, or if the data is colder, then A300/A3000. They also carry their pros/cons, up to you to determine the best fit for your org.



Erik Weiman

unread,
Mar 7, 2024, 12:04:43 AMMar 7
to isilon-u...@googlegroups.com
The runtime for FSAnalyze is heavily dependent upon your cluster size and hardware configuration and what data you store (zillions of tiny files vs millions of large files). Add on top of that how often the job is interrupted, code version of OneFS is also relevant as there have been improvements made along the way. 
You can also leverage IndexUpdate jobs to improve FSA runtime. 
I see some clusters complete FSA in less than a couple hours but others can take a week (or more). 
--
Erik Weiman 
Sent from my iPhone 

On Mar 6, 2024, at 11:38 AM, Kenneth Van Kley <kjv...@gmail.com> wrote:

InsightIQ is ok, but it relies on the file report which only runs weekly (since it takes over 3 days to run).

bob flynn

unread,
Mar 7, 2024, 5:51:25 PMMar 7
to isilon-u...@googlegroups.com
I had a gander at disk over website. Stìll unsure as to the value add of moving from insightiq where yòu get cluster performance as well as capacity data where data disk seems to only cover capacity and file find down typically to a depth of 5 subdirs. 

We complete a fsa nightly and I use that to create grafite data to plot via grafana for end users data usage dashboards. Am I missing to make Diskover a pursusive case ?

To view this discussion on the web visit https://groups.google.com/d/msgid/isilon-user-group/D76BAFB9-B00A-4A96-A524-160C5D64BFDA%40gmail.com.
698e4b54-392d-40c0-982b-9774381ab8d3.png
storagereview-hpe-greenlake-file-storage.jpeg
dellemc-powerscale-f900-00015lf.psd

Luc Simard

unread,
Mar 7, 2024, 6:04:46 PMMar 7
to isilon-u...@googlegroups.com
DiskOverData shows duplicates and assists you in determining data TTL towards data movement options. It’s the new and I’m proved vision over the DataIQ tool. Somewhat related functions, but goes beyond analytics.


IIQ is great , in cluster and platform focused only, no visibility applicable towards other system than OneFS per se. IIQ can’t play in the other’s sandbox.



Cheers.

/ls

Luc Simard
Google Personal Email:
- simard.j...@gmail.com
Mobile gVox: 415-501-0438
Washington State, USA



To view this discussion on the web visit https://groups.google.com/d/msgid/isilon-user-group/CAPhEwtqR4pa9vL%3DdnrckQN7H%2BPscBZpin3fQ6JKeOofY%3DSmB5A%40mail.gmail.com.
<698e4b54-392d-40c0-982b-9774381ab8d3.png><storagereview-hpe-greenlake-file-storage.jpeg><dellemc-powerscale-f900-00015lf.psd>

Reply all
Reply to author
Forward
0 new messages