Monthly Delete Jobs

[Kenneth Van Kley]

I have a customer who adds over 8 million files a day to our Isilon cluster.  The retention on these files is only 3 months and then they purge them.

Currently, they have a monthly chron job that runs from one of their NAS clients to do the "rm -rf" process, but it takes about 1.5 months to run for each month they purge.

Obviously this isn't working, so they ask me to delete their files using TreeDelete instead which takes about 2-3 days to complete.

Isilon cluster is 8 nodes, x410's.

Customer is using RHEL VM's and NFSv4 to connect (very low latency from client to cluster).

Since I don't want to be in the "monthly Treedelete" business, I'm trying to find a way to get the customer to be able to execute this on their own without actually having them login to the Isilon and run TreeDelete (security and scope issues).

Any suggestions?

[Josh Hampton]

Why not monthly cron the treedelete job?  If it's the same directory every time.

[Dan Pritts]

If you want to keep this on a client, you could parallelize the rm operation.  rm -rf will do the rm's serially, which means waiting for an NFS roundtrip.  Presumably there's a well-known tree structure you could  use? 



If all the client activity is (or could be) done from a single vm, you could create new virtual disks every day/every month, and store the files in the virtual disk.   3 months later, you have one file to delete, not 8 million. 

this is obviously achievable at the hypervisor level if you are using the cluster as VM storage.  If you're not, it's still not that hard. 

dd if=/dev/zero of=/file/on/cluster bs=1m count=this_many_megs
mke2fs -t ext4 /file/on/cluster   #  substitute your favorite filesystem here, mkfs.xfs, whatever - worth experimenting with if performance becomes an issue 
mount -o loop,rw /file/on/cluster /mnt/point

If the output size is highly variable, so this uses too much space, you could dork around with doing this via LVM and expanding the filesystem as needed.   Perhaps more complicated than it is worth. 


This method might perform better for general access, too.


This is of course horrible, but desperate times call for desperate measures.   And I can imagine  8 milllon reasons why you'd need multiple-client access, but just wanted to throw this out there.  


The other solution is of course, well, don't do that.  This is filesystem abuse.  Try hadoop, or an SQL database.

thanks
danno

Josh Hampton

November 30, 2016 at 12:25 PM

Why not monthly cron the treedelete job?  If it's the same directory every time.

On Wednesday, November 30, 2016 at 7:56:06 AM UTC-8, Kenneth Van Kley wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kenneth Van Kley

November 30, 2016 at 10:56 AM

I have a customer who adds over 8 million files a day to our Isilon cluster.  The retention on these files is only 3 months and then they purge them.

Currently, they have a monthly chron job that runs from one of their NAS clients to do the "rm -rf" process, but it takes about 1.5 months to run for each month they purge.

Obviously this isn't working, so they ask me to delete their files using TreeDelete instead which takes about 2-3 days to complete.

Isilon cluster is 8 nodes, x410's.

Customer is using RHEL VM's and NFSv4 to connect (very low latency from client to cluster).

Since I don't want to be in the "monthly Treedelete" business, I'm trying to find a way to get the customer to be able to execute this on their own without actually having them login to the Isilon and run TreeDelete (security and scope issues).

Any suggestions?

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Dan Pritts

ICPSR Computing & Network Services

University of Michigan 

[Saker Klippsten]

We wrote a service that sits in between the end user which they interface via a web-app. The end user  can browse and select local paths from their machine , or paste in a list of paths.

The Web-Service uses Active Directory combined with Two Factor Authentication to ensure the End-User has permissions to run such a powerful command. In Addition We check ensure the End-User has permissions to access each submitted path. If they do not have permission it will toss an error up. This gets passed off to a service  that runs on a Linux Machine which then after running its checks the above takes the path submitted and then ssh's into the cluster to submit the treedelete commands. On  the Web-Page the End User can see the status of their submitted Jobs this gets updated every 5 min.

I believe if my memory serves me correctly there being a limit to 10-15 treedelete commands that can be submitted. Our Service acts as a Queue and monitors this and submits more as necessary or holds onto it. 

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-group+unsubscribe@googlegroups.com.

[Kenneth Van Kley]

Management wants the customer to be responsible for deleting their own files.

[Kenneth Van Kley]

OK, that's pretty slick...  Is that something you can share?

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

[Dan Pritts]

dd if=/dev/zero of=/file/on/cluster bs=1m count=this_many_megs
mke2fs -t ext4 /file/on/cluster   #  substitute your favorite filesystem here, mkfs.xfs, whatever - worth experimenting with if performance becomes an issue 
mount -o loop,rw /file/on/cluster /mnt/point

If the output size is highly variable, so this uses too much space, you could dork around with doing this via LVM and expanding the filesystem as needed.   Perhaps more complicated than it is worth. 

You could alternately use a sparse file, in which case the cluster will have to deal with the extreme fragmentation of that file. 

[Saker Klippsten]

Unfortunately not :(  but it only took a couple weeks of dev time.. in between other jobs for them to complete. Also we developed this when V7 came out. I still do not think in the latest API for V8 they support treedelete commands but that was our reason for going SSH.. 

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-group+unsubscribe@googlegroups.com.

[Dustin Dietz]

I did a quick comparison between two alternative options of deleting files. Each test was against a separate directory containing 500,000 files.

time rm -rf *

real    64m52.374s

user    0m2.812s

sys     1m8.698s

time perl -e 'unlink <*>'

real    37m57.296s

user    0m1.940s

sys     1m4.042s

[Dan Pritts]

If there were any subdirectories in there, perl didn't delete them, but rm did. 

If the results are repeatable I'm surprised.  I would expect the NFS roundtrip times to dominate the timing. 

Dustin Dietz

November 30, 2016 at 3:26 PM

I did a quick comparison between two alternative options of deleting files. Each test was against a separate directory containing 500,000 files.

time rm -rf *

real    64m52.374s

user    0m2.812s

sys     1m8.698s

time perl -e 'unlink <*>'

real    37m57.296s

user    0m1.940s

sys     1m4.042s

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Saker Klippsten

November 30, 2016 at 2:18 PM

Unfortunately not :(  but it only took a couple weeks of dev time.. in between other jobs for them to complete. Also we developed this when V7 came out. I still do not think in the latest API for V8 they support treedelete commands but that was our reason for going SSH.. 

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Kenneth Van Kley

November 30, 2016 at 1:50 PM

OK, that's pretty slick...  Is that something you can share?

On Wednesday, November 30, 2016 at 12:13:26 PM UTC-6, Saker Klippsten wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Saker Klippsten

November 30, 2016 at 1:13 PM

We wrote a service that sits in between the end user which they interface via a web-app. The end user  can browse and select local paths from their machine , or paste in a list of paths.

The Web-Service uses Active Directory combined with Two Factor Authentication to ensure the End-User has permissions to run such a powerful command. In Addition We check ensure the End-User has permissions to access each submitted path. If they do not have permission it will toss an error up. This gets passed off to a service  that runs on a Linux Machine which then after running its checks the above takes the path submitted and then ssh's into the cluster to submit the treedelete commands. On  the Web-Page the End User can see the status of their submitted Jobs this gets updated every 5 min.

I believe if my memory serves me correctly there being a limit to 10-15 treedelete commands that can be submitted. Our Service acts as a Queue and monitors this and submits more as necessary or holds onto it. 

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Josh Hampton

November 30, 2016 at 12:25 PM

Why not monthly cron the treedelete job?  If it's the same directory every time.

On Wednesday, November 30, 2016 at 7:56:06 AM UTC-8, Kenneth Van Kley wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

[Dustin Dietz]

Sorry if I didn't make it clear. Both directories had a flat structure with 500,000 files (specifically to make the process simpler). No subdirs. 

On Wed, Nov 30, 2016 at 3:23 PM, Dan Pritts <da...@umich.edu> wrote:

If there were any subdirectories in there, perl didn't delete them, but rm did. 

If the results are repeatable I'm surprised.  I would expect the NFS roundtrip times to dominate the timing. 

Dustin Dietz

November 30, 2016 at 3:26 PM

I did a quick comparison between two alternative options of deleting files. Each test was against a separate directory containing 500,000 files.

time rm -rf *

real    64m52.374s

user    0m2.812s

sys     1m8.698s

time perl -e 'unlink <*>'

real    37m57.296s

user    0m1.940s

sys     1m4.042s

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-group+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Saker Klippsten

November 30, 2016 at 2:18 PM

Unfortunately not :(  but it only took a couple weeks of dev time.. in between other jobs for them to complete. Also we developed this when V7 came out. I still do not think in the latest API for V8 they support treedelete commands but that was our reason for going SSH.. 

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-group+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Kenneth Van Kley

November 30, 2016 at 1:50 PM

OK, that's pretty slick...  Is that something you can share?

On Wednesday, November 30, 2016 at 12:13:26 PM UTC-6, Saker Klippsten wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-group+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Saker Klippsten

November 30, 2016 at 1:13 PM

We wrote a service that sits in between the end user which they interface via a web-app. The end user  can browse and select local paths from their machine , or paste in a list of paths.

The Web-Service uses Active Directory combined with Two Factor Authentication to ensure the End-User has permissions to run such a powerful command. In Addition We check ensure the End-User has permissions to access each submitted path. If they do not have permission it will toss an error up. This gets passed off to a service  that runs on a Linux Machine which then after running its checks the above takes the path submitted and then ssh's into the cluster to submit the treedelete commands. On  the Web-Page the End User can see the status of their submitted Jobs this gets updated every 5 min.

I believe if my memory serves me correctly there being a limit to 10-15 treedelete commands that can be submitted. Our Service acts as a Queue and monitors this and submits more as necessary or holds onto it. 

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-group+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Josh Hampton

November 30, 2016 at 12:25 PM

Why not monthly cron the treedelete job?  If it's the same directory every time.

On Wednesday, November 30, 2016 at 7:56:06 AM UTC-8, Kenneth Van Kley wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-group+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Dan Pritts

ICPSR Computing & Network Services

University of Michigan 

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-group+unsubscribe@googlegroups.com.

[Jerry Uanino]

Pack those files.
Use a file per day or an archive where you keep appending to a zip or tar. Even packing by hour is better than nothing. Make the app more intelligent. You're asking for it with more than 10k files per dir. At that point you probably want a db instead of using your file system as a meta store. I'm guessing these files are super tiny and also trigger the larger overhead on isilon anyway?

On Wed, Nov 30, 2016 at 1:42 PM Dustin Dietz <dustin...@gmail.com> wrote:

Sorry if I didn't make it clear. Both directories had a flat structure with 500,000 files (specifically to make the process simpler). No subdirs. 

On Wed, Nov 30, 2016 at 3:23 PM, Dan Pritts <da...@umich.edu> wrote:

If there were any subdirectories in there, perl didn't delete them, but rm did. 

If the results are repeatable I'm surprised.  I would expect the NFS roundtrip times to dominate the timing. 

Dustin Dietz

November 30, 2016 at 3:26 PM

I did a quick comparison between two alternative options of deleting files. Each test was against a separate directory containing 500,000 files.

time rm -rf *

real    64m52.374s

user    0m2.812s

sys     1m8.698s

time perl -e 'unlink <*>'

real    37m57.296s

user    0m1.940s

sys     1m4.042s

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Saker Klippsten

November 30, 2016 at 2:18 PM

Unfortunately not :(  but it only took a couple weeks of dev time.. in between other jobs for them to complete. Also we developed this when V7 came out. I still do not think in the latest API for V8 they support treedelete commands but that was our reason for going SSH.. 

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Kenneth Van Kley

November 30, 2016 at 1:50 PM

OK, that's pretty slick...  Is that something you can share?

On Wednesday, November 30, 2016 at 12:13:26 PM UTC-6, Saker Klippsten wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Saker Klippsten

November 30, 2016 at 1:13 PM

We wrote a service that sits in between the end user which they interface via a web-app. The end user  can browse and select local paths from their machine , or paste in a list of paths.

The Web-Service uses Active Directory combined with Two Factor Authentication to ensure the End-User has permissions to run such a powerful command. In Addition We check ensure the End-User has permissions to access each submitted path. If they do not have permission it will toss an error up. This gets passed off to a service  that runs on a Linux Machine which then after running its checks the above takes the path submitted and then ssh's into the cluster to submit the treedelete commands. On  the Web-Page the End User can see the status of their submitted Jobs this gets updated every 5 min.

I believe if my memory serves me correctly there being a limit to 10-15 treedelete commands that can be submitted. Our Service acts as a Queue and monitors this and submits more as necessary or holds onto it. 

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Josh Hampton

November 30, 2016 at 12:25 PM

Why not monthly cron the treedelete job?  If it's the same directory every time.

On Wednesday, November 30, 2016 at 7:56:06 AM UTC-8, Kenneth Van Kley wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Dan Pritts

ICPSR Computing & Network Services

University of Michigan 

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

[Dan Pritts]

I just tried the same thing with directories with 50k files each, and rm was faster (1:59 vs 2:12).     RHEL6 client, NFS3, onefs 7.2.1.2.

I had created the files 8 hours earlier, and explicitly dropped the caches on the client, so it probably wasn't in cache. 

Curious that your 500k files took so tremendously longer than my 50k files.  I'm on X200 & X210 nodes, no SSD. 

Dustin Dietz wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Peter Serocka]

On 2016 Dec 1. md, at 02:43 st, Kenneth Van Kley wrote:

> Management wants the customer to be responsible for deleting their own files.

Have them move the enclosing dir to a pre-defined Trash folder,
under the same export and Smartquota domain. That's fast,
and they can do it whenever they want.

A weekly or even daily cron job on the Isilon
would launch TreeDelete to clean up the Trash;
ideally wrapped in a small script which renames
the Trash folder, creates a fresh one, and
then deletes the renamed one.

Though it must say, my first idea was exactly
the kind of service that Saker described...

In case you want to try out parallel removes
from the NFS client, you may start with xarg's ability
to run concurrent jobs, something like:

find /path/to/dir -type f -print0 | xargs -0 -t -P 10 rm

-P 10 is the (max) number of parallel jobs
-t print each job's command line on start

Though you might find -- with all files
sitting in a single subdir -- that locking on that subdir
can effectively serialize all removes, and not
much can be gained. (Which leaves us wondering
what magic sauce TreeDelete uses to remove
multiple files from one directory ;)

hth

-- Peter

[Laurence Schuler]

Be aware that the 'rm -rf *' is using 'globbing' in the shell and is NOT a very efficient method of selecting multiple files. You'd be better off trying something like 'find . -delete' and see how much time it takes.

--Larry Schuler

[Robert Chang]

Hi folks, interesting discussion.

Speaking of OneFS APIs, a lesser-known API is the RAN API, stands for Restful Access to Namespace.  This API allows you to access the files and directories on the cluster and "do stuff" to them.

Link to the V8 API is here (OneFS v8.0 and above) -- http://doc.isilon.com/onefs/8.0.0/api/en-us/00-ifs-pub-onefs-api-reference.htm

The area that is relevant to this thread is -- 

1) http://doc.isilon.com/onefs/8.0.0/api/en-us/04-ifs-br-file-system-access-apis.htm

2) On the nav pane on the right, select "Delete a directory".

I've verified that this API resource exists in OneFS v7.2.0 code base.  Look for a PDF named "docu56063", page 268.

[Peter Serocka]

Does it use a TreeDelete job for recursive deletion?

— Peter

> On 2016 Dec 2 Fri, at 08:05, Robert Chang <robert.ch...@gmail.com> wrote:
>
> Hi folks, interesting discussion.
>
> Speaking of OneFS APIs, a lesser-known API is the RAN API, stands for Restful Access to Namespace. This API allows you to access the files and directories on the cluster and "do stuff" to them.
>
> Link to the V8 API is here (OneFS v8.0 and above) -- http://doc.isilon.com/onefs/8.0.0/api/en-us/00-ifs-pub-onefs-api-reference.htm
>
> The area that is relevant to this thread is --
> 1) http://doc.isilon.com/onefs/8.0.0/api/en-us/04-ifs-br-file-system-access-apis.htm
> 2) On the nav pane on the right, select "Delete a directory".
>
> I've verified that this API resource exists in OneFS v7.2.0 code base. Look for a PDF named "docu56063", page 268.
>
>
> On Wednesday, November 30, 2016 at 11:18:43 AM UTC-8, Saker Klippsten wrote:
> Unfortunately not :( but it only took a couple weeks of dev time.. in between other jobs for them to complete. Also we developed this when V7 came out. I still do not think in the latest API for V8 they support treedelete commands but that was our reason for going SSH..
>
>
> On Wed, Nov 30, 2016 at 10:50 AM, Kenneth Van Kley <kjv...@gmail.com> wrote:
> OK, that's pretty slick... Is that something you can share?
>
> On Wednesday, November 30, 2016 at 12:13:26 PM UTC-6, Saker Klippsten wrote:
> We wrote a service that sits in between the end user which they interface via a web-app. The end user can browse and select local paths from their machine , or paste in a list of paths.
> The Web-Service uses Active Directory combined with Two Factor Authentication to ensure the End-User has permissions to run such a powerful command. In Addition We check ensure the End-User has permissions to access each submitted path. If they do not have permission it will toss an error up. This gets passed off to a service that runs on a Linux Machine which then after running its checks the above takes the path submitted and then ssh's into the cluster to submit the treedelete commands. On the Web-Page the End User can see the status of their submitted Jobs this gets updated every 5 min.
>
> I believe if my memory serves me correctly there being a limit to 10-15 treedelete commands that can be submitted. Our Service acts as a Queue and monitors this and submits more as necessary or holds onto it.
>
>
>
>
>
>
>
>

[Saker Klippsten]

Treedelete as far as I know is not in the API.
I have requested this a couple times..

I highly doubt it uses treedelete ...
-s

[Peter Serocka]

POST /platform/3/job/jobs

in OneFS should do it. Yet: a user can be granted privileges
to *launch* a job, but once running, it executes
as root. This is why a middle-man agent like yours
is still needed.

Deleting large trees can take very long either way, with
or without the TreeDelete job. With the RAN API,
I wonder wether one is going to run into protocol timeouts
when the DELETE call is supposed to return only after the
actual deletion has finished....

fwiw

-- Peter

[Robert Chang]

Hi Pete,

TreeDelete is not used in the recursive delete call invoked through the RAN API.

The product manager is aware of this customer ask and we are working on getting this queued up.

Thanks.

[Peter Serocka]

Thank you Robert! (and it's probably trickier than it appears at first sight.)

-- Peter