Moving files around OneFS

[Donald King]

Hey guys, sorry for what might be a noob-ish question.  I have two folders on my Isilon each with their own SMB share using AD for authentication and permissions.  I need to move about 1.5TB worth of data (about 100,000 files) from one to the other.  I can not remove or change the existing shares as they have other data.  My first thought was to move the files using mv from the shell, but then I would have to repropagate the permissions once they reach the destination.  Of course I can dedicate a server to move them over SMB and keep the permissions but that would take some time too.  Anyone ran into this and recommend a way to move forward?

Thanks.

[Jerry Uanino]

The mv command might preserve those.  I would check with support on that or maybe someone here could comment. Otherwise you could write a recursive chmod script to reapply acts of they are not complicated. 

Do you have complicated perms?  

On Monday, November 3, 2014, Donald King <donald...@gmail.com> wrote:

Hey guys, sorry for what might be a noob-ish question.  I have two folders on my Isilon each with their own SMB share using AD for authentication and permissions.  I need to move about 1.5TB worth of data (about 100,000 files) from one to the other.  I can not remove or change the existing shares as they have other data.  My first thought was to move the files using mv from the shell, but then I would have to repropagate the permissions once they reach the destination.  Of course I can dedicate a server to move them over SMB and keep the permissions but that would take some time too.  Anyone ran into this and recommend a way to move forward?

Thanks.

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Pritts]

I don't think the "mv" command will break the permissions, unless there's some weirdness like two separate AD domains.  I would recommend testing to be sure, of course.

If you have quotas on one or the other area, you'll get a permission denied error with mv.  You can disable and then re-enable the quotas as needed.  Alternately, use "cp -l" to create hard links at the destination, then remove the original, as suggested in a previous conversation here.  I think that will still preserve the ACLs but you should test to be sure.

good luck

danno

Donald King

November 3, 2014 at 10:46 AM

Hey guys, sorry for what might be a noob-ish question.  I have two folders on my Isilon each with their own SMB share using AD for authentication and permissions.  I need to move about 1.5TB worth of data (about 100,000 files) from one to the other.  I can not remove or change the existing shares as they have other data.  My first thought was to move the files using mv from the shell, but then I would have to repropagate the permissions once they reach the destination.  Of course I can dedicate a server to move them over SMB and keep the permissions but that would take some time too.  Anyone ran into this and recommend a way to move forward?

Thanks.

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Dan Pritts
ICPSR Computing & Network Services
University of Michigan
+1 (734)615-7362

[Donald King]

Thanks for the ideas guys.  It is one AD domain and the permissions are not complicated - a few groups have R/W permissions to the entire share.  

[Jason Davis]

If you're licensed for SyncIQ, then you could do a "localhost" for source and destination Sync... works pretty well.

Another forum member had a quick write up about it over here: http://storagemonkey.wordpress.com/2013/06/03/isilon-synciq-loopback-replication/ 

[John Beranek - PA]

mv will certainly work, preserve permissions and be quick, with the caveat of mv'ing across quota boundaries as mentioned by Daniel.

One the other hand, 100,000 files and 1.5TB of data doesn't sound like all that much to move around...just in the last 24 hours I managed to copy 700GB in around 2 million files onto a cluster over SMB "accidentally", and then promptly deleted all those files with a TreeDelete job.

The thing that took the longest was the SyncIQ to the secondary cluster.

Cheers,

John

[julian firminger]

EMCopy if you dont mind transiting out of the cluster and back.  Only 1.5TB this should not be an issue.  I moved chunks of 20-30TB using this with no issue (other than time)

Or, if non-inherited permissions are a major issue you need to look at scripting a "chmod +a#" after your move.

mv will not carry specificly set permissions on folders, it will carry inheritance only.  

[Donald King]

Thanks everyone for their input.  I ended up using mv command line on the cluster and it went smoothly.  I will definitely keep emcopy in mind for future data moves and copies.

I am not licensed for SyncIQ yet.  I actually have a new cluster still on pallets in my data center I need to set up for that reason, to act as a backup cluster for my production and receive sync'd snapshots.  I hope to get that replication going in the next few weeks after I upgrade to 7.1 on production.

But that leads me to another question.  In the future if I move this much data from one share to another I guess it will cause a very large snapshot to replicate?  If you assume I have both shares set with the same snap/sync policies, what will the sync result be?  I will be replicating snapshots over a 1Gbps WAN connection with about 55 ms of latency (from Florida to California), so I will want to avoid sending large snapshots if I can.  

-Don

On Monday, November 3, 2014 10:46:14 AM UTC-5, Donald King wrote:

[Dan Pritts]

EMCopy if you dont mind transiting out of the cluster and back.  Only 1.5TB this should not be an issue.  I moved chunks of 20-30TB using this with no issue (other than time)

tangentially - I had trouble with EMCopy's SID mapping feature.  Just didn't work reliably.  I reported the bug, hopefully it's been fixed. 

mv will not carry specificly set permissions on folders, it will carry inheritance only. 

Are you sure about that, Julian?  I'm not sure why that would be. 

[John Beranek - PA]

I had various issues with EMCopy, and just decided that Robocopy is generally much more flexible, and equally performant...

John

[Peter Serocka]

On 2014 Nov 6. md, at 21:52 st, Donald King wrote:

But that leads me to another question.  In the future if I move this much data from one share to another I guess it will cause a very large snapshot to replicate?  If you assume I have both shares set with the same snap/sync policies, what will the sync result be?  I will be replicating snapshots over a 1Gbps WAN connection with about 55 ms of latency (from Florida to California), so I will want to avoid sending large snapshots if I can.  

Snapshots are never sent over the wire... 

both sides create snapshots on their own :)

-- Peter

Peter Serocka

CAS-MPG Partner Institute for Computational Biology (PICB)

Shanghai Institutes for Biological Sciences (SIBS)

Chinese Academy of Sciences (CAS)

320 Yue Yang Rd, Shanghai 200031, China

pser...@picb.ac.cn

[Donald King]

Interesting.  I have no experience yet with snapshots or replication, and I am looking forward to diving into it. We have been dumping data to tape thus far.  You can imagine how joyful that is.....

Thanks everyone for their help!

-Don