isi_vol_copy_vnx slower that rsync?

[Scot]

Copying Celerra data over to Isilon using isi_vol_copy_vnx. 

 # isi_vol_copy_vnx <ip>:/share  -sa ndmp:passXXX /ifs/blaaa/destination -full 

Background: 

I have a group of Linux rsync VM's on the same 10G backup network so I can run 10-20 rsyncs in parallel for subfolders of the share. With this setup I can usually get 800Mb's from each VM. BUT, The VM is reading the file of Celerra over NFS then writting it back out over the same link to Isilon.  

I figured a solution like isi_vol_copy_vnx running on the array would be much faster but no! 

I have a 'backup' Isilon IP pool configured and bound to my 10G-Agg Isilon interfaces and the Celerra also has 10G nics on the same VLAN.   

I can see from netstat on the Celerra and the Isilon that NDP is going over the 10G Backup interface. 

I can literally read the name of each file as it is copied in std out. It's slow as hell! 

 Anyone looked into  isi_vol_copy or isi_vol_copy_vnx performance?

1555010018: Extracting ./blaaa/filename1 ino:8259351 total extracted:5779000

1555010018: Extracting ./blaaa/filename2 ino:8261525 total extracted:5780000

1555010018: Extracting ./blaaa/filename3 ino:8263487 total extracted:5781000

Would I be better off sticking with running rsync directly from an Isilon node or starting rsyncd on the Isilon or is there something else going on with NDMP on Isilon?  

Looked for some KB or write-ups on Dell Community, this forum and Google but haven't seen much but this and a few notes on people trying to get ndmp working.  

https://www.emc.com/collateral/white-papers/h12517-wp-emc-nfs-file-migration.pdf 

Appreciate any feedback. 

Thanks 

Scot 

[Luc Simard]

Scot

If you are looking for efficiency and Speed, reach out to DataDobi, for an alternative, I’d recommend you reach out to your SE and Sales team, to engage Dell EMC professional services, they have tools to do this and it would give you a turn key solution.

The challenges around this type of activity is often the forgotten matching of persons and user credentials between platforms. I still have nightmares (j ) from previous manual migrations.

Luc Simard - 415-501-0438

simard.j...@gmail.com

Messages may contain confidential information.

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dan Pritts]

I ended up using a combination of rsync and robocopy. 

I'm a *nix guy so i was slow to accept it, but robocopy is a very good tool; it does multithreaded copies and might saturate your network and/or disk.  It also has the significant advantage (for anyone doing windows anyway) of copying the full ACLs.

You can also probably still download EMCopy from support; it is similar to robocopy.  When I tried it it had some bug that broke it totally for me, but I recall they did eventually fix the bug.  I don't know what advantages it might have.

But honestly, if you don't need the acls and can run many rsyncs in parallel I'd just stick with that.  It's a time tested tool that you can trust.  This other stuff...makes me nervous.

danno

Luc Simard wrote on 4/11/19 4:39 PM:

--

Dan Pritts
ICPSR Computing & Network Services
University of Michigan

[Scot]

isi_vol_copy_vnx worked ok with smaller volumes including acls but the speed was just terrible. 

Tried mbuffer from https://www.schirmacher.de/display/Linux/Fastest+rsync+command .

 I think NFS is getting in the way back to writing a reading from the same nic. 

tar cf - * | mbuffer -m 1024M |(cd /isilon/AssetsYear/2017/08/; tar xf -) 

\summary:  558 GiByte in 4h 35min 28.0sec - average of 34.6 MiB/s

Maybe running directly from an Isilon node may help... 

Has anyone used rsyncd on the Isilon ? 

Also I can mount the Celerra share on a node but that definatly does mess up ACLs because of root and access zones.  

 

  

[Dan Pritts]

Scot wrote on 4/12/19 12:06 PM:

> isi_vol_copy_vnx worked ok with smaller volumes including acls but the
> speed was just terrible.
>
> Tried mbuffer from
> https://www.schirmacher.de/display/Linux/Fastest+rsync+command .
>  I think NFS is getting in the way back to writing a reading from the
> same nic.

NFS and SMB are very chatty and most of these tools are single-threaded,
so you will have lots of delays during round trips.  Also if you have a
lot of small files you have a ton of creation/acl operations which are
also single threaded.

That's the big advantage of robocopy.  It can have many copies in flight
at the same time.  Parallelizing them like this lets the bandwidth of
these pauses from one thread be used by the other threads.

> Has anyone used rsyncd on the Isilon ?

I've used rsync, but not rsyncd.  Works fine, but not ACL aware IIRC.

> Also I can mount the Celerra share on a node but that definatly does
> mess up ACLs because of root and access zones.

try "isi_run" to run your rsync in the context of whatever zone you are
copying into.  Unsupported, YMMV, I have not tried this with rsync.

danno

[Scot]

Thanks for the tip! 

I can get about 1.6Gb/s - 1.46Gb/s out of running rsync mounting src and destination over 10G using 20 copies of rsync on one VM on the same VLAN. 

Guess I should be happy with that but really not when you pay 1/2 million for storage arrays and reduced to man time and rsync to move your data to the new array.  

I know there are other parallel rsync solutions out there but this is a perl script Ive been using since the 90's keep -t(n) forks running. 

/root/run_fork_cmdsv2.pl -s /vdm7a/AssetsYear/2017//09/ -d /isilon/AssetsYear/2017/09/ -t 20 -l /var/rsync/logs/2017/

Guess it's job security, but I could get me teenager to do this. 

Scot 

[bob flynn]

dobiminer is a good solution. downside is its a licensed product, but in the scheme of things..

--

[Scot]

Don't tell me that !  I'm planning on using SyncIQ to replicate the same content to a DR site 😕

I believe you can only run isi_vol_copy_vnx on a per volume basis. 

In the background it's talking a snaphot off the Celerra, mounting it and running /usr/bin/isi_emctar 

[Chris Pepper]

I’ve been running multiple rsync-over-ssh (not rsyncd) between Isilon clusters for years in a dozen hourly scripts. Just retired it this month — fragility became a burden for a production system, and it needed manual tweaking every time either cluster gained or lost nodes.

Note that ssh to a cluster raises hostkey issues, which you can work around in OpenSSH.

http://www.extrapepperoni.com/post/2011/07/OpenSSH-is-smart-about-cluster-hostkeys

The replacement system runs on Linux nodes and uses the ‘parallel’ command to break up jobs.

Chris