Offload the copy process from client machine's to internal cluster cp

[Tom Stepniewski]

Is there a relatively easy and user-friendly way to offload the copy process from the client with mounted shares from the cluster, using a 1gb network, to the cluster?

Details: I have a client that mounts two SMB shares (Share-A & Share-B) served from the same cluster (Share-A=/ifs/data/boo/Share-A. Share-B=/ifs/data/boo/Share-B). A user then needs to copy several very large media files (200GB+) using the copy process/command on the local machine. This uses the network to execute the copy, which is very slow. I can execute the copy via CLI, and since it's the same cluster (cp /ifs/data/boo/Share-A/file /ifs/data/boo/Share-A/.), that is blazing fast. The shares cannot be modified to accommodate these actions due to security and other considerations. I'm not comfortable with providing CLI access to the users (risk & skillset). Is there a way to utilize RPC or some kind of API that would allow a user (Windows/Mac OS) to have the ability to select source, select target, but have the cluster run the cp? So far I've eliminated symlinks, and rsync of specific "dropbox"-like folders within shares in conjunction with cronjobs...

Using OneFS 8.0.0.4

SMB mounts on cluster

client OS is Windows 7 & macOS Sierra (soon to be HighSierra) 

Any thoughts appreciated..

Tom

[Peter Serocka]

OneFS has an API for file operations which offers an “internal” file copy.

From a usage standpoint, this is even less that a CLI;
one would use the API to program a CLI or GUI tool for the end users.

Permission-wise, the edn users need to be granted API access
to an enclosing folder, /ifs/data/boo in your example.

On the other hand, if you can’t modify the existing shares
but can create an additional share on /ifs/data/boo,
perhaps exclusively to a dedication client machine,
then you can experiment with SMB “server-side copies”.

Let me know if you need more info.

— Peter

> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Tom Stepniewski]

Peter,

Would love more info. Feel free to reach out off-forum, if you prefer.

> To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-group+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/isilon-user-group/Fcrfazi0LMI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to isilon-user-group+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Tom Stepniewski

Post Production IT Engineer

EMC Isilon / Telestream Vantage / IBM Aspera

tstepn...@axs.tv

o: 303-542-5697  

[Adam Fox]

It's a relatively simple operation to do but, as Peter said, it will require someone to write an application to do the work because it's an API.  The user running the application, even if it's some kind of service account must have the RBAC privileges to access the namespace (not on by default for anyone but admins) and the necessary file permissions to do the copy.  Once that is settled, it's pretty easy.  I'll use an example using curl to keep it platform independent but you can use whatever platform you like to build your tool.

So I start off on a Linux client that has nothing mounted.  I have a OneFS tarball stiing in /ifs. It's about 187M.  So not huge but not tiny either.  Let's say I want to copy it to /ifs/fox.  So on the Linux client I can simply do:

[foxa3@AFOX-51-c66-vm ~]$ time curl -k -u root:XXX -H "x-isi-ifs-copy-source: /namespace/ifs/OneFS_v8.0.1.2_Install.tar.gz" -X PUT https://10.111.158.130:8080/namespace/ifs/fox/OneFS_v8.0.1.2_Install.tar.gz

real    0m0.638s

user    0m0.048s

sys     0m0.062s

Hope this helps:

-- Adam Fox

adam...@yahoo.com

---

From: Tom Stepniewski <tstepn...@axs.tv>
To: isilon-u...@googlegroups.com
Sent: Friday, February 2, 2018 5:05 PM
Subject: Re: Isilon-Users Offload the copy process from client machine's to internal cluster cp

Peter,

Would love more info. Feel free to reach out off-forum, if you prefer.

On Fri, Feb 2, 2018 at 2:32 PM, Peter Serocka <pser...@picb.ac.cn> wrote:

OneFS has an API for file operations which offers an “internal” file copy.

From a usage standpoint, this is even less that a CLI;
one would use the API to program a CLI or GUI tool for the end users.

Permission-wise, the edn users need to be granted API access
to an enclosing folder, /ifs/data/boo in your example.

On the other hand, if you can’t modify the existing shares
but can create an additional share on /ifs/data/boo,
perhaps exclusively to a dedication client machine,
then you can experiment with SMB “server-side copies”.

Let me know if you need more info.

— Peter

> On 2018 Feb 2 Fri, at 19:32, Tom Stepniewski <tstepn...@axs.tv> wrote:
>
> Is there a relatively easy and user-friendly way to offload the copy process from the client with mounted shares from the cluster, using a 1gb network, to the cluster?
>

> Details: I have a client that mounts two SMB shares (Share-A & Share-B) served from the same cluster (Share-A=/ifs/data/boo/Share- A. Share-B=/ifs/data/boo/Share-B) . A user then needs to copy several very large media files (200GB+) using the copy process/command on the local machine. This uses the network to execute the copy, which is very slow. I can execute the copy via CLI, and since it's the same cluster (cp /ifs/data/boo/Share-A/file /ifs/data/boo/Share-A/.), that is blazing fast. The shares cannot be modified to accommodate these actions due to security and other considerations. I'm not comfortable with providing CLI access to the users (risk & skillset). Is there a way to utilize RPC or some kind of API that would allow a user (Windows/Mac OS) to have the ability to select source, select target, but have the cluster run the cp? So far I've eliminated symlinks, and rsync of specific "dropbox"-like folders within shares in conjunction with cronjobs...

>
> Using OneFS 8.0.0.4
> SMB mounts on cluster
> client OS is Windows 7 & macOS Sierra (soon to be HighSierra)
>
>
> Any thoughts appreciated..
>
> Tom
>
>
>
> --

> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-group+unsubscribe@ googlegroups.com.

> For more options, visit https://groups.google.com/d/ optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/ topic/isilon-user-group/ Fcrfazi0LMI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to isilon-user-group+unsubscribe@ googlegroups.com.
For more options, visit https://groups.google.com/d/ optout.

--

Tom Stepniewski

Post Production IT Engineer

EMC Isilon / Telestream Vantage / IBM Aspera

tstepn...@axs.tv

o: 303-542-5697  

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.

[Christian Korneck]

On Fri, Feb 2, 2018 at 7:32 PM, Tom Stepniewski <tstepn...@axs.tv> wrote:

Is there a relatively easy and user-friendly way to offload the copy process from the client with mounted shares from the cluster, using a 1gb network, to the cluster?

I believe OneFS 8 introduced [1]  SMB2 "server side copy", which is enabled by default. (Or can be enabled with: isi smb settings global modify --server-side-copy=yes ) As to [2] the client needs to be at least Windows 8.0

[1] https://blog.dellemc.com/en-us/announcing-onefs-8-0-0/

[2] https://wiki.samba.org/index.php/Server-Side_Copy

I believe "it just works" given smb shares on a OneFS 8.x cluster and a Windows8/8.1/10 client, but haven't verified. Maybe someone else has practical experience with it and can confirm?

Best,

Chris

[Patrick Andrew]

This is correct. In Riptide this is supported with SMB2 on Windows 8+ and OneFS 8.0.1+

-Patrick
Isilon Engineer