multiple scp using xargs

[sysdrk]

I have a lot of files I need to copy to another server using scp
(secure copy). For efficiency reasons I want to minimize the number
of scp invocations. So, logically I would like to do this:

find ... [files] | xargs -i scp {} user@server:/somedir

Unfortunately, I have to use the replacement string {} since I need to
provide a directory as the last parameter to scp. However, this
causes xargs to invoke scp for each individual file.

I can see doing this using a double xargs like this:

find ... [files] | xargs echo scp user@server:/somedir |
sed 's/.*somedir//' |
xargs -i scp {} user@server:/somedir

Here the first xargs is used to put as many files together on a single
line as possible and then the second xargs is used to feed each line
of files into an scp.

Is there a better way to do this? Somehow it seems like there should
be a way to get xargs to do this with a single xargs but I haven't
seen a way.

Denis

[Ben Bacarisse]

sysdrk <sys...@wowway.com> writes:

> I have a lot of files I need to copy to another server using scp
> (secure copy). For efficiency reasons I want to minimize the number
> of scp invocations. So, logically I would like to do this:
>
> find ... [files] | xargs -i scp {} user@server:/somedir
>
> Unfortunately, I have to use the replacement string {} since I need to
> provide a directory as the last parameter to scp. However, this
> causes xargs to invoke scp for each individual file.

If your names are "space safe" (it seems so from your example) then

find ... | xargs | xargs -i scp {} user@server:/somedir

may do what you want.

--
Ben.

[Bill Marcum]

On 2010-01-05, sysdrk <sys...@wowway.com> wrote:
>
> I can see doing this using a double xargs like this:
>
> find ... [files] | xargs echo scp user@server:/somedir |
> sed 's/.*somedir//' |
> xargs -i scp {} user@server:/somedir
>
> Here the first xargs is used to put as many files together on a single
> line as possible and then the second xargs is used to feed each line
> of files into an scp.
>
> Is there a better way to do this? Somehow it seems like there should
> be a way to get xargs to do this with a single xargs but I haven't
> seen a way.
>

You could use xargs to call a script or function that takes the
destination as the first argument and calls scp.

scp_to () {
dest="$1"
shift
scp "$@" "$dest"
}

[Icarus Sparry]

Other options you might consider
1) Use something like cpio, pax or a "tar" that can read the list of
files from standard input. Then use "ssh" rather than "scp" to create a
pipeline like
find ... | pax -w | ssh user@server 'cd /somedir; pax -r'

2) Write your own wrapper for scp that reorders the arguments, e.g. myscp
#!/bin/sh
dest=$1
shift
scp "$@" "$dest"

and then use
find ... | xargs myscp user@server:/somedir

Of course you don't actually need to create your own script to do this,
you can use
find ... | xargs sh -c 'scp "$@" user@somehost:/somedir' dummy_arg0
but I think another file is cleaner, and you don't need to worry if you
need the dummy_arg0 or not.

[Karsten Kruse]

sysdrk schrieb:

> I have a lot of files I need to copy to another server using scp
> (secure copy). For efficiency reasons I want to minimize the number
> of scp invocations. So, logically I would like to do this:

> find ... [files] | xargs -i scp {} user@server:/somedir

> Unfortunately, I have to use the replacement string {} since I need to
> provide a directory as the last parameter to scp. However, this
> causes xargs to invoke scp for each individual file.

If the list of files is short, without spaces and such, try this:

scp $(find ...) user@server:/somedir

Otherwise try this:

find ... | tar -c | ssh user@host 'tar -x -C /targetdir'

Karsten

--

() My homepage is http://www.tecneeq.de/ and your homepage sucksￜ!
<\/>
_/\_ ￜ) Unless it has animated gifs from 1996, then it rocks!

[sysdrk]

> > I have a lot of files I need to copy to another server using scp
> > (secure copy).  For efficiency reasons I want to minimize the number
> > of scp invocations.  So, logically I would like to do this:
> > find ... [files]  |  xargs -i scp {} user@server:/somedir
> > Unfortunately, I have to use the replacement string {} since I need to
> > provide a directory as the last parameter to scp.  However, this
> > causes xargs to invoke scp for each individual file.

Thanks for all the replies. It looks like the wrapper suggestion is
the easiest and cleanest solution so I'm going to go with that.

Denis