expect script for scp and xargs?
kimberly...@gmail.com
[AIX 5.3]
Perhaps this is a simple question: we have a script that uses expect
to scp files. we came across the infamous 'too many files" error
message and thought we would use xargs to fix it; however, I can't
seem to make it work properly. i've tried echoing out the action and
appending |xargs scp at the end - which does work, but not for large
numbers of files - same message. Does anyone have any ideas? I've
restored the script to how it looks now.
thanks in advance.
#!/usr/local/bin/expect
# aa_scp3 - Script to run Secure Copy. It handles interactive
dialogue
# with remote host.
# Remote username, password and hostname must be set as
environment variables.
# Expect and its dependencies must be installed.
set timeout -1 ;# Set timeout to infinity (i.e. no timeout).
set USAGE "Usage: aa_scp3 <scp_flags> <local_file> <remote_file>
<put/
get>\n"
if {[llength $argv] != 4} {
puts "ERROR: Invalid number of arguments.\n"
puts "$USAGE"
#exit 11
}
set SCRIPT_DIR [file dirname $argv0]
source $SCRIPT_DIR/aa_expect_ssh.lib
#
# Set remote user, password and hostname.
#
set R_USER [get_env_value "db_login"]
if {[string length $R_USER] == 0} {exit 12}
set R_PASS [get_env_value "db_password"]
if {[string length $R_USER] == 0} {exit 12}
set R_HOST [get_env_value "net_connect"]
if {[string length $R_USER] == 0} {exit 12}
#
# Set scp flags, local filename, remote filename and copy direction
(put or get).
#
set ARGS [lindex $argv 0]
set L_FILE [lindex $argv 1]
set R_FILE [lindex $argv 2]
set DIRECTION [lindex $argv 3]
#
# Check for '-' before scp flags specified on command line.
#
if {([string length $ARGS] > 0) && ([string first "-" $ARGS] != 0)} {
puts "ERROR: Optional flags for scp must precede with '-'!\n"
exit 14
}
#
# Construct command line to run based on copy direction.
#
if {[string compare -nocase $DIRECTION "put"] == 0} {
set action "scp $ARGS $L_FILE $R_USER@$R_HOST:$R_FILE"
} elseif {[string compare -nocase $DIRECTION "get"] == 0} {
set action "scp $ARGS $R_USER@$R_HOST:$R_FILE $L_FILE"
} else {
puts "ERROR: Invalid scp direction. put or get expected.\n"
puts "$USAGE"
exit 15
}
set status [run_ssh_command $action $R_PASS]
puts "Exiting with status = $status"
exit $status
Cameron Laird
<kimberly...@gmail.com> wrote:
>I was told I might get some help here...
>
>[AIX 5.3]
>
>Perhaps this is a simple question: we have a script that uses expect
>to scp files. we came across the infamous 'too many files" error
>message and thought we would use xargs to fix it; however, I can't
>seem to make it work properly. i've tried echoing out the action and
>appending |xargs scp at the end - which does work, but not for large
>numbers of files - same message. Does anyone have any ideas? I've
>restored the script to how it looks now.
.
.
>set R_USER [get_env_value "db_login"]
>if {[string length $R_USER] == 0} {exit 12}
>set R_PASS [get_env_value "db_password"]
>if {[string length $R_USER] == 0} {exit 12}
>set R_HOST [get_env_value "net_connect"]
>if {[string length $R_USER] == 0} {exit 12}
.
.
>set ARGS [lindex $argv 0]
>set L_FILE [lindex $argv 1]
>set R_FILE [lindex $argv 2]
>set DIRECTION [lindex $argv 3]
.
.
A. I don't understand the question. I'll re-read
your description when I have more time; perhaps
it'll become clear.
B. Do you truly want to exit if the environment isn't
properly configured without logging what the
fault was? Consider, in any case, this alter-
native:
proc get_nonnull_env_value key {
set result [get_env_value $key]
if {![string length $result]} {
exit 12
}
return $result
}
set R_USER [get_nonnull_env_value db_login]
set R_PASS [get_nonnull_env_value db_password]
set R_HOST [get_nonnull_env_value net_connect]
.
.
.
foreach {ARGS L_FILE R_FILE DIRECTION} $argv {}
Uwe Klein
> I was told I might get some help here...
>
> [AIX 5.3]
>
> Perhaps this is a simple question: we have a script that uses expect
> to scp files. we came across the infamous 'too many files" error
> message and thought we would use xargs to fix it; however, I can't
> seem to make it work properly. i've tried echoing out the action and
> appending |xargs scp at the end - which does work, but not for large
> numbers of files - same message. Does anyone have any ideas? I've
> restored the script to how it looks now.
using xargs will not get you around that obstacle ( arglength limit in the remote shell).
What do your filepatterns look like?
either partition your filespec into smaller parts
( works very well for numbered files like funnipic0012456.jpg)
i.e with a pattern like "funnypic*.jpg"
you iterate your scps
through funnypic*[01].jpg, funnypic*[23].jpg,.., funnypic*[89].jpg
or some similar strategy.
or you create a local list of files to transfer
via executing "find $frompattern" and reading in the names
into expect.
and than start to run scp with batches of "reasonable" amounts of filenames.
uwe
kimberly...@gmail.com
point what worked with key-based exchange (not passwords) was
set action "echo $ARGS|xargs ls|xargs scp -I {} $R_USER@$R_HOST:
$R_FILE"
at least on the put part. But when password authentication was
defaulted to, it appeared as though it was putting in a echoed out
file name for the password, rather than the password. Do you have any
ideas? I don't want to do a for..i.. looping statement, if I have
5000 files, that sounds slow. filepatterns are filename.log, or
filename.name.log, nothing odd or weird. I also read somewhere AIX can
have MAX_ARGS set at runtime?
On Aug 11, 2:01 am, Uwe Klein <uwe_klein_habertw...@t-online.de>
wrote:
kimberly...@gmail.com
"echo %L_FILE|xargs ls|xargs -I {} scp $ARGS {} $R_USER@$R_HOST:
$R_FILE"
Cameron Laird
<kimberly...@gmail.com> wrote:
>I think feeding batches into scp was what I was attempting. At one
>point what worked with key-based exchange (not passwords) was
>set action "echo $ARGS|xargs ls|xargs scp -I {} $R_USER@$R_HOST:
>$R_FILE"
> at least on the put part. But when password authentication was
>defaulted to, it appeared as though it was putting in a echoed out
>file name for the password, rather than the password. Do you have any
>ideas? I don't want to do a for..i.. looping statement, if I have
>5000 files, that sounds slow. filepatterns are filename.log, or
>filename.name.log, nothing odd or weird. I also read somewhere AIX can
>have MAX_ARGS set at runtime?
.
.
If I wanted to know more about MAX_ARGS configuration,
I'd ask the folks at comp.unix.aix for help.
If I understand you correctly, performance concerns
have cause you to avoid an explicit iteration in favor
of an xargs-based solution. You also require password-
base authentication. The latter is going to be hard to
combine with xargs (especially in any way that doesn't
lead to its own performance issues).
My recommendation, therefore, is to write the
"for ..."- based solution, and see if *that* is fast
enough for you. If not, we can try other ideas ...
If performance is a major issue, I would think that
Expect multitasking is one of the first options worth
consideration.