scp session stalled

[Peng Yu]

I am trying to copy a huge file by scp from one ubuntu system to
another ubuntu system, which is estimated to take 2.5 hours. However,
scp never finish copying, and it always stalls at some point, and I
have to type Enter to let it continue. Do you know what cause this
problem and how to fix it? Thank you!

[Nico Kadel-Garcia]

Break up the file into pieces, with a command like "split". Use rsync
over ssh to mirror the contents of the split directory. When complete,
remerge the "split" pieces with the "cat" command. And run checksums
on the resulting file on both ends.

[mikea]

Nico Kadel-Garcia <nka...@gmail.com> wrote in <f1568020-92da-4117...@r1g2000yqj.googlegroups.com>:

Thereby fixing the symptoms, not the problem.

Anyone got an idea on how to do problem determination for this problem?

--
Mike Andrews, W5EGO
mi...@mikea.ath.cx
Tired old sysadmin

[Nico Kadel-Garcia]

On Apr 4, 9:23 am, mikea <mi...@mikea.ath.cx> wrote:

> Nico Kadel-Garcia <nka...@gmail.com> wrote in <f1568020-92da-4117-a64d-8651a914d...@r1g2000yqj.googlegroups.com>:
>
> > On Mar 27, 10:08?am, Peng Yu <pengyu...@gmail.com> wrote:
> >> I am trying to copy a huge file by scp from one ubuntu system to
> >> another ubuntu system, which is estimated to take 2.5 hours. However,
> >> scp never finish copying, and it always stalls at some point, and I
> >> have to type Enter to let it continue. Do you know what cause this
> >> problem and how to fix it? Thank you!
>
> > Break up the file into pieces, with a command like "split". Use rsync
> > over ssh to mirror the contents of the split directory. When complete,
> > remerge the "split" pieces with the "cat" command. And run checksums
> > on the resulting file on both ends.
>
> Thereby fixing the symptoms, not the problem.
>
> Anyone got an idea on how to do problem determination for this problem?

Thereby fixing the *technique*, which also avoids the problem. 2.5
hours continuous transfers without some sort of checkpoint was
problematic back in the days of modems, using SSH for the connetion
doesn't help the issue. With frequent oversubscription of home
bandwidth, and occasional interruptions with switch reconfigurations
and router changes even in commercial grade connections, it's just not
reliable to have 2.5 hours of continous data transmision.

[Fred Mobach]

Nico Kadel-Garcia wrote:

Another option might be to use rsync --partial on the complete file, so
in case the transfer is interrupted restarting the rsync process will
add the next part of the file to its destination.
--
Fred Mobach - fr...@mobach.nl
website : https://fred.mobach.nl
.... In God we trust ....
.. The rest we monitor ..

[mikea]

Nico Kadel-Garcia <nka...@gmail.com> wrote in <013d85b2-a934-41b6...@x12g2000yqx.googlegroups.com>:

I know people who have similar problems on their internal nets, which
run at speeds from 100 Mbit to OC-192. These nets run through high-end
Cisco and/or Juniper switches, and the connections are fully nailed-up.

If there is a problem in scp, then the problem in scp needs to be fixed,
and the first step in fixing the problem is deternining the location of
the problem. Avoiding or circumventing the problem may get the work
done, but at some point someone should recreate the problem on systems
running with as much instrumentation as may be required to locate the
bug and fix the code.

Avoiding the problem also avoids the fix.

--
The National Security Agency is currently recruiting highly-qualified
physicists, engineers, computer scientists, and mathematicians to work on
research in communications phenomena. If you are interested in a career
with the NSA, pick up any phone and call anybody.