ssh and scp get stuck after some amount of data

[Bruno Haible]

Hi,

I'm using https://people.debian.org/~sthibault/hurd-i386/debian-hurd-20170115.img.gz
in a virtual machine (virtualbox - very robust with a number of other guest OSes).

Now I need to transfer a 2 GB tree to the host machine (declared as virtual
Ethernet card for the guest).

First attempt:
# tar cf - directory | ssh br...@10.0.2.2 tar xf -
It hangs after transferring 1.6 GB. I.e. no more data arrives within 15 minutes.

Rebooted.

Second attempt:
# tar cfJ tgz directory
# scp tgz br...@10.0.2.2:tgz
br...@10.0.2.2's password:
tgz 21% 74 MB 0.0KB/s - stalled -
It hangs after transferring 74 MB. I.e. no more data arrives within 30 minutes.

What's the workaround to get Ethernet connections, ssh in this case, work
reliably?

Bruno

[Bruno Haible]

Hi,

I tried:

> # tar cf - directory | ssh br...@10.0.2.2 tar xf -
> It hangs after transferring 1.6 GB. I.e. no more data arrives within 15 minutes.

Found a workaround: Throttling of the bandwidth.
- Throttling at the network adapter level [1] is not applicable to Hurd.
- The 'throttle' program [2] is no longer available.
- But a replacement program [3] is available.

The command that worked for me (it limits the bandwidth to 1 MB/sec):

# tar cf - directory | ~/throttle.py --bandwidth 1024576 | ssh br...@10.0.2.2 tar xf -

But really, this is only a workaround. It smells like a bug in ssh or the Hurd.

Bruno

[1] https://askubuntu.com/questions/307907/can-we-manually-throttle-the-bandwidth-of-a-network-interface
[2] https://linux.die.net/man/1/throttle
[3] https://github.com/Phredward/throttle

[Thomas Schwinge]

Hi!

Ha, I remembered right that I had seen such a problem reported before...

On 2018-01-15T12:31:04+0100, Bruno Haible <br...@clisp.org> wrote:
> I tried:
>> # tar cf - directory | ssh br...@10.0.2.2 tar xf -
>> It hangs after transferring 1.6 GB. I.e. no more data arrives within 15 minutes.

You were lucky: for me it stopped much earlier (a few dozen MiB) -- the
file I'm trying to transfer is just 460 MiB in size. ;-)

> Found a workaround: Throttling of the bandwidth.
> - Throttling at the network adapter level [1] is not applicable to Hurd.
> - The 'throttle' program [2] is no longer available.
> - But a replacement program [3] is available.

Or use the '--bwlimit' functionality of 'rsync', or the '--rate-limit'
functionality of 'pv', which are often already packaged, readily

available.

> The command that worked for me (it limits the bandwidth to 1 MB/sec):
>
> # tar cf - directory | ~/throttle.py --bandwidth 1024576 | ssh br...@10.0.2.2 tar xf -

I thus tried:

$ pv --rate-limit 1M [file] | ssh [...] 'cat > [file]'

..., which crashed after 368 MiB. After rebooting, a 'rsync -Pa
--inplace --bwlimit=500K' was then able to complete the transfer; the two
files' checksums do match.

> But really, this is only a workaround. It smells like a bug in ssh or the Hurd.

As all networking seems to go down, maybe it's that the GNU Hurd
networking stack ('pfinet', 'netdde') gets "overwhelmed" by that much
data.


(That was on a Debian GNU/Hurd installation that's more than a one year
out of date, so there's a -- slight? ;-D -- chance that this has been
fixed already.)


Grüße
Thomas

[Bruno Haible]

Thomas Schwinge wrote:
> I thus tried:
>
> $ pv --rate-limit 1M [file] | ssh [...] 'cat > [file]'
>
> ..., which crashed after 368 MiB.

Probably you need a rate limit smaller than 1 MB/sec, then?

Bruno