SSH dropping/reconnecting

248 views
Skip to first unread message

rossdav

unread,
Mar 25, 2010, 5:29:41 PM3/25/10
to DataStorageUnit
John has tried to help with this, but no luck thus far, so I am
posting to this group for insight. This issue arose only after John
changed locations for his hardware, so I know it is a network-related
issue, and any insight is welcome.

I use duplicity and rsync and scp in combination for my backups -
some things I am not worried about encrypting, so I use RSYNC or
Filezilla (SCP). Other things I do care about, hence duplicity.

Here's what is happening. I can connect to the data storage server
without any problems. What occurs, though, is my connection is dropped
after about 30 seconds. If using filezilla, it then reconnects,
starts transferring again, then drops after 30 seconds or so, etc..
Rsync is not as sophisticated, so it just fails when the connection is
dropped.
======================
Command: reput "/<filename>" "<filename>"
Status: reput: restarting at file position 387445285
Status: local:/vms/<filename> => remote:/home/.../<filename>
Error: Connection timed out
Error: File transfer failed after transferring 1,421,312 bytes in 29
seconds
Status: Connecting to storage.datastorageunit.com...
Response: fzSftp started
Command: open "xxxx...@storage.datastorageunit.com" 22
Command: Pass: *********
Status: Connected to storage.datastorageunit.com
Status: Starting upload of /<filname>
Command: cd "/home/xxxxxxx"
Response: New directory is: "/home/dxxxxxx"
Status: Retrieving directory listing...
Command: ls
Status: Listing directory /home/xxxxxxxxx
Command: reput "/<filename>" "<filename>"
Status: reput: restarting at file position 388891173
Status: local:/<filenam> => remote:/home/...../<filename>
Error: Connection timed out
Error: File transfer failed after transferring 1,708,032 bytes in 30
seconds
Status: Connecting to storage.datastorageunit.com...
Response: fzSftp started
===========================

Suggestions include using '-C' switch on scp (can't do in filezilla)
or '-z' option in rsync; this appears to HELP, but not completely
SOLVE, the dropped connection issue. Both flags refer to compressing
the datastream, and indeed it seems work longer and faster, but I
still get dropped at some point.

Someone else on another board suggested checking duplex/speed settings
to see if they 'match' the remote host, but I am dubious of this
helping my situation.

These symptoms have occurred at work (using both a cabled and a
wireless connection), a hotspot (wireless) and home (wireless) on this
same laptop. T61P, Fedora 12 (64-bit), 6GB RAM

Thanks for any insight.

Martin Larsen

unread,
Mar 25, 2010, 5:53:35 PM3/25/10
to datasto...@googlegroups.com
Hi,

I don't know what causes your troubles and I have so far not encountered any problems myself after the location change, however I have a couple of suggestions :

rsync has the --partial and --partial-dir=DIR options that allow an interrupted file transfer to resume instead of starting over again with that file. Since rsync will return with an error code if it is interrupted, you could make a loop in which you repeatedly started the rsync'ing until it was finished.

My other suggestion is taking a look at lftp which supposedly handles connection problems very well. I have never myself had such connection problems though, but I find it a very very good backup/mirroring software.

Martin

knnniggett

unread,
Mar 28, 2010, 1:32:55 PM3/28/10
to DataStorageUnit
From my own past experience, I can recall having transfer issues
remotely similar to what you describe when my MTU was set incorrectly.
Are you using the standard MTU of 1500 or are you using something
else? Check both your computer and your router.

Also, should we assume that John checked the server log files for
abnormalities during your file transfer attempts?

On Mar 25, 4:53 pm, Martin Larsen <marlar...@gmail.com> wrote:
> Hi,
>
> I don't know what causes your troubles and I have so far not encountered any
> problems myself after the location change, however I have a couple of
> suggestions :
>
> rsync has the --partial and --partial-dir=DIR options that allow an
> interrupted file transfer to resume instead of starting over again with that
> file. Since rsync will return with an error code if it is interrupted, you
> could make a loop in which you repeatedly started the rsync'ing until it was
> finished.
>
> My other suggestion is taking a look at lftp which supposedly handles
> connection problems very well. I have never myself had such connection
> problems though, but I find it a very very good backup/mirroring software.
>
> Martin
>

John Wooton

unread,
Mar 28, 2010, 1:46:58 PM3/28/10
to datasto...@googlegroups.com
I've not seen anything abnormal on my end.  However, I do want to let everyone know that there have been some downtime in the last couple days that result from hosting provider issues.  However, they were a few hours in length and happened twice I believe.  This wouldn't correspond with a disconnect that immediately lets you reconnect.  Also, so far no other subscriber has reported the same issues.  I think if it were server side, there would be several more complaints.  The service currently has approximately 75 active subscribers and avarages about 8-12 actively connected subscribers at any given time.



To unsubscribe from this group, send email to datastorageunit+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.

knnniggett

unread,
Mar 28, 2010, 3:41:04 PM3/28/10
to DataStorageUnit
Because Fedora 12 is what one could call a "bleeding edge" operating
system, it's possible that the OS is using some underlying tool or
library that is newer than what is on the other end.
I admit this is a complete WAG, but perhaps the client is using some
parameter that the server does not understand completely. Something
like this could show up in the log on the server and not the client if
the client is not aware there is a problem. ...of course if this were
the case you would likely see a lot of reports of similar problems in
the Fedora forums (I have not checked).

Anyway, rossdav, would you be willing to try your transfer while
booted from an Ubuntu 9.10 Live cd? That would match the OS at the
other end if I recall. That would certainly confirm or deny the issue
has got something to do with Fedora.


On Mar 28, 12:46 pm, John Wooton <j...@datastorageunit.com> wrote:
> I've not seen anything abnormal on my end.  However, I do want to let
> everyone know that there have been some downtime in the last couple days
> that result from hosting provider issues.  However, they were a few hours in
> length and happened twice I believe.  This wouldn't correspond with a
> disconnect that immediately lets you reconnect.  Also, so far no other
> subscriber has reported the same issues.  I think if it were server side,
> there would be several more complaints.  The service currently has
> approximately 75 active subscribers and avarages about 8-12 actively
> connected subscribers at any given time.
>

john

unread,
Mar 28, 2010, 5:59:18 PM3/28/10
to datasto...@googlegroups.com
I think that's an excellent suggestion, however I was under the assumption that he had attempted a filezilla transfer from a windows and linux machine.  I have did a lot of transfers from filezilla as well as CoreFTP with no problems in WinXP and windows7.  Initially my biggest suspicion was with his ISP...but he says he has the issue at other locations using a different ISP also...which kills my ISP theory if that's the case.

I do think the LiveCD test would be a good test...and not a difficult test to run either. 
 or reply to this email with the words "REMOVE
> > ME" as the subject.

To unsubscribe from this group, send email to datastorageunit+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.

chris

unread,
Mar 30, 2010, 2:00:28 AM3/30/10
to DataStorageUnit
I haven't had any ssh issues. Maybe you could install wireshark or
tcpdump and see what's going on over the wire. If you don't know how
to read the output, you could save it and email it. I have experience
with protocol analyzers.

John - DataStorageUnit - Owner

unread,
Mar 31, 2010, 8:21:45 PM3/31/10
to DataStorageUnit
Just wanted to follow up on this issue...let us know how the "LiveCD"
test goes.

jmayoralas

unread,
Apr 5, 2010, 12:15:11 PM4/5/10
to DataStorageUnit
I'm having some disconnect problems too, but not exactly as mentioned
above.

I'm using sshfs to mount my DataStorageUnit account on a local
directory, then encfs to mount the unencrypted volume on another local
directory and do an rsync to transfer the changes.

rsync says "sending incremental file list", after a while, rsync
starts to transfer new files (each file has 10 MB aprox) and during
the first transfer, after a minute or so (not exactly 60 secs), my
local computer drops the connection with the message:

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]:
Broken pipe (32)

After that, a mount command shows that the sshfs filesystem is not
mounted anymore, and the encfs filesystem is still mounted but
pointing to none, and obviously, I loss the access to both, my
unencrypted volume and my remote sshfs filesystem.

I was able to transfer 48 gigs of data before this problem, but in the
last week or so, I'm unable to transfer even 10 MB of new data. I haven
´t changed anything on my side, so, I don't know what to look for.

On 1 abr, 02:21, John - DataStorageUnit - Owner

John Wooton

unread,
Apr 5, 2010, 1:23:08 PM4/5/10
to datasto...@googlegroups.com
yea...I don't think your issue is related to the other since his was so consistent about failing in such short time frames.  Right now it shows you have a live session that's been connected for 11 minutes thus far.  However, doesn't look like any data is being transferred to the server.

If you want to keep testing I can monitor some logs to see if I can see anything happen on the server end.  I'll keep looking occasionally over the next hour or so.

Jose Luis Fernandez-Mayoralas

unread,
Apr 5, 2010, 1:32:50 PM4/5/10
to datasto...@googlegroups.com
I tried with some small files mounting directly the remote volumen with sshfs and no encfs. After a couple of files of 600 KB, the connection dropped again.

So don't seem to be related with transfers of large amount of data.

Rigth now I'm not at home, so by now I can't do more tests.

Saludos...

John Wooton

unread,
Apr 5, 2010, 2:33:54 PM4/5/10
to datasto...@googlegroups.com
ok...I'm glad you tried without encFS to eliminate that as a possibility.  So far, have all your tests been with Rsync as a client?  Can you try FileZilla or some other SFTP client? 

Also, what type of connection do you have at home...  DSL, Cable, Satellite?

I know you're out for the time being...just get back with me when you get a chance.

knnniggett

unread,
Apr 5, 2010, 11:01:12 PM4/5/10
to DataStorageUnit
When you connect to dsu, are you using any special mount options with
sshfs,encfs,rysnc, etc?
For example, sshfs has a mount option called "reconnect".

Turns out I can reboot my router, mid-transfer, and when the router
comes back up, my rsync transfer picks up right where it left off.

On Apr 5, 1:33 pm, John Wooton <j...@datastorageunit.com> wrote:
> ok...I'm glad you tried without encFS to eliminate that as a possibility.
>  So far, have all your tests been with Rsync as a client?  Can you try
> FileZilla or some other SFTP client?
>
> Also, what type of connection do you have at home...  DSL, Cable, Satellite?
>
> I know you're out for the time being...just get back with me when you get a
> chance.
>
> On Mon, Apr 5, 2010 at 1:32 PM, Jose Luis Fernandez-Mayoralas
> <y...@jluis.me>wrote:
>
> > I tried with some small files mounting directly the remote volumen with
> > sshfs and no encfs. After a couple of files of 600 KB, the connection
> > dropped again.
>
> > So don't seem to be related with transfers of large amount of data.
>
> > Rigth now I'm not at home, so by now I can't do more tests.
>
> > Saludos...
>

> > El 05/04/2010, a las 19:23, John Wooton <j...@datastorageunit.com>


> > escribió:
>
> > yea...I don't think your issue is related to the other since his was so
> > consistent about failing in such short time frames.  Right now it shows you
> > have a live session that's been connected for 11 minutes thus far.  However,
> > doesn't look like any data is being transferred to the server.
>
> > If you want to keep testing I can monitor some logs to see if I can see
> > anything happen on the server end.  I'll keep looking occasionally over the
> > next hour or so.
>

> >> > > > > > > <http://unsubscribegooglegroups.com>


> >> unsubscribegooglegroups.com or reply to this email with the words "REMOVE
> >> > > > > > > ME" as the subject.
>

> >> > > > > To unsubscribe from this group, send email to datastorageunit+<http://unsubscribegooglegroups.com>

Jose Luis Fernandez-Mayoralas

unread,
Apr 6, 2010, 5:34:18 AM4/6/10
to datasto...@googlegroups.com
I'm right now at the work, and I'm doing some more tests.

Setting the ServerAliveInterval option in sshfs to something like 15 (default is 120) doesn't help.

Also, setting the reconnect option keeps dropping the connection (knnniggett, thanks for the tip).

Even a mere cp over the sshfs filesystem (no encfs, no rsync) breaks after a few bytes transferred. Nothing in messages, syslog, dmesg... I'm really blocked right now.

I'm using Ubuntu 9.10, and my sshfs command to mount is:

sshfs jmayo...@storage.datastorageunit.com:copias /mnt/dsucopias/ -o allow_other,uid=1000,reconnect

EncFS line:

encfs /mnt/dsucopias/ /home/jluis/dsucopias/

The rsync command:

rsync -rltDvz --bwlimit=60 /mnt/disco_externo/Fotos /home/jluis/dsucopias

My /etc/ssh/ssh_config

Host *
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials no
    ServerAliveInterval 120

I have a Cable internet provider (12mb/1mb) (this is the first time I have problems like this), and all my test are using rsync as a client.

When I get back to home, I'll try to do some test from another computer, to discard problems with this installation.

--
To unsubscribe, reply using "remove me" as the subject.

knnniggett

unread,
Apr 6, 2010, 9:14:22 AM4/6/10
to DataStorageUnit
Here are a couple more suggestions that seem to work well on my
system:
While I too have set the ServerAliveInterval (mine is at 60), I also
have ServerAliveCountMax set to 100 (default is only 3).
You might try turning compression on with the "compression yes" option
in you ssh_config and/or calling sshfs with the "-C" option.

If you are behind a router at home, can you temporarily take the
router out of the equation and plug your machine directly to the
internet? Does that make a difference?

On Apr 6, 4:34 am, Jose Luis Fernandez-Mayoralas <y...@jluis.me>
wrote:


> I'm right now at the work, and I'm doing some more tests.
>
> Setting the ServerAliveInterval option in sshfs to something like 15
> (default is 120) doesn't help.
>
> Also, setting the reconnect option keeps dropping the connection
> (knnniggett, thanks for the tip).
>
> Even a mere cp over the sshfs filesystem (no encfs, no rsync) breaks after a
> few bytes transferred. Nothing in messages, syslog, dmesg... I'm really
> blocked right now.
>
> I'm using Ubuntu 9.10, and my sshfs command to mount is:
>

> sshfs jmayora...@storage.datastorageunit.com:copias /mnt/dsucopias/ -o


> allow_other,uid=1000,reconnect
>
> EncFS line:
>
> encfs /mnt/dsucopias/ /home/jluis/dsucopias/
>
> The rsync command:
>
> rsync -rltDvz --bwlimit=60 /mnt/disco_externo/Fotos /home/jluis/dsucopias
>
> My /etc/ssh/ssh_config
>
> Host *
>     SendEnv LANG LC_*
>     HashKnownHosts yes
>     GSSAPIAuthentication yes
>     GSSAPIDelegateCredentials no
>     ServerAliveInterval 120
>
> I have a Cable internet provider (12mb/1mb) (this is the first time I have
> problems like this), and all my test are using rsync as a client.
>
> When I get back to home, I'll try to do some test from another computer, to
> discard problems with this installation.
>

> ...
>
> read more »

John Wooton

unread,
Apr 6, 2010, 9:18:55 AM4/6/10
to datasto...@googlegroups.com
I think maybe trying from a LiveCD would be a good test.  

I see where you connected again from a different IP a couple times today:
Here is the log for the last one:

(I'm assuming your attempt also dropped here...if so, that's a pritty quick disconnect)
Apr  6 08:54:01 data2 sshd[29532]: Accepted password for **yourusername** from 195.XXX.XXX.XXX port 2449 ssh2
Apr  6 08:54:01 data2 sshd[29532]: pam_unix(sshd:session): session opened for user **yourusername** by (uid=0)
Apr  6 08:57:19 data2 sshd[29532]: pam_unix(sshd:session): session closed for user **yourusername**


If they have a fast connection there, maybe you can download a LiveCD ISO to try from there, and home when you get back.  I think the LiveCD test is a good one, as it's easy for someone else to use the same ISO version and boot up to confirm whether or not they get similar results.

Also, I very much appreciate everyone else giving input to try and help.  You guys have given some good suggestions.

Jose Luis Fernandez-Mayoralas

unread,
Apr 6, 2010, 11:03:42 AM4/6/10
to datasto...@googlegroups.com
Yes, well, in this last connection I was using ssh in terminal mode, I did not transfer anything. Always transfer data from my home computer, not from computers at the work.

I just tried to send data using a Mac, MacFuse 2.1.5 and MacFusion 2.0.3. This computer is in the same network that the linux box I was using for tests. I can mount the remote filesystem in my local Mac OS X, but when I try to copy something, I get the same results that in my linux box. Failed copy operation after 700-800 KB transferred, and filesystem unmounted without any warning.

This discards problems with my linux installation and points to some issues between my local internet provider (or my own router) and the DSU servers.

I'll try to do some tests using another internet provider to discard the local internet provider too, but will be hard to test removing my own router from the equation.

Jose Luis Fernandez-Mayoralas

unread,
Apr 7, 2010, 4:20:39 AM4/7/10
to datasto...@googlegroups.com
I did some more research on this topic with the following results:

- cp over sshfs from linux box internet provider 1 (KO)
- cp over sshfs from snow leopard internet provider 1 (KO)
- cp over sshfs from snow leopard internet provider 2 (KO)
- cp over sshfs from vmware linux box internet provider 1 (KO)
- cp over sshfs from vmware linux box internet provider 3 (KO)
- scp from vmware linux box internet provider 3 (KO)
- scp from linux box internet provider 1 (KO)
- sftp from snow leopard internet provider 1 (Cyberduck client) (OK)

The only test that seem to work properly is the last one. I don't know how cyberduck (and sftp protocol) works, but for me this is not a solution.

I Spoofed the traffic with tcpdump and wireshark. There is a point in the communication where the server request for a retransmission of a previous datagram, this datagram is retransmitted by the client, but the client doesn't receive an ACK from te server, then the client keeps retransmitting this datagram without any response from the server, after 30 secs of retries the client inits a new connection (this is the action of the reconnect parameter in sshfs), and server acknowledges this new connection and do a new key negotiation with the client. But in this point, sshfs is giving I/O errors and the transfer is aborted, and the connection closed.

I don't know what other tests I can do... but all the tests points to a SSH layer communication problem, and this started when DSU changed something, hardware, hosting, I don't know.

John Wooton

unread,
Apr 7, 2010, 8:49:34 AM4/7/10
to datasto...@googlegroups.com
how about giving rsync a try without tunnelling through sshfs.  You can mount your encfs directory on the server instead of doing that from home.  Is that acceptable?

About service changes...
The only service change that has taken place, was moving to a new Service provider...so yes, the ISP did change.  

hunsaker

unread,
Apr 8, 2010, 1:11:35 AM4/8/10
to DataStorageUnit
I am using rsync (over ssh) and keep getting disconnected. I have
restarted several times in a row and it does not take long. Using
fully up-to-date Fedora 12.

rsync -av --progress --partial folder host:

"""
Write failed: Broken pipekB/s 0:00:06

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]:
Broken pipe (32)

rsync: connection unexpectedly closed (7382 bytes received so far)
[sender]
rsync error: error in rsync protocol data stream (code 12) at
io.c(601) [sender=3.0.7]
"""

Using --bwlimit with low KBps seems to let it run a little longer
(eventually always fails within 15-30 minutes). Not sure if there is
any actual correlation though.

hunsaker

unread,
Apr 8, 2010, 1:27:53 AM4/8/10
to DataStorageUnit
I have confirmed there definitely is a correlation between value of
bwlimit and amount of data transferred before connection drops. The
lower the value passed to bwlimit, the more data transferred before
connection failure.

Jose Luis Fernandez-Mayoralas

unread,
Apr 8, 2010, 1:43:34 AM4/8/10
to datasto...@googlegroups.com
I can confirm what hunsaker says.

At least now I can send my new data to my backup storage, with many many retries, but at the end of the day, my data arrives. I don't like the idea of mount my unencrypted volumen on the server, but if this is the only way...

John, I think your install has some problem in the communications layer, and problems started when you changed the hosting.

hunsaker

unread,
Apr 8, 2010, 1:57:06 AM4/8/10
to DataStorageUnit
Fedora 12 has version 3.0.7 and Ubuntu on the servers has version
3.0.6 installed. I could not find any reports of the same problem
being due to the version difference.

I have not seen the issue with other servers that are Fedora 12 or
CentOS 5 (running rsync version 2.6.8). I do not have another Ubuntu
server to test.


Martin Larsen

unread,
Apr 8, 2010, 4:05:59 AM4/8/10
to datasto...@googlegroups.com
I have never had any connection problems on either of the hosting
providers. I transfer gigabytes of data without interruption and it
takes weeks.

However, my upload speed is low (512kb/s) so perhaps the problem is not
triggered at that speed?

I will make a test upload from one of my work servers with lots of
bandwidth and see what happens.

Jose Luis Fernandez-Mayoralas

unread,
Apr 8, 2010, 4:11:33 AM4/8/10
to datasto...@googlegroups.com
Hmmm... after read this, I remembered that I increase my upload ration from 512kb to 1 mb two weeks ago... maybe the problem are related with this increment.

Martin Larsen

unread,
Apr 8, 2010, 4:28:05 AM4/8/10
to datasto...@googlegroups.com
Ok, I just tried rsyncing a 500 MB file from my work server with an avarage upload speed at 4.5 MB/S. There was no connection problems. It was not on a encfs filesystem, though. Are the problems only related to using encfs?

Jose Luis Fernandez-Mayoralas

unread,
Apr 8, 2010, 4:33:40 AM4/8/10
to datasto...@googlegroups.com
Nop... a plain rsync against the server, and no encfs fails too.

John Wooton

unread,
Apr 8, 2010, 8:47:00 AM4/8/10
to datasto...@googlegroups.com
I personally would still like to see the "LiveCD" test happen.  I have noticed that so far all the accounts with problems seem to have Fedora in common, although someone did mention having the same issue with a Mac.  However, I'd still like to see a live CD test with Ubuntu 9.10 specifically.  I don't want anyone to think I'm blaming Fedora already...It would be good to know if it does make a difference though.

Jose Luis Fernandez-Mayoralas

unread,
Apr 8, 2010, 8:56:03 AM4/8/10
to datasto...@googlegroups.com
John, my main linux box is Ubuntu 9.10, and the ssh, rsync versions are same that yours.

John Wooton

unread,
Apr 8, 2010, 10:29:55 AM4/8/10
to datasto...@googlegroups.com
I think the other two guys should try it then.  The only reason I still say so is because you have the issue following them no matter what ISP they try from apparently.  Their reports are that they have this issue from the same pc at home and work, so I still think it's a relevant test...for them at least.  And in my opinion, it's still not going to hurt even if you do have the same OS installed since a live CD is still more of a controlled variable...(even though one would expect the same results in your case having that as your OS.)  That being said...it's still up to you as to whether or not you want to give it a try.  All that being said...I do lean toward it being an issue along the internet at some point...not likely with the Server Side ISP based on the vast majority of other Subscribers not having any issues.  

I also just wanted to mention that at this time there are 56 paying subscribers currently...and another 20 or so Trial Accounts.  6 people are connected and uploading as I speak.  3 people are reporting this issue, only 2 of which are actively working on it that I know of....(none of the six connected are any of those reporting issues, so I take it that those transfers are going fine).  The other hasn't logged in since late March and didn't give a lot of feedback on tests suggested by people in the Group so for all we know he's good at this point.  That being the case, it's hard for me to lean toward the problem being with the ISP.  However, I am definitely open to the possibility that there is an issue resulting from data in your specific case now taking a different route to a new ISP since the server move.  There are a lot of variables between any 2 points along the internet.  This may just be one of those situations where the particular path your client is taking is running into some issues along the way now...somewhere between the two points that is causing just enough jitter, latency, or whatever to cause rsync to drop.  In one of the two "active" cases...the connection appears to be coming from Spain...which would obviously be a lot of hops.

That being the case, other protocols...and maybe even different versions of the same protocol, may be more resilient to what is happening.   And since we have no control over the situation between point A and B...all we can try is things like the LiveCD test, or the bwlimit setting that was brought up....(which By-The-Way was a really good find).  

Let me know what you think. 

John - DataStorageUnit - Owner

unread,
Apr 8, 2010, 10:46:35 AM4/8/10
to DataStorageUnit
One more thing I wanted to add real quick...as this seems like a
"very" relevant time to bring it up. Although the physical location
isn't changing...the ISP for the co-location services DataCenter will
be changing their ISP in the upcoming months. I don't have an exact
date yet, but I have been told it will be before April if I'm not
mistaken. Reasoning is to increase bandwidth for their entire
facility.

That being said, I hope things don't act up too much for that
switch...and on a positive note, maybe the few folks that are having
issues may also see their issues disappear with this change.

hunsaker

unread,
Apr 8, 2010, 11:15:46 AM4/8/10
to DataStorageUnit
I am downloading Ubuntu 9.10 now. However, in the meantime...

Running rsync with multiple v options shows a problem in the io code:

"""
rsync: connection unexpectedly closed (8880 bytes received so far)
[sender]
[sender] _exit_cleanup(code=12, file=io.c, line=601): entered
rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]
[sender] _exit_cleanup(code=12, file=io.c, line=601): about to call
exit(255)
"""

I added the --blocking-io option which did not help.

Thinking it may be a problem with ssh now, I tried (as suggested
earlier, I think, ha):

"""
$ scp -r folder storage2.datastorageunit.com:foldertest
file1
95% 2320KB 1.7MB/s 00:00 ETAWrite failed: Broken pipe
lost connection
"""

Therefore it must be a problem with ssh and how fast the data is
transferring.

hunsaker

unread,
Apr 8, 2010, 11:25:45 AM4/8/10
to DataStorageUnit
ssh is seeing the connection drop:

"""
debug3: Wrote -1 bytes for a total of 277109


Write failed: Broken pipe
lost connection
"""

I'll try Ubuntu later today and post the results. If it works with
Ubuntu then I'll assume Fedora 12 has bad code somewhere and I'll
write a nice rsync retry script.

knnniggett

unread,
Apr 8, 2010, 8:58:37 PM4/8/10
to DataStorageUnit
I have not had any upload issues either, but maybe this info will help
someone.
While initially I was uploading to dsu using a basically stock CentOS
5.4 x86_64 system and cable broadband with advertised 2Mbps upload,
due to an issue not related to this thread I have custom rolled my own
latest and greatest RPM's for encfs, fuse, sshfs, rsync and their
dependencies. Upgrading didn't change the reliably of the transfer so
IMO I don't think this particular issue has got to do with the
differences in software versions between client and server. However,
with that said, I am willing to provide those RPM's to anyone who may
be interested.

The comments about using bwlimit to slow down the transfer, to me,
seem to suggest that you might want to temporarily take your router
out of the picture. The way I see it, your router is about the only
thing that you can control between the source and destination so why
not just try removing it? If it makes no difference then there is no
harm done. If it does make a difference, now we can focus our
troubleshooting the router itself.

Lastly, I don't know if anyone has noticed, but I have available for
download from the files section of this forum a shell script that I
have written for my system. I named it dsu_backup. It uses encfs or
truecrypt to locally mount your encrypted volume and then it uses
rsync to transfer the files. You have to set up a few things and
populate the script with your info, but once you do that you can run
it as a cron job and forget about it. Feel free to use it, critique
it, or use pieces of it to make your own script. Take a special look
at the parameters I am passing to sshfs, encfs, and rsync... they seem
to work really well for me anyway. Also, please feel free to ask
questions if you have trouble setting it up.

On Apr 8, 3:11 am, Jose Luis Fernandez-Mayoralas <y...@jluis.me>
wrote:

Jose Luis Fernandez-Mayoralas

unread,
Apr 9, 2010, 2:21:26 AM4/9/10
to datasto...@googlegroups.com
On Thu, Apr 8, 2010 at 4:29 PM, John Wooton <jo...@datastorageunit.com> wrote:
I think the other two guys should try it then.  The only reason I still say so is because you have the issue following them no matter what ISP they try from apparently.  Their reports are that they have this issue from the same pc at home and work, so I still think it's a relevant test...for them at least.  And in my opinion, it's still not going to hurt even if you do have the same OS installed since a live CD is still more of a controlled variable...(even though one would expect the same results in your case having that as your OS.)  That being said...it's still up to you as to whether or not you want to give it a try.  All that being said...I do lean toward it being an issue along the internet at some point...not likely with the Server Side ISP based on the vast majority of other Subscribers not having any issues.  


I don't know if you are refering to me in this quote, but I think a vmware image test in two distinct environments is equivalent to do it with a LiveCD as you suggest. I can't see the LiveCD advantage over the virtual machine, and as I explained above, that test doesn't make any difference. I get connections drops yes or .... yes.

between the two points that is causing just enough jitter, latency, or whatever to cause rsync to drop.  In one of the two "active" cases...the connection appears to be coming from Spain...which would obviously be a lot of hops.

That's my case. I'm from Spain and I'm assuming now my connections problems. I did tons of tests, none of them produces good results. It's up to me to decide either if  I continue using your services or start to look for another solution for my online backups. I'll wait for the upcoming changes in your ISP to see if things get better.

That said, I want to thank you for the effort you do in this group for give a proper support. Keep up the good work.

jmayoralas

unread,
May 13, 2010, 8:18:31 AM5/13/10
to DataStorageUnit
Only to let you know that I solved my drop connection problem in
transfers over ssh protocol.

I was using a Linksys WRT320N as my main router, with firmware dd-wrt
installed on it. I changed the router for a Fonera 2.0n, based on
OpenWRT firmware and now the connection is solid and stable.

It's weird that the problems with the linksys router only happens with
datastorageunit.com, and no with other ssh servers, but... at least,
now it's working right.

On 9 abr, 08:21, Jose Luis Fernandez-Mayoralas <y...@jluis.me> wrote:

John Wooton

unread,
May 13, 2010, 8:24:08 AM5/13/10
to datasto...@googlegroups.com
yea...the part about it being specific is odd.  Glad things are sorted out.
Reply all
Reply to author
Forward
0 new messages