I use duplicity and rsync and scp in combination for my backups -
some things I am not worried about encrypting, so I use RSYNC or
Filezilla (SCP). Other things I do care about, hence duplicity.
Here's what is happening. I can connect to the data storage server
without any problems. What occurs, though, is my connection is dropped
after about 30 seconds. If using filezilla, it then reconnects,
starts transferring again, then drops after 30 seconds or so, etc..
Rsync is not as sophisticated, so it just fails when the connection is
dropped.
======================
Command: reput "/<filename>" "<filename>"
Status: reput: restarting at file position 387445285
Status: local:/vms/<filename> => remote:/home/.../<filename>
Error: Connection timed out
Error: File transfer failed after transferring 1,421,312 bytes in 29
seconds
Status: Connecting to storage.datastorageunit.com...
Response: fzSftp started
Command: open "xxxx...@storage.datastorageunit.com" 22
Command: Pass: *********
Status: Connected to storage.datastorageunit.com
Status: Starting upload of /<filname>
Command: cd "/home/xxxxxxx"
Response: New directory is: "/home/dxxxxxx"
Status: Retrieving directory listing...
Command: ls
Status: Listing directory /home/xxxxxxxxx
Command: reput "/<filename>" "<filename>"
Status: reput: restarting at file position 388891173
Status: local:/<filenam> => remote:/home/...../<filename>
Error: Connection timed out
Error: File transfer failed after transferring 1,708,032 bytes in 30
seconds
Status: Connecting to storage.datastorageunit.com...
Response: fzSftp started
===========================
Suggestions include using '-C' switch on scp (can't do in filezilla)
or '-z' option in rsync; this appears to HELP, but not completely
SOLVE, the dropped connection issue. Both flags refer to compressing
the datastream, and indeed it seems work longer and faster, but I
still get dropped at some point.
Someone else on another board suggested checking duplex/speed settings
to see if they 'match' the remote host, but I am dubious of this
helping my situation.
These symptoms have occurred at work (using both a cabled and a
wireless connection), a hotspot (wireless) and home (wireless) on this
same laptop. T61P, Fedora 12 (64-bit), 6GB RAM
Thanks for any insight.
Also, should we assume that John checked the server log files for
abnormalities during your file transfer attempts?
On Mar 25, 4:53 pm, Martin Larsen <marlar...@gmail.com> wrote:
> Hi,
>
> I don't know what causes your troubles and I have so far not encountered any
> problems myself after the location change, however I have a couple of
> suggestions :
>
> rsync has the --partial and --partial-dir=DIR options that allow an
> interrupted file transfer to resume instead of starting over again with that
> file. Since rsync will return with an error code if it is interrupted, you
> could make a loop in which you repeatedly started the rsync'ing until it was
> finished.
>
> My other suggestion is taking a look at lftp which supposedly handles
> connection problems very well. I have never myself had such connection
> problems though, but I find it a very very good backup/mirroring software.
>
> Martin
>
To unsubscribe from this group, send email to datastorageunit+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.
Anyway, rossdav, would you be willing to try your transfer while
booted from an Ubuntu 9.10 Live cd? That would match the OS at the
other end if I recall. That would certainly confirm or deny the issue
has got something to do with Fedora.
On Mar 28, 12:46 pm, John Wooton <j...@datastorageunit.com> wrote:
> I've not seen anything abnormal on my end. However, I do want to let
> everyone know that there have been some downtime in the last couple days
> that result from hosting provider issues. However, they were a few hours in
> length and happened twice I believe. This wouldn't correspond with a
> disconnect that immediately lets you reconnect. Also, so far no other
> subscriber has reported the same issues. I think if it were server side,
> there would be several more complaints. The service currently has
> approximately 75 active subscribers and avarages about 8-12 actively
> connected subscribers at any given time.
>
or reply to this email with the words "REMOVE > > ME" as the subject. To unsubscribe from this group, send email to datastorageunit+unsubscribegooglegroups.com or reply to this email with the words "REMOVE ME" as the subject.
I'm using sshfs to mount my DataStorageUnit account on a local
directory, then encfs to mount the unencrypted volume on another local
directory and do an rsync to transfer the changes.
rsync says "sending incremental file list", after a while, rsync
starts to transfer new files (each file has 10 MB aprox) and during
the first transfer, after a minute or so (not exactly 60 secs), my
local computer drops the connection with the message:
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]:
Broken pipe (32)
After that, a mount command shows that the sshfs filesystem is not
mounted anymore, and the encfs filesystem is still mounted but
pointing to none, and obviously, I loss the access to both, my
unencrypted volume and my remote sshfs filesystem.
I was able to transfer 48 gigs of data before this problem, but in the
last week or so, I'm unable to transfer even 10 MB of new data. I haven
´t changed anything on my side, so, I don't know what to look for.
On 1 abr, 02:21, John - DataStorageUnit - Owner
Turns out I can reboot my router, mid-transfer, and when the router
comes back up, my rsync transfer picks up right where it left off.
On Apr 5, 1:33 pm, John Wooton <j...@datastorageunit.com> wrote:
> ok...I'm glad you tried without encFS to eliminate that as a possibility.
> So far, have all your tests been with Rsync as a client? Can you try
> FileZilla or some other SFTP client?
>
> Also, what type of connection do you have at home... DSL, Cable, Satellite?
>
> I know you're out for the time being...just get back with me when you get a
> chance.
>
> On Mon, Apr 5, 2010 at 1:32 PM, Jose Luis Fernandez-Mayoralas
> <y...@jluis.me>wrote:
>
> > I tried with some small files mounting directly the remote volumen with
> > sshfs and no encfs. After a couple of files of 600 KB, the connection
> > dropped again.
>
> > So don't seem to be related with transfers of large amount of data.
>
> > Rigth now I'm not at home, so by now I can't do more tests.
>
> > Saludos...
>
> > El 05/04/2010, a las 19:23, John Wooton <j...@datastorageunit.com>
> > escribió:
>
> > yea...I don't think your issue is related to the other since his was so
> > consistent about failing in such short time frames. Right now it shows you
> > have a live session that's been connected for 11 minutes thus far. However,
> > doesn't look like any data is being transferred to the server.
>
> > If you want to keep testing I can monitor some logs to see if I can see
> > anything happen on the server end. I'll keep looking occasionally over the
> > next hour or so.
>
> >> > > > > > > <http://unsubscribegooglegroups.com>
> >> unsubscribegooglegroups.com or reply to this email with the words "REMOVE
> >> > > > > > > ME" as the subject.
>
> >> > > > > To unsubscribe from this group, send email to datastorageunit+<http://unsubscribegooglegroups.com>
--To unsubscribe, reply using "remove me" as the subject.
If you are behind a router at home, can you temporarily take the
router out of the equation and plug your machine directly to the
internet? Does that make a difference?
On Apr 6, 4:34 am, Jose Luis Fernandez-Mayoralas <y...@jluis.me>
wrote:
> I'm right now at the work, and I'm doing some more tests.
>
> Setting the ServerAliveInterval option in sshfs to something like 15
> (default is 120) doesn't help.
>
> Also, setting the reconnect option keeps dropping the connection
> (knnniggett, thanks for the tip).
>
> Even a mere cp over the sshfs filesystem (no encfs, no rsync) breaks after a
> few bytes transferred. Nothing in messages, syslog, dmesg... I'm really
> blocked right now.
>
> I'm using Ubuntu 9.10, and my sshfs command to mount is:
>
> sshfs jmayora...@storage.datastorageunit.com:copias /mnt/dsucopias/ -o
> allow_other,uid=1000,reconnect
>
> EncFS line:
>
> encfs /mnt/dsucopias/ /home/jluis/dsucopias/
>
> The rsync command:
>
> rsync -rltDvz --bwlimit=60 /mnt/disco_externo/Fotos /home/jluis/dsucopias
>
> My /etc/ssh/ssh_config
>
> Host *
> SendEnv LANG LC_*
> HashKnownHosts yes
> GSSAPIAuthentication yes
> GSSAPIDelegateCredentials no
> ServerAliveInterval 120
>
> I have a Cable internet provider (12mb/1mb) (this is the first time I have
> problems like this), and all my test are using rsync as a client.
>
> When I get back to home, I'll try to do some test from another computer, to
> discard problems with this installation.
>
> ...
>
> read more »
rsync -av --progress --partial folder host:
"""
Write failed: Broken pipekB/s 0:00:06
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]:
Broken pipe (32)
rsync: connection unexpectedly closed (7382 bytes received so far)
[sender]
rsync error: error in rsync protocol data stream (code 12) at
io.c(601) [sender=3.0.7]
"""
Using --bwlimit with low KBps seems to let it run a little longer
(eventually always fails within 15-30 minutes). Not sure if there is
any actual correlation though.
I have not seen the issue with other servers that are Fedora 12 or
CentOS 5 (running rsync version 2.6.8). I do not have another Ubuntu
server to test.
However, my upload speed is low (512kb/s) so perhaps the problem is not
triggered at that speed?
I will make a test upload from one of my work servers with lots of
bandwidth and see what happens.
That being said, I hope things don't act up too much for that
switch...and on a positive note, maybe the few folks that are having
issues may also see their issues disappear with this change.
Running rsync with multiple v options shows a problem in the io code:
"""
rsync: connection unexpectedly closed (8880 bytes received so far)
[sender]
[sender] _exit_cleanup(code=12, file=io.c, line=601): entered
rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]
[sender] _exit_cleanup(code=12, file=io.c, line=601): about to call
exit(255)
"""
I added the --blocking-io option which did not help.
Thinking it may be a problem with ssh now, I tried (as suggested
earlier, I think, ha):
"""
$ scp -r folder storage2.datastorageunit.com:foldertest
file1
95% 2320KB 1.7MB/s 00:00 ETAWrite failed: Broken pipe
lost connection
"""
Therefore it must be a problem with ssh and how fast the data is
transferring.
"""
debug3: Wrote -1 bytes for a total of 277109
Write failed: Broken pipe
lost connection
"""
I'll try Ubuntu later today and post the results. If it works with
Ubuntu then I'll assume Fedora 12 has bad code somewhere and I'll
write a nice rsync retry script.
The comments about using bwlimit to slow down the transfer, to me,
seem to suggest that you might want to temporarily take your router
out of the picture. The way I see it, your router is about the only
thing that you can control between the source and destination so why
not just try removing it? If it makes no difference then there is no
harm done. If it does make a difference, now we can focus our
troubleshooting the router itself.
Lastly, I don't know if anyone has noticed, but I have available for
download from the files section of this forum a shell script that I
have written for my system. I named it dsu_backup. It uses encfs or
truecrypt to locally mount your encrypted volume and then it uses
rsync to transfer the files. You have to set up a few things and
populate the script with your info, but once you do that you can run
it as a cron job and forget about it. Feel free to use it, critique
it, or use pieces of it to make your own script. Take a special look
at the parameters I am passing to sshfs, encfs, and rsync... they seem
to work really well for me anyway. Also, please feel free to ask
questions if you have trouble setting it up.
On Apr 8, 3:11 am, Jose Luis Fernandez-Mayoralas <y...@jluis.me>
wrote:
I think the other two guys should try it then. The only reason I still say so is because you have the issue following them no matter what ISP they try from apparently. Their reports are that they have this issue from the same pc at home and work, so I still think it's a relevant test...for them at least. And in my opinion, it's still not going to hurt even if you do have the same OS installed since a live CD is still more of a controlled variable...(even though one would expect the same results in your case having that as your OS.) That being said...it's still up to you as to whether or not you want to give it a try. All that being said...I do lean toward it being an issue along the internet at some point...not likely with the Server Side ISP based on the vast majority of other Subscribers not having any issues.
between the two points that is causing just enough jitter, latency, or whatever to cause rsync to drop. In one of the two "active" cases...the connection appears to be coming from Spain...which would obviously be a lot of hops.